Access columns in a CSV.Row by index - julia

Question

I have a very large (~120 GB) CSV file with ~100 columns. I want to iterate through the file line by line using CSV.File and aggregate certain ranges of columns. However, it appears that there is no getindex method for the CSV.Row type. Here is a simplified example:
using CSV
using DataFrames
df = DataFrame(reshape(1:60, 6, 10)) # column names are x1 through x10
CSV.write("test_data.csv", df)
file = CSV.File("test_data.csv")
row1 = first(file)
row1.x3 # Works fine
# Both of these throw method errors:
row1[4]
row1[4:7]
Suppose that for each row I want to sum columns [1:3; 8:10] in a variable a and sum columns 4:7 in a variable b. The final output should be a data frame with columns a and b. Is there an easy way to do this when iterating through CSV.Rows?

Answer 1

Here's a version that lets you avoid having to think about translating to the "drop"/"take" logic:
using CSV, Tables, DataFrames
df = DataFrame(reshape(1:60, 6, 10)) # column names are x1 through x10
CSV.write("test_data.csv", df)
function aggregate_file(path, a_inds, b_inds)
file = CSV.File(path)
a, b = Int64[], Int64[]
a_cols = propertynames(file)[a_inds]
b_cols = propertynames(file)[b_inds]
for row in file
push!(a, sum(getproperty.(Ref(row), a_cols)))
push!(b, sum(getproperty.(Ref(row), b_cols)))
end
DataFrame(a = a, b = b)
end
julia> aggregate_file("test_data.csv", [1:3; 8:10], 4:7)
6×2 DataFrame
│ Row │ a │ b │
│ │ Int64 │ Int64 │
├─────┼───────┼───────┤
│ 1 │ 168 │ 112 │
│ 2 │ 174 │ 116 │
│ 3 │ 180 │ 120 │
│ 4 │ 186 │ 124 │
│ 5 │ 192 │ 128 │
│ 6 │ 198 │ 132 │

Answer 2

You can use Tables.eachcolumn from the Tables.jl, since CSV.File supports the Tables interface. Tables.eachcolumn will create an iterator over the columns. Then you can use a combination of Iterators.take and Iterators.drop to access the desired ranges of columns:
using CSV
using Tables
using DataFrames
using Base.Iterators: take
using Base.Iterators: drop
function aggregate_file(path)
file = CSV.File(path)
a, b = Int64[], Int64[]
for row in file
cols = Tables.eachcolumn(row)
sum1to3 = sum(take(cols, 3))
sum8to10 = sum(drop(cols, 7))
push!(a, sum1to3 + sum8to10)
sum4to7 = sum(drop(take(cols, 7), 3))
push!(b, sum4to7)
end
DataFrame(a = a, b = b)
end
julia> aggregate_file("test_data.csv")
6×2 DataFrame
│ Row │ a │ b │
│ │ Int64 │ Int64 │
├─────┼───────┼───────┤
│ 1 │ 168 │ 112 │
│ 2 │ 174 │ 116 │
│ 3 │ 180 │ 120 │
│ 4 │ 186 │ 124 │
│ 5 │ 192 │ 128 │
│ 6 │ 198 │ 132 │
If you need to aggregate over an arbitrary set of column indices, you could use enumerate on the column iterator:
inds = [2, 4, 7]
sum(j for (i, j) in enumerate(cols) if i in inds)
EDIT:
I did a performance comparison of mine and #IanFiske's answer. His version appears to be faster and use less memory:
julia> using BenchmarkTools
julia> #btime aggregate_file("test_data.csv");
118.687 μs (550 allocations: 24.42 KiB)
julia> #btime aggregate_file("test_data.csv", [1:3; 8:10], 4:7);
62.416 μs (236 allocations: 14.48 KiB)