Create lagged variables in panel data in Julia - julia

Question

I am trying to generate lagged variables using panel data (ID and year). By each ID, it might have different years of panel and sometimes years are not continuous within ID group. For example, we have the data set below:
ID
Year
x
1
2001
3
1
2002
1
1
2006
2
1
2007
2
2
2002
1
2
2003
5
3
2006
2
3
2007
2
3
2008
4
And the lagged variable for x that I want to generate is:
ID
Year
x
x_lag
1
2001
3
.
1
2002
1
3
1
2006
2
.
1
2007
2
2
2
2002
1
.
2
2003
5
1
3
2006
2
.
3
2007
2
2
3
2008
4
2
I found some other answers to how to create lagged variables by groups but it does not work for me because some of IDs in my data set have discontinuous years (ex. row 2-3 in the example above).
So, I am using the function that I have written down below:
function lagged(data,x)
for c in x
data[:,c*"_lag"] .= 0.0
end
allowmissing!(data)
for row in eachrow(data)
for c in x
if filter(y -> y.id == row.id && y.year == row.year - 1, data)[:,c] == []
row[c*"_lag"] = missing
else
row[c*"_lag"] = filter(y -> y.id == row.id && y.year == row.year - 1, data)[:,c][1]
end
end
end
return data
end
But it is extremely slow... Is there any faster way to create lagged variables in panel data with discontinuous years? Thanks!

Answer 1

Is this what you need?
julia> df = DataFrame(ID=[1,1,1,1,2,2,3,3,3],
Year=[2001, 2002, 2006, 2007, 2002, 2003, 2006, 2007, 2008],
x=[3,1,2,2,1,5,2,2,4])
9×3 DataFrame
Row │ ID Year x
│ Int64 Int64 Int64
─────┼─────────────────────
1 │ 1 2001 3
2 │ 1 2002 1
3 │ 1 2006 2
4 │ 1 2007 2
5 │ 2 2002 1
6 │ 2 2003 5
7 │ 3 2006 2
8 │ 3 2007 2
9 │ 3 2008 4
julia> function lag_v(year, x)
result = missings(eltype(x), length(x))
length(x) < 2 && return result
last = first(year)
for i in 2:length(x)
current = year[i]
current-last == 1 && (result[i] = x[i-1])
last = current
end
return result
end
lag_v (generic function with 1 method)
julia> transform(groupby(df, :ID), [:Year, :x] => lag_v => :x_lag)
9×4 DataFrame
Row │ ID Year x x_lag
│ Int64 Int64 Int64 Int64?
─────┼──────────────────────────────
1 │ 1 2001 3 missing
2 │ 1 2002 1 3
3 │ 1 2006 2 missing
4 │ 1 2007 2 2
5 │ 2 2002 1 missing
6 │ 2 2003 5 1
7 │ 3 2006 2 missing
8 │ 3 2007 2 2
9 │ 3 2008 4 2
and is it fast enough?

Answer 2

An approach using ShiftedArrays
Data
julia> using ShiftedArrays
julia> using DataFrames
julia> df = DataFrame(ID = [1, 1, 1, 1, 2, 2, 3, 3, 3],
Year = [2001, 2002, 2006, 2007, 2002, 2003, 2006, 2007, 2008],
x = [3, 1, 2, 2, 1, 5, 2, 2, 4])
Add helper column is to check if lagged Year == Year, then
remove missings, create and fill x_lag column and finally remove the helper column.
julia> function do_lag!(frame)
for grp in groupby(frame, :ID)
grp.is = (ShiftedArrays.lag(grp.Year, 1) .+ 1) .== grp.Year
end
frame.is[ismissing.(frame.is)] .= false
frame.x_lag = Vector{Union{Int, String}}(repeat(["."], size(frame)[1]))
frame.x_lag[findall(frame.is)] =
ShiftedArrays.lag(frame.x)[findall(frame.is)]
select!(frame, Not([:is]))
end
julia> do_lag!(df)
9×4 DataFrame
Row │ ID Year x x_lag
│ Int64 Int64 Int64 Union…
─────┼─────────────────────────────
1 │ 1 2001 3 .
2 │ 1 2002 1 3
3 │ 1 2006 2 .
4 │ 1 2007 2 2
5 │ 2 2002 1 .
6 │ 2 2003 5 1
7 │ 3 2006 2 .
8 │ 3 2007 2 2
9 │ 3 2008 4 2

Answer 3

And another option, without defining a function, but using a leftjoin with next-year version of df:
julia> df = DataFrame(ID=[1,1,1,1,2,2,3,3,3],
Year=[2001, 2002, 2006, 2007, 2002, 2003, 2006, 2007, 2008],
x=[3,1,2,2,1,5,2,2,4]);
julia> leftjoin(df, select(df, :ID,
:Year => ByRow(Base.Fix2(+,1)) => :Year,
:x => :lag_x); on=[:ID, :Year])
9×4 DataFrame
Row │ ID Year x lag_x
│ Int64 Int64 Int64 Int64?
─────┼──────────────────────────────
1 │ 1 2002 1 3
2 │ 1 2007 2 2
3 │ 2 2003 5 1
4 │ 3 2007 2 2
5 │ 3 2008 4 2
6 │ 1 2001 3 missing
7 │ 1 2006 2 missing
8 │ 2 2002 1 missing
9 │ 3 2006 2 missing
It's easier to see what this join does after sorting (though not necessary):
julia> sort!(leftjoin(df,
select(df, :ID, :Year => ByRow(Base.Fix2(+,1)) => :Year,
:x => :lag_x); on=[:ID, :Year]), [:ID, :Year])
9×4 DataFrame
Row │ ID Year x lag_x
│ Int64 Int64 Int64 Int64?
─────┼──────────────────────────────
1 │ 1 2001 3 missing
2 │ 1 2002 1 3
3 │ 1 2006 2 missing
4 │ 1 2007 2 2
5 │ 2 2002 1 missing
6 │ 2 2003 5 1
7 │ 3 2006 2 missing
8 │ 3 2007 2 2
9 │ 3 2008 4 2
This solution has the benefit of being easier to use when eventually the underlying DataFrame storage becomes a database (like SQLite, or Postgres), once the OP's operation scales to 100M users :-)

Answer 4

Why don't you just do
transform!(groupby(df, :id), :x => lag, :year => lag)
And then only keep the x_lag values where year_lag is 1?
EDIT
Now that I'm not on my phone here's what I meant in DataFrames minilanguage:
julia> using DataFrames; import ShiftedArrays: lag
julia> df = DataFrame(ID=[1,1,1,1,2,2,3,3,3],
Year=[2001, 2002, 2006, 2007, 2002, 2003, 2006, 2007, 2008],
x=[3,1,2,2,1,5,2,2,4]);
julia> transform!(groupby(df, :ID),
[:Year, :x] => ((y, x) -> ifelse.(coalesce.(y .- lag(y) .== 1, false), lag(x), missing)) => :x_lag)
9×4 DataFrame
Row │ ID Year x x_lag
│ Int64 Int64 Int64 Int64?
─────┼──────────────────────────────
1 │ 1 2001 3 missing
2 │ 1 2002 1 3
3 │ 1 2006 2 missing
4 │ 1 2007 2 2
5 │ 2 2002 1 missing
6 │ 2 2003 5 1
7 │ 3 2006 2 missing
8 │ 3 2007 2 2
9 │ 3 2008 4 2
I haven't benchmarked all the solutions in this thread, but testing this on a 10-million row DataFrame this runs in about 1 second on my laptop, which seems acceptable?
julia> df_test = DataFrame(id = repeat(1:1_000_000, inner = 10), year = reduce(vcat, [sort(rand(2000:2015, 10)) for _ ∈ 1:1_000_000]), x = rand(10_000_000));
julia> #time transform!(groupby(df_test, :id),
[:year, :x] => ((y, x) -> ifelse.(coalesce.(y .- lag(y) .== 1, false), lag(x), missing)) => :x_lag);
1.139166 seconds (9.33 M allocations: 940.402 MiB, 7.77% gc time, 16.18% compilation time)