Julia: Why is the memory blowing up inside this loop? - julia

Question

I have some multi-threaded code in which each thread calls a function f(df::DataFrame) which reads a column of that DataFrame and finds the indices where the column is greater than 0:
function f(df::DataFrame)
X = df[:time]
return findall(x->x>0, X)
end
Inside the main thread I read in an R *.rds file which Julia converts to a DataFrame which I'm passing to f() as follows:
rds = "blabla.rds"
objs = load(rds);
params = collect(0.5:0.005:0.7)
for i in 1:length(objs)
cols = [string(name) for name in names(objs.data[i]) if occursin("bla",string(name))]
hypers = [(a,b) for a in cols, b in params] # length ~2000
Threads.#threads for hi in 1:length(hypers) # MEMORY BLOWS UP HERE
df = f(objs.data[i])
end
end
Each df that is passed to f() is roughly 0.7GB. Analysing the memory usage when the multi-threaded loop is run, the memory usage goes up to ~30GB. There are 25 threads and ~2000 calls to f(). Any idea why the memory is exploding?
NOTE: The problem seems to be ameliorated by calling GC.gc() inside the loop every so often, which seems like a botch...
NOTE also: This happens whether or not I use a regular or multi-threaded loop.
EDIT:
Profiling the code as follows:
function foo(objs)
for i in 1:length(objs)
df = objs.data[i]
Threads.#threads for hi in 1:2000
tmp = f(df)
end
end
end
#benchmark(foo($objs))
gives
BenchmarkTools.Trial:
memory estimate: 32.93 GiB
allocs estimate: 48820
--------------
minimum time: 2.577 s (0.00% GC)
median time: 2.614 s (0.00% GC)
mean time: 2.614 s (0.00% GC)
maximum time: 2.651 s (0.00% GC)
--------------
samples: 2
evals/sample: 1