How do I load multiple CSV into DataFrames in Julia? - julia

Question

I already know how to load a single CSV into a DataFrame:
using CSV
using DataFrames
df = DataFrame(CSV.File("C:\\Users\\username\\Table_01.csv"))
How would I do this when I have several CSV files, e.g. Table_01.csv, Table_02.csv, Table_03.csv?
Would I create a bunch of empty DataFrames and use a for loop to fill them? Or is there an easier way in Julia? Many thanks in advance!

Answer 1

If you want multiple data frames (not a single data frame holding the data from multiple files) there are several options.
Let me start with the simplest approach using broadcasting:
dfs = DataFrame.(CSV.File.(["Table_01.csv", "Table_02.csv", "Table_03.csv"]))
or
dfs = #. DataFrame(CSV.File(["Table_01.csv", "Table_02.csv", "Table_03.csv"]))
or (with a bit of more advanced stuff, using function composition):
(DataFrame∘CSV.File).(["Table_01.csv", "Table_02.csv", "Table_03.csv"])
or using chaining:
CSV.File.(["Table_01.csv", "Table_02.csv", "Table_03.csv"]) .|> DataFrame
Now other options are map as it was suggested in the comment:
map(DataFrame∘CSV.File, ["Table_01.csv", "Table_02.csv", "Table_03.csv"])
or just use a comprehension:
[DataFrame(CSV.File(f)) for f in ["Table_01.csv", "Table_02.csv", "Table_03.csv"]]
(I am listing the options to show different syntactic possibilities in Julia)

Answer 2

This is how I have done it, but there might be an easier way.
using DataFrames, Glob
import CSV
function readcsvs(path)
files=glob("*.csv", path) #Vector of filenames. Glob allows you to use the asterisk.
numfiles=length(files) #Number of files to read.
tempdfs=Vector{DataFrame}(undef, numfiles) #Create a vector of empty dataframes.
for i in 1:numfiles
tempdfs[i]=CSV.read(files[i]) #Read each CSV into its own dataframe.
end
masterdf=outerjoin(tempdfs..., on="Column In Common") #Join the temporary dataframes into one dataframe.
end

Answer 3

A simple solution where you don't have to explicitly enter filenames:
using CSV, Glob, DataFrames
path = raw"C:\..." # directory of your files (raw is useful in Windows to add a \)
files=glob("*.csv", path) # to load all CSVs from a folder (* means arbitrary pattern)
dfs = DataFrame.( CSV.File.( files ) ) # creates a list of dataframes
# add an index column to be able to later discern the different sources
for i in 1:length(dfs)
dfs[i][!, :sample] .= i # I called the new col sample
end
# finally, if you want, reduce your collection of dfs via vertical concatenation
df = reduce(vcat, dfs)