Efficient way to reshape a dataset from long to wide - r

Question

I have a medical dataset which looks like this:
patient_id disease_id
1111111111 DISEASE:1
1111111111 DISEASE:2
1111111111 DISEASE:3
1111111111 DISEASE:4
1111111111 DISEASE:5
1111111111 DISEASE:6
1111111111 DISEASE:6
1111111112 DISEASE:1
1111111112 DISEASE:2
1111111112 DISEASE:4
1111111113 DISEASE:1
1111111113 DISEASE:5
which I need to feed into a neural network/random forest model. So, the only natural data representation to feed into the models I thought of was:
patient_id DISEASE:1 DISEASE:2 DISEASE:3 DISEASE:4 DISEASE:5 DISEASE:6 ...
11111111111 1 1 1 1 1 1 ...
11111111112 1 1 0 1 0 0 ...
11111111113 1 0 0 0 1 0 ...
But my dataset is very big (~50GB, 1.5 GB compressed) and has tons of disease_ids so that the reshaping this data in the most efficient way possible in R requires 11.7 TB of space in compressed in RDs format (I know this because I divided the dataset into 100 chunks and reshaping of a single one resulted in 117 GB heavy RDs file; merging 100 of these would produce something larger than 11.7TB).
Now, I have 5 dataset this big that I need to merge together, so I feel a bit stuck. I need to come up with a more efficient data representation but don't know how as I am dealing with categorical variables which will require 1-hot encoding. Can anyone suggest any alternative ways to deal with a data like this.

Answer 1

Given the size of the input you will want to do stream processing and R is not that suitable for such processing so here we use a simple gawk program instead.
gawk is available in Rtools on Windows and comes natively with most UNIX/Linux systems.
In the first pass the gawk program creates an associative array disease from the disease field, i.e. second field, of the input. Presumably the number of diseases is much smaller than the length of the file so this would likely fit into memory.
Then in a second pass it reads each group of records corresponding to a patient assuming that all records for a patient are consecutive. For each patient it outputs a single row with the patient id and a sequence of 0's and 1's such that the ith indicates absence or presence of the ith disease.
FNR == 1 { next } # skip header on both passes
# first pass - create disease array
FNR == NR {
disease[$2] = 0;
next;
}
# second pass - create and output flattened records
{
if ($1 != prevkey && FNR > 2) {
printf("%s ", prevkey);
for(d in disease) printf("%d ", disease[d]);
printf("\n");
for(d in disease) disease[d] = 0;
}
disease[$2] = 1;
prevkey = $1;
}
END {
if (FNR == NR) for(d in disease) {
print d;
} else {
printf("%s ", prevkey);
for(d in disease) printf("%d ", disease[d]);
printf("\n");
}
}
If we put the above gawk code in model_mat.awk then we can run it like this -- note that the file must be specified twice -- once for each of the two passes:
gawk -f model_mat.awk disease.txt disease.txt
The output is the following where we are assuming it is wanted that each disease be indicated by 1 if it is present or 0 if not.
1111111111 1 1 1 1 1 1
1111111112 1 1 0 1 0 0
1111111113 1 0 0 0 1 0
If we run it with only one disease.txt argument then it will only run the first pass and then at the end list the diseases without duplicates:
gawk -f model_mat.awk disease.txt
giving:
DISEASE:1
DISEASE:2
DISEASE:3
DISEASE:4
DISEASE:5
DISEASE:6
Listing diseases
An alternative for listing diseases is this UNIX pipeline which lists the diseases without duplicates and sorts them. sed removes the header, cut takes the third space separated field (it is the third because there are two spaces between the two fields) and sort sorts it taking unique elements.
sed 1d disease.txt | cut -f 3 -d " " | sort -u > diseases-sorted.txt
Sorting and Merging
The GNU sort utility can sort and merge files larger than memory and has a parallel option to speed it up. Also see the free cmsort utility (Windows only).
csvfix
Below are some scripts using the free csvfix command line utility. You may need to modify the quotes depending on the command line processor/shell you are using and will need to put each on a single line or appropriately escape the newline (backslash for bash, circumflex for Windows cmd). We have shown each pipeline spread over separate lines for clarity.
The first pipeline below creates a one column list of diseases in disease-list.txt . The first csvfix command in it removes the header, the second csvfix command extracts the second field (i.e. drops the patient id) and the last csvfix command reduces it to unique diseases.
The second pipeline below creates a file with one row per patient with the patient id followed by the diseases for that patient. The first csvfix command in it removes the header, the second converts it to csv format and the last csvfix command flattens it.
csvfix remove -if "$line == 1" -smq disease.txt |
csvfix read_dsv -s " " -cm -f 2 |
csvfix uniq -smq > disease-list.txt
csvfix remove -if "$line == 1" -smq disease.txt |
csvfix read_dsv -s " " -cm -f 1,2 |
csvfix flatten -smq > flat.txt

Answer 2

You are raising interesting question. Analyzing that volume of data with R will be a real change.
So, I can only give you general advices. First, I think you need to dissociate RAM and disk storage. Using Rds won't help you regarding the efficiency of the reshaping but will produce smaller data on disk than csv.
Concerning the efficiency of the reshaping
data.table
If you want an in-memory approach, I don't see any other possibility than using data.table::dcast. In that case, follow #Ronak Shah recommendation:
library(data.table)
setDT(df)
df[, n := 1]
dcast(unique(df), patient_id~ disease_id, value.var = "n", fill = 0)
?data.table::dcast :
In the spirit of data.table, it is very fast and memory efficient, making it well-suited to handling large data sets in RAM. More importantly, it is capable of handling very large data quite efficiently in terms of memory usage.
Other solutions
With data that voluminous, I think in-memory is not the most appropriate approach. You might have a look to database approaches (especially postgreSQL) or Spark.
Databases
You have several options to use postgreSQL in R. One of them is dbplyr: if you know the tidyverse syntax you will find familiar verbs. The pivot operation is a little bit trickier for database than standard R dataframe but you might find some ways to do that. You won't have difficulties to find some people more expert on databases than me that can give you very interesting tricks.
Spark
Spark can be a very good candidate to perform the reshaping if you can spread your tasks among executors in a serveur. If you are on a personal computer (standalone mode) you can still parallelize tasks between your cores but do not forget to change the spark.memory.fraction parameter of the session otherwise I think you might experience out of memory problems. I am more used to pyspark than sparkR but I think the logic will be the same.
Since Spark 1.6, you can pivot your data (ex: pyspark doc). This enables a wide to long conversion. Something in this spirit (pyspark code)
df.withColumn("n", psf.lit(1)).pivot("patient_id").sum("n")
Concerning the size on disk
You use Rds. You have some format more compressed, e.g. fst. parquet files are also very compressed, maybe one of the best options to store voluminous data. You can read them with SparkR or using arrow package