R converting to long format, pattern - r

Question

I would like to convert a data.table like this one from wide format to long.
set.seed(1)
DT <- data.table(
ID = c(1:5, NA),
Name = c("Bob","Ana","Smith","Sam","Big","Lulu"),
Kind_2001 = factor(sample(c(letters[1:3], NA), 6, TRUE)),
Kind_2002 = factor(sample(c(letters[1:3], NA), 6, TRUE)),
Kind_2003 = factor(sample(c(letters[1:3], NA), 6, TRUE)),
Conc_2001 = sample(99,6),
Conc_2002 = sample(79,6),
Conc_2003 = sample(49,6)
)
ID Name Kind_2001 Kind_2002 Kind_2003 Conc_2001 Conc_2002 Conc_2003
1 Bob b NA c 38 22 24
2 Ana b c b 77 31 29
3 Smith c c NA 91 2 49
4 Sam NA a b 21 30 9
5 Big a a c 62 66 38
NA Lulu NA a NA 12 26 30
And I would like to get something like this:
ID Name Year Kind Conc
1 Bob 2001 b 38
1 Bob 2002 NA 22
1 Bob 2003 c 24
2 Ana 2001 b 77
2 Ana 2002 c 31
2 Ana 2003 b 29
...
The real table has many more variables, I'm looking for a solution without explicitly saying every column name or number, detecting automatically the pattern.
I have two kind of columns, some ending with an underscore and a four digit year, such as _2001, and the other without that ending.
Some can have an underscore in the middle of the name (this will be kept untransformed).
I would like to transform the columns ending with a year to long format.
I've tried with
melt(DT, id=1:2, variable.name = "year")
or with
melt(DT, id=1:2, measure=patterns("_2[0-9][0-9][0-9]$"))
but I'm not getting what I want.
Maybe I first need to filter the names with gsub.
PD: I've found this solution.
posi <- grep("_[0-9][0-9][0-9][0-9]$",colnames(DT))
work <- unique(gsub("_[0-9][0-9][0-9][0-9]$","",colnames(DT)[posi]))
melt(DT, measure=patterns(paste0("^",work)), variable="year", value.name=work)
It almost works but the year column is not populated properly. I'm missing something or it's a bug.
And I'm sure it could be written simpler.
ID Name year Kind Conc
1 Bob 1 b 38
2 Ana 1 b 77
3 Smith 1 c 91
4 Sam 1 NA 21
5 Big 1 a 62
NA Lulu 1 NA 12
1 Bob 2 NA 22
2 Ana 2 c 31
3 Smith 2 c 2
4 Sam 2 a 30
5 Big 2 a 66
NA Lulu 2 a 26
1 Bob 3 c 24
2 Ana 3 b 29
3 Smith 3 NA 49
4 Sam 3 b 9
5 Big 3 c 38
NA Lulu 3 NA 30
Regards
I've tried eddi solution with my database and I get the error:
"Error: cannot allocate vector of size 756.5 Mb"
even though I have 16GB of memory.

Answer 1

We can solve this on scale using reshape() from base R, without having to explicitly name variables.
# First we get indices of colnames that have format "_1234" at the end
tomelt <- grep("_([0-9]{4})$",names(DT))
# Now we use these indices to reshape data
reshape(DT, varying = tomelt, sep = "_",
direction = 'long', idvar = "ID", timevar = "Year)
# ID Name Year Kind Conc
# 1: 1 Bob 2001 b 38
# 2: 2 Ana 2001 b 77
# 3: 3 Smith 2001 c 91
# 4: 4 Sam 2001 NA 21
# 5: 5 Big 2001 a 62
# 6: NA Lulu 2001 NA 12
...

Answer 2

If we are looking for data.table solution, extract the prefix part from the names of the "DT" and use the unique elements as patterns in the measure argument in melt. Similarly, the suffix from "Year" is extracted and replace the numeric index with that.
nm <- unique(sub("_\\d+", "", names(DT)[-(1:2)]))
yr <- unique(sub("\\D+_", "", names(DT)[-(1:2)]))
melt(DT, measure = patterns(paste0("^", nm)), value.name = nm,
variable.name = "Year")[, Year := yr[Year]][]
# ID Name Year Kind Conc
# 1: 1 Bob 2001 b 38
# 2: 2 Ana 2001 b 77
# 3: 3 Smith 2001 c 91
# 4: 4 Sam 2001 NA 21
# 5: 5 Big 2001 a 62
# 6: NA Lulu 2001 NA 12
# 7: 1 Bob 2002 NA 22
# 8: 2 Ana 2002 c 31
# 9: 3 Smith 2002 c 2
#10: 4 Sam 2002 a 30
#11: 5 Big 2002 a 66
#12: NA Lulu 2002 a 26
#13: 1 Bob 2003 c 24
#14: 2 Ana 2003 b 29
#15: 3 Smith 2003 NA 49
#16: 4 Sam 2003 b 9
#17: 5 Big 2003 c 38
#18: NA Lulu 2003 NA 30

Answer 3

Here's an option that's more robust with respect to the order of your columns, as well as missing/extra years:
dcast(melt(DT, id.vars = c("ID", "Name"))
[, .(ID, Name, sub('_.*', '', variable), sub('.*_', '', variable), value)],
ID + Name + V4 ~ V3)
# ID Name V4 Conc Kind
# 1: 1 Bob 2001 38 b
# 2: 1 Bob 2002 22 NA
# 3: 1 Bob 2003 24 c
# 4: 2 Ana 2001 77 b
# 5: 2 Ana 2002 31 c
# 6: 2 Ana 2003 29 b
# 7: 3 Smith 2001 91 c
# 8: 3 Smith 2002 2 c
# 9: 3 Smith 2003 49 NA
#10: 4 Sam 2001 21 NA
#11: 4 Sam 2002 30 a
#12: 4 Sam 2003 9 b
#13: 5 Big 2001 62 a
#14: 5 Big 2002 66 a
#15: 5 Big 2003 38 c
#16: NA Lulu 2001 12 NA
#17: NA Lulu 2002 26 a
#18: NA Lulu 2003 30 NA
Edit for many id columns:
idvars = grep("_", names(DT), invert = TRUE)
dcast(melt(DT, id.vars = idvars)
[, `:=`(var = sub('_.*', '', variable),
year = sub('.*_', '', variable),
variable = NULL)],
... ~ var, value.var='value')

Answer 4

In case anybody is interested I post here my full solution,
able to work with datasets bigger than memory. It uses some of your ideas and some mine.
My data is the file file.csv (or you can even do it with a compressed file using fread("unzip -c name.zip").
## Initialization
nline <- 1500000 # total number of lines or use wc -l to do it automatically.
chunk <- 5000 # change it according to your memory and number of columns.
times <- ceiling(nline/chunk)
name <- names(fread("file.csv", stringsAsFactors=F, integer64 = "character", nrows=0, na.strings=c("", "NA")) )
idvars = grep("_20[0-9][0-9]$",name , invert = TRUE)
# Now we loop every chunk
for(iter in 0:(times-1)) {
my <- fread("file.csv", stringsAsFactors=F, integer64 = "character", skip=1+(iter*chunk), nrows=chunk, na.strings=c("", "NA"))
colnames(my) <- name
temp <- melt(my, id.vars = idvars)
newfile <- dcast(
temp[, `:=`(var = sub('_20[0-9][0-9]$', '', variable), year = sub('.*_', '', variable), variable = NULL)],
... ~ var, value.var='value')
fwrite(newfile, "long.csv", quote=FALSE, sep=",", append=T)
rm(temp); rm(newfile); rm(my); gc()
}
#
As said before the problem with this method is that it converts all the value to character but if you save them to a file and read the file again (as here) you get the proper classes.
In case of very large files this method is very slow.
I encourage you to improve this solution or suggest any generic solution with tidyr, splitstackshape or other packages.
Or even better it would be great to do it with a database such as sqlite.
The solution should work on datasets with unordered columns or even with "_" in the middle of the name, such as:
set.seed(1)
DT <- data.table(
ID = c(1:15),
Name = c("Bob","Ana","Smith","Sam","Big","Lulu", "Loli", "Chochi", "Tom", "Dick", "Pet", "Shin", "Rock", "Pep", "XXX"),
Kind_2001 = factor(sample(c(letters[1:3], NA), 15, TRUE)),
Kind_2002 = factor(sample(c(letters[1:3], NA), 15, TRUE)),
Kind_2003 = factor(sample(c(letters[1:3], NA), 15, TRUE)),
Conc_2004 = sample(49,15),
aa_Conc_2001 = c(sample(99,14), NA),
Conc_2002 = sample(79,15)
)