split data.table - r

Question

I have a data.table which I want to split into two. I do this as follows:
dt <- data.table(a=c(1,2,3,3),b=c(1,1,2,2))
sdt <- split(dt,dt$b==2)
but if I want to to something like this as a next step
sdt[[1]][,c:=.N,by=a]
I get the following warning message.
Warning message: In [.data.table(sdt[[1]], , :=(c, .N), by = a) :
Invalid .internal.selfref detected and fixed by taking a copy of the
whole table, so that := can add this new column by reference. At an
earlier point, this data.table has been copied by R. Avoid key<-,
names<- and attr<- which in R currently (and oddly) may copy the whole
data.table. Use set* syntax instead to avoid copying: setkey(),
setnames() and setattr(). Also, list(DT1,DT2) will copy the entire DT1
and DT2 (R's list() copies named objects), use reflist() instead if
needed (to be implemented). If this message doesn't help, please
report to datatable-help so the root cause can be fixed.
Just wondering if there is a better way of splitting the table so that it would be more efficient (and would not get this message)?

Answer 1

This works in v1.8.7 (and may work in v1.8.6 too) :
> sdt = lapply(split(1:nrow(dt), dt$b==2), function(x)dt[x])
> sdt
$`FALSE`
a b
1: 1 1
2: 2 1
$`TRUE`
a b
1: 3 2
2: 3 2
> sdt[[1]][,c:=.N,by=a] # now no warning
> sdt
$`FALSE`
a b c
1: 1 1 1
2: 2 1 1
$`TRUE`
a b
1: 3 2
2: 3 2
But, as #mnel said, that's inefficient. Please avoid splitting if possible.

Answer 2

I was looking for some way to do a split in data.table, I came across this old question.
Sometime a split is what you want to do, and the data.table "by" approach is not convenient.
Actually you can easily do your split by hand with data.table only instructions and it works very efficiently:
SplitDataTable <- function(dt,attr) {
boundaries=c(0,which(head(dt[[attr]],-1)!=tail(dt[[attr]],-1)),nrow(dt))
return(
mapply(
function(start,end) {dt[start:end,]},
head(boundaries,-1)+1,
tail(boundaries,-1),
SIMPLIFY=F))
}

Answer 3

As mentionned above (#jangorecki), the package data.table already has its own function for splitting. In that simplified case we can use:
> dt <- data.table(a = c(1, 2, 3, 3), b = c(1, 1, 2, 2))
> split(dt, by = "b")
$`1`
a b
1: 1 1
2: 2 1
$`2`
a b
1: 3 2
2: 3 2
For more difficult/concrete cases, I would recommend to create a new variable in the data.table using the by reference functions := or set and then call the function split. If you care about performance, make sure to always remain in the data.table environment e.g., dt[, SplitCriteria := (...)] rather than computing the splitting variable externallly.