Parallel Processing in R using "parallel" package - r

Question

I have two data frames:
> head(k)
V1
1 1814338070
2 1199215279
3 1283239083
4 1201972527
5 404900682
6 3093614019
> head(g)
start end state value
1 16777216 16777471 queensland 15169
2 16777472 16778239 fujian 0
3 16778240 16779263 victoria 56203
4 16779264 16781311 guangdong 0
5 16781312 16781823 tokyo 0
6 16781824 16782335 aichi 0
> dim(k)
[1] 624979 1
> dim(g)
[1] 5510305 4
I want to compare each value in data.frame(k) and match if it fits between the range of start and end of data.frame(g) and if it does return the value of state and value from data.frame(g)
The problem I have is due to the dimensions of both the data frame and to do the match and return my desired values it takes 5 hours on my computer. I've used the following method but I'm unable to make use of all cores on my computer and not even make it work correctly:
return_first_match_position <- function(int, start,end) {
match = which(int >= start & int <= end)
if(length(match) > 0){
return(match[1])
}
else {
return(match)
}
}
library(parallel)
cl = makeCluster(detectCores())
matches = Vectorize(return_first_match_position, 'int')(k$V1,g$start, g$end)
p = parSapply(cl, Vectorize(return_first_match_position, 'int')(k$V1,g$start, g$end), return_first_match_position)
stopCluster(cl)
desired output is % number of times state and value show up for every match of the number from data.frame(k) in data.frame(g)
Was wondering there there is an intelligent way of doing parallel processing in R ?
And can anyone please suggest (any sources) how I can learn/improve writing functions in R?

Answer 1

I think you want to do a rolling join. This can be done very efficiently with data.table:
DF1 <- data.frame(V1=c(1.5, 2, 0.3, 1.7, 0.5))
DF2 <- data.frame(start=0:3, end=0.9:3.9,
state=c("queensland", "fujian", "victoria", "guangdong"),
value=1:4)
library(data.table)
DT1 <- data.table(DF1, key="V1")
DT1[, pos:=V1]
# V1 pos
#1: 0.3 0.3
#2: 0.5 0.5
#3: 1.5 1.5
#4: 1.7 1.7
#5: 2.0 2.0
DT2 <- data.table(DF2, key="start")
# start end state value
#1: 0 0.9 queensland 1
#2: 1 1.9 fujian 2
#3: 2 2.9 victoria 3
#4: 3 3.9 guangdong 4
DT2[DT1, roll=TRUE]
# start end state value pos
#1: 0 0.9 queensland 1 0.3
#2: 0 0.9 queensland 1 0.5
#3: 1 1.9 fujian 2 1.5
#4: 1 1.9 fujian 2 1.7
#5: 2 2.9 victoria 3 2.0

Answer 2

so instead of editing the last one a lot (pretty much making a new one).. is this what you want:
I noticed that your end is always 1 before the next rows start, so what you want ( i think) is to just find out how many were within each interval and give that interval the state,value for that range. so
set.seed(123)
c1=seq(1,25,4)
c2=seq(4,30,4)
c3=letters[1:7]
c4=sample(seq(1,7),7)
c.all=cbind(c1,c2,c3,c4)
> c.all ### example data.frame that looks similar to yours
c1 c2 c3 c4
[1,] "1" "4" "a" "3"
[2,] "5" "8" "b" "7"
[3,] "9" "12" "c" "2"
[4,] "13" "16" "d" "1"
[5,] "17" "20" "e" "6"
[6,] "21" "24" "f" "5"
[7,] "25" "28" "g" "4"
k1 <- sample(seq(1,18),20,replace=T)
k1
[1] 2 1 15 14 4 15 3 17 18 1 4 3 16 15 2 4 8 11 7 16
fallsin <- cut(k1, c(as.numeric(c.all[,1]), max(c.all[,2])), labels=paste(c.all[,3], c.all[,4],sep=':'), right=F)
fallsin
[1] a:3 a:3 e:6 e:6 a:3 e:6 a:3 f:5 f:5 a:3 a:3 a:3 e:6 e:6 a:3 a:3 c:2 d:1 b:7 e:6
Levels: a:3 b:7 c:2 d:1 e:6 f:5 g:4
prop.table(table(fallsin))
a:3 b:7 c:2 d:1 e:6 f:5 g:4
0.45 0.05 0.05 0.05 0.30 0.10 0.00
where the names of the columns are the 'state:value' and the numbers are the percent of k1 that fall within the range of that label