I have written a function with 3 nested foreach loops, running in parallel. The goal of the function is to split a list of 30 [10,5] matrices (i.e. [[30]][10,5]) into a list of 5 [10,30] matrices (i.e. [[5]][10,30]).
However, I am trying to run this function with 1,000,000 paths (i.e. foreach (m = 1:1000000)), and obviously, the performance is terrible.
I'd like to avoid apply functions if possible because I've found that they don't work well when used in conjunction with parallel foreach loops:
library(foreach)
library(doParallel)
# input matr: a list of 30 [10,5] matrices
matrix_splitter <- function(matr) {
time_horizon <- 30
paths <- 10
asset <- 5
security_paths <- foreach(i = 1:asset, .combine = rbind, .packages = "doParallel", .export = "daily") %dopar% {
foreach(m = 1:paths, .combine = rbind, .packages = "doParallel", .export = "daily") %dopar% {
foreach(p = daily, .combine = c) %dopar% {
p[m,i]
}
}
}
df_securities <- as.data.frame(security_paths)
split(df_securities, sample(rep(1:paths), asset))
}
Overall, I'm trying to convert this data format:
[[30]]
[,1] [,2] [,3] [,4] [,5]
[1,] 0.2800977 2.06715521 0.9196326 0.3560659 1.36126507
[2,] -0.5119867 0.24329025 0.1513218 -1.2528092 -0.04795098
[3,] -2.0293933 -1.17989270 0.3053376 -0.9528611 0.86758140
[4,] -0.6419024 -0.24846720 -0.6640066 -1.7104961 -0.32759406
[5,] -0.4340359 -0.44034013 3.3440507 0.7380613 2.01237069
[6,] -0.6679914 -0.01332117 1.9286056 -0.7194116 0.15549978
[7,] 0.5919820 0.11616685 -0.8424634 -0.7652715 1.34176688
[8,] 0.8079152 0.40592119 -0.4291811 0.9358829 -0.97479314
[9,] -0.0265207 -0.03598320 1.1287344 0.4732984 1.37792596
[10,] 1.0553966 0.65776721 -1.2833613 -0.2414846 0.81528686
To this format (obviously up to V30):
$`5`
V1 V2 V3 V4 V5 V6 V7
result.2 -0.11822260 1.7712833 1.97737285 -1.6643193 0.4788075 1.2394064 1.4800787
result.7 -1.23251178 0.4267885 -0.07728632 0.3463092 0.8766395 0.6324840 0.5946710
result.2.1 -1.27309457 -0.3128173 -0.79561297 -0.4713307 -0.4344864 0.4688124 -0.5646857
result.7.1 0.51702719 -1.6242650 -2.37976199 -0.1088408 0.4846507 -0.7594376 0.9326529
result.2.2 1.77550390 0.9279155 0.26168402 0.4893835 1.4131326 0.5989508 -0.3434010
result.7.2 -0.01590682 -0.5568578 1.35789122 -0.1385092 -0.4501515 -0.2581724 0.5451699
result.2.3 0.30400225 -1.0245640 -0.05285694 -0.1354228 0.3070331 -0.7618850 1.0330961
result.7.3 -0.08139912 0.4106541 1.40418839 0.2471505 1.2106539 1.3844721 0.4006751
result.2.4 0.94977544 -0.8045054 1.48791211 1.4361686 -0.3789274 -1.9570125 -1.6576634
result.7.4 0.70449194 1.6887800 0.56447340 0.6465640 2.6865388 -0.7367524 0.6242624
V8 V9 V10 V11 V12 V13
result.2 -0.432404728 -1.6225350 0.09855465 0.17371907 0.3081843 0.15148452
result.7 -0.597420706 0.6173004 0.07518596 2.01741406 0.1767152 -0.39219471
result.2.1 0.918408322 -1.6896424 -0.13409626 0.38674224 0.3491750 -1.61083286
result.7.1 2.564057340 -0.7696399 1.06103614 1.38528367 1.1684045 -0.08467871
result.2.2 0.951995816 0.1910284 1.79943500 2.13909498 0.2847664 0.31094568
result.7.2 -0.479349220 -0.2368760 0.04298525 -0.40385960 0.3986555 -1.93499213
result.2.3 -1.382370069 1.0459845 -0.33106323 -0.43362925 0.7045572 -0.30211601
result.7.3 -1.457106442 0.1487447 -2.52392942 -0.02399523 -1.0349746 0.87666365
result.2.4 -0.848879365 0.7521024 0.16790915 0.47112444 0.8886361 -0.12733039
result.7.4 -0.003350467 0.4021858 -1.80031445 -1.42399232 1.0507765 -0.36193846
The package plyr is designed for this problem thanks to alply. The idea is: unlist your list, fromat it in the appropriate way in an array, and convert this array to a list of matrix using alply.
Example of transformation of a list of 2 matrix 3x5 to a list of 5 matrix 2x3:
library(plyr)
lst = list(matrix(1:15, ncol=5), matrix(10:24, ncol=5))
alply(array(unlist(lst), c(2,3,5)),3)
#$`1`
# [,1] [,2] [,3]
#[1,] 1 3 5
#[2,] 2 4 6
#$`2`
# [,1] [,2] [,3]
#[1,] 7 9 11
#[2,] 8 10 12
#$`3`
# [,1] [,2] [,3]
#[1,] 13 15 11
#[2,] 14 10 12
#$`4`
# [,1] [,2] [,3]
#[1,] 13 15 17
#[2,] 14 16 18
#$`5`
# [,1] [,2] [,3]
#[1,] 19 21 23
#[2,] 20 22 24
I believe you are looking for the answer to this:
Function to split a matrix into sub-matrices in R
You would just use do.call(rbind, matlist) as an input to those functions.
I would convert all of your list into a great big vector, and then re-dimension it.
For my solution, I started with:
[[28]]
[,1] [,2] [,3] [,4] [,5]
[1,] 1 11 21 31 41
[2,] 2 12 22 32 42
[3,] 3 13 23 33 43
[4,] 4 14 24 34 44
[5,] 5 15 25 35 45
[6,] 6 16 26 36 46
[7,] 7 17 27 37 47
[8,] 8 18 28 38 48
[9,] 9 19 29 39 49
[10,] 10 20 30 40 50
Repeated thirty times. This is the variable orig. My code:
flattened.vec <- unlist(orig) #flatten the list of matrices into one big vector
dim(flattened.vec) <-c(10,150) #need to rearrange the vector so the re-shape comes out right
transposed.matrix <- t(flattened.vec) #transposing to make sure right elements go to the right place
new.matrix.list <- split(transposed.matrix,cut(seq_along(transposed.matrix)%%5, 10, labels = FALSE)) #split the big, transposed matrix into 5 10x30 matrices
This code gives you 5 vectors, that you need to dim(10,30) and then use t() on them in a foreach to get 5 30X10 vectors (I would normally use an apply function, and am not familiar with the foreach library).
End result for one of the 5 matrices result after doing so:
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12] [,13] [,14] [,15] [,16] [,17]
[1,] 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
[2,] 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2
[3,] 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3
[4,] 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4 4
[5,] 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5 5
[6,] 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6
[7,] 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7
[8,] 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8 8
[9,] 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9 9
[10,] 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10
[,18] [,19] [,20] [,21] [,22] [,23] [,24] [,25] [,26] [,27] [,28] [,29] [,30]
[1,] 1 1 1 1 1 1 1 1 1 1 1 1 1
[2,] 2 2 2 2 2 2 2 2 2 2 2 2 2
[3,] 3 3 3 3 3 3 3 3 3 3 3 3 3
[4,] 4 4 4 4 4 4 4 4 4 4 4 4 4
[5,] 5 5 5 5 5 5 5 5 5 5 5 5 5
[6,] 6 6 6 6 6 6 6 6 6 6 6 6 6
[7,] 7 7 7 7 7 7 7 7 7 7 7 7 7
[8,] 8 8 8 8 8 8 8 8 8 8 8 8 8
[9,] 9 9 9 9 9 9 9 9 9 9 9 9 9
[10,] 10 10 10 10 10 10 10 10 10 10 10 10 10
Incidentally, this is probably what the plyr package does on its own already (as posted by Colonel Beauvel), just manually instead of using an external library
Related
I have a matrix:
mat <- matrix(c(2,11,3,1,2,4,55,65,12,4,6,6,7,9,3,23,16,77,5,5,7),ncol = 3, byrow = TRUE)
[,1] [,2] [,3]
[1,] 2 11 3
[2,] 1 2 4
[3,] 55 65 12
[4,] 4 6 6
[5,] 7 9 3
[6,] 23 16 77
[7,] 5 5 7
I want to add a column with rows index. This index will starts at 1 and repeats the same index, until it arrived to a row where the rowsums is > 100 to move to the next value.
Indx[,2][,3][,4]
[1,] 1 2 11 3
[2,] 1 1 2 4
[3,] 2 55 65 12
[4,] 3 4 6 6
[5,] 3 7 9 3
[6,] 4 23 16 77
[7,] 5 5 5 7
Using rle:
matRle <- rle(rowSums(mat) > 100)$lengths
cbind(rep(seq(length(matRle)), matRle), mat)
# [,1] [,2] [,3] [,4]
# [1,] 1 2 11 3
# [2,] 1 1 2 4
# [3,] 2 55 65 12
# [4,] 3 4 6 6
# [5,] 3 7 9 3
# [6,] 4 23 16 77
# [7,] 5 5 5 7
A solution using dplyr.
library(dplyr)
mat2 <- mat %>%
as.data.frame() %>%
mutate(Indx = cumsum(rowSums(dat) > 100 | lag(rowSums(dat) > 100, default = TRUE))) %>%
select(Indx, paste0("V", 1:ncol(mat))) %>%
as.matrix()
mat2
# Indx V1 V2 V3
# [1,] 1 2 11 3
# [2,] 1 1 2 4
# [3,] 2 55 65 12
# [4,] 3 4 6 6
# [5,] 3 7 9 3
# [6,] 4 23 16 77
# [7,] 5 5 5 7
cbind(cumsum(replace(a<-rowSums(mat)>100,which(a==1)+1,1))+1,mat)
[,1] [,2] [,3] [,4]
[1,] 1 2 11 3
[2,] 1 1 2 4
[3,] 2 55 65 12
[4,] 3 4 6 6
[5,] 3 7 9 3
[6,] 4 23 16 77
[7,] 5 5 5 7
What does this do??:
first obtain the rowSums which are greater than 100
a<-rowSums(mat)>100
Then the next row for every row>100, should have the next index. Thus do a replace and cumsum:
cumsum(replace(a,which(a==1)+1,1))
Now you will realize that this starts from zero, so you add 1.
We could do this with rleid from data.table
library(data.table)
cbind(Indx = rleid(rowSums(mat) > 100), mat)
# Indx
#[1,] 1 2 11 3
#[2,] 1 1 2 4
#[3,] 2 55 65 12
#[4,] 3 4 6 6
#[5,] 3 7 9 3
#[6,] 4 23 16 77
#[7,] 5 5 5 7
The command
matrix(sample.int(12, 9*12, TRUE), 9, 12)
generates an integer random matrix (9 rows and 12 columns) with integer values from 1 to 12. I wonder if there is a version of this code that generates a matrix whose rows are integer random rows with value from 1 to 12 (without repetition). I was able to find a "trivial" answer to this question; with
matrix(sample.int(m, 1*12), 9, 12, byrow=TRUE)
I obtain a matrix of this kind, but the rows are all equal to each other (this is the same row repeated 9 times).
The replicate function (which repeats an operation like sample(12) a specified number of times) returns a matrix whose column major orientation can be flipped to your desired row orientation with t:
t( replicate(9, {sample(12)} ) )
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12]
[1,] 9 11 5 3 4 6 2 8 10 12 7 1
[2,] 4 5 12 6 8 2 9 1 11 10 7 3
[3,] 9 8 10 12 2 6 3 7 4 1 11 5
[4,] 4 9 1 2 6 11 8 5 7 3 12 10
[5,] 1 2 4 5 11 6 3 8 10 9 12 7
[6,] 4 8 10 12 5 9 2 7 11 1 3 6
[7,] 5 7 8 4 1 6 10 11 2 3 12 9
[8,] 2 4 10 1 12 5 7 6 11 3 8 9
[9,] 2 7 9 11 8 1 12 10 6 5 3 4
The replicate function is used in a lot of simulation code.
Is there a function in R which switches the first element with the last one in a vector? I have a for loop which need that reordering. From:
months = seq(1:12)
[1] 1 2 3 4 5 6 7 8 9 10 11 12
I would like to have:
[1] 12 1 2 3 4 5 6 7 8 9 10 11
and then again:
[1] 11 12 1 2 3 4 5 6 7 8 9 10
...
until the 12th position.
If you need a matrix output
cbind(c(months),embed(c(months, months), 12)[-13,-12])
# [,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12]
# [1,] 1 12 11 10 9 8 7 6 5 4 3 2
# [2,] 2 1 12 11 10 9 8 7 6 5 4 3
# [3,] 3 2 1 12 11 10 9 8 7 6 5 4
# [4,] 4 3 2 1 12 11 10 9 8 7 6 5
# [5,] 5 4 3 2 1 12 11 10 9 8 7 6
# [6,] 6 5 4 3 2 1 12 11 10 9 8 7
# [7,] 7 6 5 4 3 2 1 12 11 10 9 8
# [8,] 8 7 6 5 4 3 2 1 12 11 10 9
# [9,] 9 8 7 6 5 4 3 2 1 12 11 10
#[10,] 10 9 8 7 6 5 4 3 2 1 12 11
#[11,] 11 10 9 8 7 6 5 4 3 2 1 12
#[12,] 12 11 10 9 8 7 6 5 4 3 2 1
Or another approached suggested by #Marat Talipov
z <- length(months)
i <- rep(seq(z),z) + rep(seq(z),each=z) - 1
matrix(months[ifelse(i>z,i-z,i)],ncol=z)
I'm afraid that you have to come up with a home-made function, something like this one:
rotate <- function(v,i=1) {
i <- i %% length(v)
if (i==0) return(v)
v[c(seq(i+1,length(v)),seq(i))]
}
Couple of examples:
v <- seq(12)
rotate(v,1)
# [1] 2 3 4 5 6 7 8 9 10 11 12 1
rotate(v,-1)
# [1] 12 1 2 3 4 5 6 7 8 9 10 11
You can also use tail and head functions:
x = c(tail(x,n), head(x,-n))
and modify n to rotate n times
The permute package can do this for you:
ap <- allPerms(length(months),
control = how(within = Within(type = "series"),
observed = TRUE))
ap[rev(seq_len(nrow(ap))), ]
(because of the way allPerms() does its work, we need to reverse the order of the rows, which is what the last line does.)
This gives:
> ap[rev(seq_len(nrow(ap))), ]
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10] [,11] [,12]
[1,] 1 2 3 4 5 6 7 8 9 10 11 12
[2,] 12 1 2 3 4 5 6 7 8 9 10 11
[3,] 11 12 1 2 3 4 5 6 7 8 9 10
[4,] 10 11 12 1 2 3 4 5 6 7 8 9
[5,] 9 10 11 12 1 2 3 4 5 6 7 8
[6,] 8 9 10 11 12 1 2 3 4 5 6 7
[7,] 7 8 9 10 11 12 1 2 3 4 5 6
[8,] 6 7 8 9 10 11 12 1 2 3 4 5
[9,] 5 6 7 8 9 10 11 12 1 2 3 4
[10,] 4 5 6 7 8 9 10 11 12 1 2 3
[11,] 3 4 5 6 7 8 9 10 11 12 1 2
[12,] 2 3 4 5 6 7 8 9 10 11 12 1
Technically this only works because months is the vector 1:12 and allPerms() returns a permutation matrix of the indices of the thing you want permuted. For different inputs, use ap to index into the thing you want to permute
perms <- ap
perms[] <- months[ap[rev(seq_len(nrow(ap))), ]]
perms
I tried to solve a problem programmatically, but was unsuccessful. Here are some of the the examples to generate
cbind(1:12,13:2,3:14,15:4,5:16,17:6,7:18,19:8,9:20,21:10)
cbind(1:11,12:2,3:13,14:4,5:15,16:6,7:17,18:8,9:19)
Looking for a general solution for this problem. If the number of columns and rows are given, the program should output the matrix. Thanks.
You can do it fairly easily using apply. Just generate the matrix columns counting y steps up from the column number, and reverse the even columns;
> x=5; y=7
> apply(t(1:x), 2, function(x) if(x%%2) x-1+1:y else x-1+y:1)
[,1] [,2] [,3] [,4] [,5]
[1,] 1 8 3 10 5
[2,] 2 7 4 9 6
[3,] 3 6 5 8 7
[4,] 4 5 6 7 8
[5,] 5 4 7 6 9
[6,] 6 3 8 5 10
[7,] 7 2 9 4 11
m = embed(1:19, 9)
m[, seq(2, ncol(m), by=2)] = m[nrow(m):1, seq(2, ncol(m), by=2)]
m[,ncol(m):1]
produces
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9]
[1,] 1 12 3 14 5 16 7 18 9
[2,] 2 11 4 13 6 15 8 17 10
[3,] 3 10 5 12 7 14 9 16 11
[4,] 4 9 6 11 8 13 10 15 12
[5,] 5 8 7 10 9 12 11 14 13
[6,] 6 7 8 9 10 11 12 13 14
[7,] 7 6 9 8 11 10 13 12 15
[8,] 8 5 10 7 12 9 14 11 16
[9,] 9 4 11 6 13 8 15 10 17
[10,] 10 3 12 5 14 7 16 9 18
[11,] 11 2 13 4 15 6 17 8 19
Is there any function in R to find most frequently occuring element in matrix??I Have a matrix containing image pixels.I want to find which image pixel occur most frequently in the image matrix.I dont want to use the for loops since it would be very time taking to iterate over all the pixels of an image.
Set up some test data.
> (image = matrix(sample(1:10, 100, replace = TRUE), nrow = 10))
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 4 4 2 7 2 2 3 8 2 5
[2,] 7 3 2 6 6 5 7 8 1 3
[3,] 7 5 7 9 4 9 4 8 2 7
[4,] 5 3 4 2 1 5 9 10 9 5
[5,] 9 10 7 2 7 4 9 1 1 9
[6,] 2 3 5 1 2 8 1 5 9 4
[7,] 5 4 10 5 9 10 1 6 1 10
[8,] 6 3 9 7 1 1 9 2 1 7
[9,] 5 9 4 8 9 9 5 10 5 4
[10,] 10 1 4 7 3 2 3 5 4 5
Do it manually.
> table(image)
image
1 2 3 4 5 6 7 8 9 10
12 12 8 12 15 4 11 5 14 7
Here we can see that the value 5 appeared most often (15 times). To get the same results programmatically:
> which.max(table(image))
5
5
Get mode (or majority value) in 1 line of code
using set.seed to generate same random matrix
> set.seed(123)
> image = matrix(sample(1:10, 100, replace = TRUE), nrow = 10)
> image
[,1] [,2] [,3] [,4] [,5] [,6] [,7] [,8] [,9] [,10]
[1,] 3 10 9 10 2 1 7 8 3 2
[2,] 8 5 7 10 5 5 1 7 7 7
[3,] 5 7 7 7 5 8 4 8 5 4
[4,] 9 6 10 8 4 2 3 1 8 7
[5,] 10 2 7 1 2 6 9 5 2 4
[6,] 1 9 8 5 2 3 5 3 5 2
[7,] 6 3 6 8 3 2 9 4 10 8
[8,] 9 1 6 3 5 8 9 7 9 1
[9,] 6 4 3 4 3 9 8 4 9 5
[10,] 5 10 2 3 9 4 5 2 2 6)
Mode value of matrix (if tie, its gives minimum tie value)
> names(which.max(table(image)))
[1] "5"
I do not know any function to do that directly but you can use these functions:
sort(table(as.vector(Matrix))