I am looking for a best practice to store multiple vector results of an evaluation performed at several different values. Currently, my working code does this:
q <- 55
value <- c(0.95, 0.99, 0.995)
a <- rep(0,q) # Just initialize the vector
b <- rep(0,q) # Just initialize the vector
for(j in 1:length(value)){
for(i in 1:q){
a[i]<-rnorm(1, i, value[j]) # just as an example function
b[i]<-rnorm(1, i, value[j]) # just as an example function
}
df[j] <- data.frame(a,b)
}
I am trying to find the best way to store individual a and b for each value level
To be able to iterate through the variable "value" later for graphing
To have the value of the variable "value" and/or a description of it available
I'm not exactly sure what you're trying to do, so let me know if this is what you're looking for.
q = 55
value <- c(sd95=0.95, sd99=0.99, sd995=0.995)
a = sapply(value, function(v) {
rnorm(q, 1:q, v)
})
In the code above, we avoid the inner loop by vectorizing. For example, rnorm(55, 1:55, 0.95) will give you 55 random normal deviates, the first drawn from a distribution with mean=1, the second from a distribution with mean=2, etc. Also, you don't need to initialize a.
sapply takes the place of the outer loop. It applies a function to each value in value and returns the three vectors of random draws as the data frame a. I've added names to the values in value and sapply uses those as the column names in the resulting data frame a. (It would be more standard to make value a list, rather than a vector with named elements. You can do that with value <- list(sd95=0.95, sd99=0.99, sd995=0.995) and the code will otherwise run the same.)
You can create multiple data frames and store them in a list as follows:
q <- list(a=10, b=20)
value <- list(sd95=0.95, sd99=0.99, sd995=0.995)
df.list = sapply(q, function(i) {
sapply(value, function(v) {
rnorm(i, 1:i, v)
})
})
This time we have two different values for q and we wrap the sapply code from above inside another call to sapply. The inner sapply does the same thing as before, but now it gets the value of q from the outer sapply (using the dummy variable i). We're creating two data frames, one called a and the other called b. a has 10 rows and b has 20 (due to the values we set in q). Both data frames are stored in a list called df.list.
Related
I am not very experienced with R, and have been struggling for days to repeat a string of code to fill a data matrix. My instinct is to create a for loop.
I am a biology student working on colour differences between sets of images, making use of the R package colordistance. The relevant data has been loaded in R as a list of 8x4 matrices (each matrix describes the colours in one image). Five images make up one set and there are 100 sets in total. Each set is identified by a number (not 1-100, it's an interrupted sequence, but I have stored the sequence of numbers in a vector called 'numberlist'). I have written the code to extract the desired data in the right format for the first set, and it is as follows;
## extract the list of matrices belonging to the first set (A3) from the the full list
A3<-histlist[grep('^3',names(histlist))]
## create a colour distance matrix (cdm), ie a pairwise comparison of "similarity" between the five matrices stored in A3
cdm3<-colordistance::getColorDistanceMatrix(A3, method="emd", plotting=FALSE)
## convert to data frame to fix row names
cdm3df<-as.data.frame(cdm3)
## remove column names
names(cdm3df)<-NULL
## return elements in the first row and column 2-5 only (retains row names).
cdm3filtered<-cdm3df[1,2:5]
Now I want to replace "3" in the code above with each number in 'numberlist' (not sure whether they should be as.factor or as.numeric). I've had many attempts starting with for (i in numberlist) {...} but with no successful output. To me it makes sense to store the output from the loop in a storage matrix; matrix(nrow=100,ncol=4) but I am very much stuck, and unable to populate my storage matrix row by row by iterating the code above...
Any help would be greatly appreciated!
Updates
What I want the outputs of the loop to to look like (+ appended in the storage matrix);
> cdm17filtered
17clr 0.09246918 0.1176651 0.1220622 0.1323586
This is my attempt:
for (i in numberlist$X) {
A[i] <- histlist[grep(paste0('^',i),names(histlist))]
cdm[i] <- colordistance::getColorDistanceMatrix(A[i], method="emd", plotting=FALSE)
cdm[i]df <- as.data.frame(cdm[i])
cdm[i]filtered <- cdm[i]df[1,2:5]
print(A[i]) # *insert in n'th column of storage matrix
}
The above is not working, and I'm missing the last bit needed to store the outputs of the loop in the storage matrix. (I was advised against using rbind to populate the storage matrix because it is slow..)
In your attempt, you use invalid R names with non-alphanumeric characters not escaped, cdm[i]df and cdm[i]filtered. It seems you intend to index from a larger container like a list of objects.
To properly generalize your process for all items in numberlist, adjust your ^3 setup. Specifically, build empty lists and in loop iteratively assign by index [i]:
# INITIALIZE LISTS (SAME LENGTH AS numberlist)
A <- vector(mode="list", length = length(numberlist))
cdm_matrices <- vector(mode="list", length = length(numberlist))
cdm_dfs <- vector(mode="list", length = length(numberlist))
cdm_filtered_dfs <- vector(mode="list", length = length(numberlist))
# POPULATE LISTS
for (i in numberlist$X) {
## extract the list of matrices belonging to the first set
A[i] <- histlist[grep(paste0('^', i), names(histlist))]
## create a colour distance matrix (cdm)
cdm_matrices[i] <- colordistance::getColorDistanceMatrix(A[i], method="emd", plotting=FALSE)
## convert to data frame to fix row names and remove column names
cdm_dfs[i] <- setNames(as.data.frame(cdm_matrices[i]), NULL)
## return elements in the first row and column 2-5 only (retains row names).
cdm_filtered_dfs[i] <- cdm_dfs[i][1,2:5]
}
Alternatively, if you only need the last object, cdm_filtered_df returned, use lapply where you do not need to use or index lists and all objects are local in scope of function (i.e., never saved in global environment):
cdm_build <- function(i) {
A <- histlist[grep(paste0('^', i), names(histlist))]
cdm <- colordistance::getColorDistanceMatrix(A, method="emd", plotting=FALSE)
cdm_df <- setNames(as.data.frame(cdm), NULL)
cdm_filtered_df <- cdm_df[1,2:5]
return(cdm_filtered_df) # REDUNDANT AS LAST LINE IS RETURNED BY DEFAULT
}
# LIST OF FILTERED CDM DATA FRAMES
cdm_filtered_dfs <- lapply(numberlist, cdm_build)
Finally, with either solution above, should you want to build a singular data frame, run rbind in a do.call():
cdm_final_df <- do.call(rbind, cdm_filtered_dfs)
I want to write a function that dynamically uses different correlation methods depending on the scale of measure of the feature (continuous, dichotomous, ordinal). The label is always continuous. My idea was to use the apply() function, so iterate over every feature (aka column), check it's scale of measure (numeric, factor with two levels, factor with more than two levels) and then use the appropriate correlation function. Unfortunately my code seems to convert every feature into a character vector and as consequence the condition in the if statement is always false for every column. I don't know why my code is doing this. How can I prevent my code from converting my features to character vectors?
set.seed(42)
foo <- sample(c("x", "y"), 200, replace = T, prob = c(0.7, 0.3))
bar <- sample(c(1,2,3,4,5),200,replace = T,prob=c(0.5,0.05,0.1,0.1,0.25))
y <- sample(c(1,2,3,4,5),200,replace = T,prob=c(0.25,0.1,0.1,0.05,0.5))
data <- data.frame(foo,bar,y)
features <- data[, !names(data) %in% 'y']
dyn.corr <- function(x,y){
# print out structure of every column
print(str(x))
# if feature is numeric and has more than two outcomes use corr.test
if(is.numeric(x) & length(unique(x))>2){
result <- corr.test(x,y)[['r']]
} else {
result <- "else"
}
}
result <- apply(features,2,dyn.corr,y)
apply is built for matrices. When you apply to a data frame, the first thing that happens is coercing your data frame to a matrix. A matrix can only have one data type, so all columns of your data are converted to the most general type among them when this happens.
Use sapply or lapply to work with columns of a data frame.
This should work fine (I tried to test, but I don't know what package to load to get the corr.test function.)
result <- sapply(features, dyn.corr, income)
I am attempting to write a function which accepts a dataframe, and then generates subset dataframes within a for() loop. As a first step, I tried the following:
dfcreator<-function(X,Z){
for(i in 1:Z){
df<-subset(X,Stratum==Z) #build dataframe from observations where index=value
assign(paste0("pop", Z),df) #name dataframe
}
}
This however does not save anything in to memory, and when I try to specify a return() I am still not getting what I need. For reference, I am using the
Sweden data set (which is native to RStudio).
EDIT Per Melissa's Advice!
I tried to implement the following code:
sampler <- function(df, n,...) {
return(df[sample(nrow(df),n),])
}
sample_list<-map2(data_list, stratumSizeVec, sampler)
where stratumSizeVec is a 1X7 df and data_list is a list of seven dfs. When I do this, I get seven samples in sample list all of the same size equal to stratumSizeVec[1]. Why is map2 not inputting the in the following manner
sampler(data_list$pop0,stratumSizeVec[1])
sampler(data_list$pop1,stratumSizeVec[2])
...
sampler(data_list$pop6,stratumSizeVec[7])
Furthermore, is there a way to "nest" the map2 function within lapply?
I'm confused as to why you never actually utilize i anywhere in your loop. It looks like you're creating Z copies of the data set where Stratum == Z - is that what you are after?
as for your code, I would use the following:
data_list <- split(df, df$Stratum)
names(data_list) <- paste0("pop", sort(unique(df$Stratum)))
This doesn't define a function, we are calling base-R function (namely split) which splits up a data frame based on some vector (here, we use df$Stratum). The result is a list of data frames, each with a single value of Stratum.
Random sampling from rows
sampled_data <- lapply(data_list, function(df, n,...) { # n is the number of rows to take, the dots let you send other information to the `sample` function.
df[sample(nrow(df), n, ...),]
},
n = 5,
replace = FALSE # this is default, but the purpose of using the ... notation is to allow this (and any other options in the `sample` function) to be changed.
)
You can also define the function separately:
sampler <- function(df, n,...) {
df[sample(nrow(df), n, ...),]
}
sampled_data <- lapply(data_list, sampler, n = 10) # replace 10 with however many samples you want.
purrr:map2 method
As defined, the sampler function does not need to be modified, each element of the first list (data_list) is put into the first argument of sampler, and the corresponding element of the 2nd "list" (sampleSizeVec) is put into the 2nd argument.
library(purrr)
map2(data_list, sampleSizeVec, sampler, replace = FALSE) # replace = FALSE not needed, there as an example only.
I have this function
fun_2 <- function(x,L_inf,y){
L_inf-((L_inf-y)/
(exp(-B*(x-c(1:12))/12)))
}
B <- 0.5
The problem is similar this previous post R: How to create a loop for, for a range of data in a function?
In this case i would to apply the fun_2 for this two range of data:
L_inf_range <- seq(17,20,by=0.1) #31 values
y_range <- seq(4,22,by=0.1) # 19 values
I tried with:
sapply(L_inf_range, function(L) fun_2(12, L_inf=L,y_range))
but is not the expected output. My expected output is a new matrix genereted by sapply(or other kind of function) where the fun_2 is apply for all the value in L_inf_range and each time for all value of y_range.
Substantially it will be a matrix where fun_2 is apply for each values of L_inf_range(31 values) minus y_range (L_inf-y in fun_2) each time.
You could also run sapply twice changing the order of your arguments to make it, a little easier.
fun_2_mod <- function(y, L_inf, x = X){ fun_2(x, L_inf, y)}
sapply(y_range, function(Y){sapply(L_inf_range, fun_2_mod, y = Y)})
You can bind the resulting list of lists together into a data frame.
I am trying to populate a data frame from within a for loop in R. The names of the columns are generated dynamically within the loop and the value of some of the loop variables is used as the values while populating the data frame. For instance the name of the current column could be some variable name as a string in the loop, and the column can take the value of the current iterator as its value in the data frame.
I tried to create an empty data frame outside the loop, like this
d = data.frame()
But I cant really do anything with it, the moment I try to populate it, I run into an error
d[1] = c(1,2)
Error in `[<-.data.frame`(`*tmp*`, 1, value = c(1, 2)) :
replacement has 2 rows, data has 0
What may be a good way to achieve what I am looking to do. Please let me know if I wasnt clear.
It is often preferable to avoid loops and use vectorized functions. If that is not possible there are two approaches:
Preallocate your data.frame. This is not recommended because indexing is slow for data.frames.
Use another data structure in the loop and transform into a data.frame afterwards. A list is very useful here.
Example to illustrate the general approach:
mylist <- list() #create an empty list
for (i in 1:5) {
vec <- numeric(5) #preallocate a numeric vector
for (j in 1:5) { #fill the vector
vec[j] <- i^j
}
mylist[[i]] <- vec #put all vectors in the list
}
df <- do.call("rbind",mylist) #combine all vectors into a matrix
In this example it is not necessary to use a list, you could preallocate a matrix. However, if you do not know how many iterations your loop will need, you should use a list.
Finally here is a vectorized alternative to the example loop:
outer(1:5,1:5,function(i,j) i^j)
As you see it's simpler and also more efficient.
You could do it like this:
iterations = 10
variables = 2
output <- matrix(ncol=variables, nrow=iterations)
for(i in 1:iterations){
output[i,] <- runif(2)
}
output
and then turn it into a data.frame
output <- data.frame(output)
class(output)
what this does:
create a matrix with rows and columns according to the expected growth
insert 2 random numbers into the matrix
convert this into a dataframe after the loop has finished.
this works too.
df = NULL
for (k in 1:10)
{
x = 1
y = 2
z = 3
df = rbind(df, data.frame(x,y,z))
}
output will look like this
df #enter
x y z #col names
1 2 3
Thanks Notable1, works for me with the tidytextr
Create a dataframe with the name of files in one column and content in other.
diretorio <- "D:/base"
arquivos <- list.files(diretorio, pattern = "*.PDF")
quantidade <- length(arquivos)
#
df = NULL
for (k in 1:quantidade) {
nome = arquivos[k]
print(nome)
Sys.sleep(1)
dados = read_pdf(arquivos[k],ocr = T)
print(dados)
Sys.sleep(1)
df = rbind(df, data.frame(nome,dados))
Sys.sleep(1)
}
Encoding(df$text) <- "UTF-8"
I had a case in where I was needing to use a data frame within a for loop function. In this case, it was the "efficient", however, keep in mind that the database was small and the iterations in the loop were very simple. But maybe the code could be useful for some one with similar conditions.
The for loop purpose was to use the raster extract function along five locations (i.e. 5 Tokio, New York, Sau Paulo, Seul & Mexico city) and each location had their respective raster grids. I had a spatial point database with more than 1000 observations allocated within the 5 different locations and I was needing to extract information from 10 different raster grids (two grids per location). Also, for the subsequent analysis, I was not only needing the raster values but also the unique ID for each observations.
After preparing the spatial data, which included the following tasks:
Import points shapefile with the readOGR function (rgdap package)
Import raster files with the raster function (raster package)
Stack grids from the same location into one file, with the function stack (raster package)
Here the for loop code with the use of a data frame:
1. Add stacked rasters per location into a list
raslist <- list(LOC1,LOC2,LOC3,LOC4,LOC5)
2. Create an empty dataframe, this will be the output file
TB <- data.frame(VAR1=double(),VAR2=double(),ID=character())
3. Set up for loop function
L1 <- seq(1,5,1) # the location ID is a numeric variable with values from 1 to 5
for (i in 1:length(L1)) {
dat=subset(points,LOCATION==i) # select corresponding points for location [i]
t=data.frame(extract(raslist[[i]],dat),dat$ID) # run extract function with points & raster stack for location [i]
names(t)=c("VAR1","VAR2","ID")
TB=rbind(TB,t)
}
was looking for the same and the following may be useful as well.
a <- vector("list", 1)
for(i in 1:3){a[[i]] <- data.frame(x= rnorm(2), y= runif(2))}
a
rbind(a[[1]], a[[2]], a[[3]])