Loop through df and create new df in R - r

I have a df (10 rows, 15 columns)
df<-data.frame(replicate(15,sample(0:1,10,rep=TRUE)))
I want to loop over each column, do something to each row and create a new df with the answer.
I actually want to do a linear regression on each column. I get back a list for each column. For example I have a second df with what I want to put into the lm. df2<-data.frame(replicate(2,sample(0:1,10,rep=TRUE)))
I then want to do something like:
new_df <- data.frame()
for (i in 1:ncol(df)){
j<-lm(df[,i] ~ df2$X1 + df2$X2)
temp_df<-j$residuals
new_df[,i]<-cbind(new_df,temp_df)
}
I get the error:
Error in data.frame(..., check.names = FALSE) : arguments imply
differing number of rows: 0, 8
I have checked other similar posts but they always seem to involve a function or something similarly complex for a newbie like me. Please help

This can be done without loops but for your understanding, using loops we can do
new_df <- df
for (i in names(df)) {
j<-lm(df[,i] ~ df$X1 + df$X2)
new_df[i] <- j$residuals
}
You are initialising an empty dataframe with 0 rows and 0 columns initially as new_df and hence when you are trying to assign the value to it, it gives you an error. Instead of that assign original df to new_df as they both are going to share the same structure and then use the above.

Update
Based on the new example
lst1 <- lapply(names(df), function(nm) {dat <- cbind(df[nm], df2[c('X1', 'X2')])
lm(paste0(nm, "~ X1 + X2"), data = dat)$residuals})
out <- setNames(data.frame(lst1), names(df))
Also, this doesn't need any loop
out2 <- lm(as.matrix(df) ~ X1 + X2, data = cbind(df, df2))$residuals
Old
We can do this easily without any loop
new_df <- df + 10
---
If we need a loop, it can be done with `lapply`
new_df <- df
new_df[] <- lapply(df, function(x) x + 10)
---
Or with a `for` loop
lst1 <- vector('list', ncol(df))
for(i in seq_along(df)) lst1[[i]] <- df[, i] + 10
new_df <- as.data.frame(lst1)
data
set.seed(24)
df <- data.frame(replicate(15,sample(0:1,10,rep=TRUE)))
df2 <- data.frame(replicate(2,sample(0:1,10,rep=TRUE)))

I would do as suggested by akrun. But if you do need (or want) to loop for some reasons you can use:
df<-data.frame(replicate(15,sample(0:1,10,rep=TRUE)))
new_df <- data.frame(replicate(15, rep(NA, 10)))
for (i in 1:ncol(df)){
new_df[ ,i] <- df[ , i] + 10
}

Related

ANOVA repeated measure on multiple data frames r

I have hundreds of data frames. I need to perform ANOVA RM tests on each of these data frames. The output should be one single data frame with the mean of each p-value.
I tried:
#crate dataframes
df1 <- data.frame(replicate(16,sample(-10:10,10,rep=TRUE)))
df2 <- data.frame(replicate(16,sample(-10:10,10,rep=TRUE)))
df3 <- data.frame(replicate(16,sample(-10:10,10,rep=TRUE)))
Group <- c(rep("A",8),rep("B",8))
Time <- c(rep("before",4),rep("after",4),rep("before",4),rep("after",4))
Name <- rep(rep(1:4, 4))
conds <- data.frame(Name,Time,Group)
#create list
list <- list(df1,df2,df3)
#for loop ANOVA repeated measures
for ( i in list){
data <- cbind(conds,i)
t=NULL
name <- colnames(data)[4:ncol(data)]
for(i in 4:ncol(data)) { z <- aov(data[,i] ~ Group*Time+Error(Name/(Group*Time)), data=data)
sz <- as.list(summary(z))
t <- as.data.frame(c(t,sz[4]$`Error: Name:Group:Time`[[1]]$`Pr(>F)`[1]))
t
}
}
mean(t)
R as a vectorized language is designed to avoid for loops where possible. You could do an sapply approach.
When you list your data frames use names like df1=, which later helps in the result on which of them were done calculations.
(And don't use list as object name since you'll get confused because there is also a list function. Also data, df and friends are "bad" names, you may always check, using e.g. ?list if the name is already occupied.)
list1 <- list(df1=df1, df2=df2, df3=df3)
res <- sapply(list1, function(x) {
dat <- cbind(conds, x)
sapply(dat[-(1:3)], function(y) {
z <- aov(y ~ Group*Time + Error(Name/(Group*Time)), data=dat)
sz <- summary(z)
p <- sz$`Error: Name:Group:Time`[[1]][1, 5]
p
})
})
From the resulting matrix we take the column means.
colMeans(res)
# df1 df2 df3
# 0.4487419 0.4806528 0.4847789
Data:
set.seed(42)
df1 <- data.frame(replicate(16,sample(-10:10,16,rep=TRUE)))
df2 <- data.frame(replicate(16,sample(-10:10,16,rep=TRUE)))
df3 <- data.frame(replicate(16,sample(-10:10,16,rep=TRUE)))
conds <- data.frame(Name=c(rep("A",8),rep("B",8)),
Time=c(rep("before",4),rep("after",4),
rep("before",4),rep("after",4)),
Group=rep(1:4, 4))

Apply a user defined function to a list of data frames

I have a series of data frames structured similarly to this:
df <- data.frame(x = c('notes','year',1995:2005), y = c(NA,'value',11:21))
df2 <- data.frame(x = c('notes','year',1995:2005), y = c(NA,'value',50:60))
In order to clean them I wrote a user defined function with a set of cleaning steps:
clean <- function(df){
colnames(df) <- df[2,]
df <- df[grep('^[0-9]{4}', df$year),]
return(df)
}
I'd now like to put my data frames in a list:
df_list <- list(df,df2)
and clean them all at once. I tried
lapply(df_list, clean)
and
for(df in df_list){
clean(df)
}
But with both methods I get the error:
Error in df[2, ] : incorrect number of dimensions
What's causing this error and how can I fix it? Is my approach to this problem wrong?
You are close, but there is one problem in code. Since you have text in your dataframe's columns, the columns are created as factors and not characters. Thus your column naming does not provide the expected result.
#need to specify strings to factors as false
df <- data.frame(x = c('notes','year',1995:2005), y = c(NA,'value',11:21), stringsAsFactors = FALSE)
df2 <- data.frame(x = c('notes','year',1995:2005), y = c(NA,'value',50:60), stringsAsFactors = FALSE)
clean <- function(df){
colnames(df) <- df[2,]
#need to specify the column to select the rows
df <- df[grep('^[0-9]{4}', df$year),]
#convert the columns to numeric values
df[, 1:ncol(df)] <- apply(df[, 1:ncol(df)], 2, as.numeric)
return(df)
}
df_list <- list(df,df2)
lapply(df_list, clean)

Forcing the use of a for loop with group_by and mutate()

I have a list of data frames (generated by the permutation order of an initial dataframe) to which I would like to apply complicated calculus using group_by_at() and mutate(). It works well with a single data frame but fail using a for loop since mutate requires the name of the dataframe and some of my calculus as well. So I thought, well, let's create a list of different dataframes all having the same name and loop over the initial sequence of names. Unfortunately the trick does not work and I get the following message:
Error: object of type 'closure' is not subsettable.
Here is the self contained example showing all my steps. I think the problem comes from mutate. So, how could I force the use of for loop with mutate?
data <- read.table(text = 'obs gender ageclass weight year subdata income
1 F 1 10 yearA sub1 1000
2 M 2 25 yearA sub1 1200
3 M 2 5 yearB sub2 1400
4 M 1 11 yearB sub1 1350',
header = TRUE)
library(dplyr)
library(GiniWegNeg)
dataA <- select(data, gender, ageclass)
dataB <- select(data, -gender, -ageclass)
rm(data)
# Generate permutation of indexes based on the number of column in dataA
library(combinat)
index <- permn(ncol(dataA))
# Attach dataA to the previous list of index
res <- lapply(index, function(x) dataA[x])
# name my list keeping track of permutation order in dataframe name
names(res) <- unlist(lapply(res,function(x) sprintf('data%s',paste0(toupper(substr(colnames(x),1,1)),collapse = ''))))
# Create a list containing the name of each data.frame name
NameList <- unlist(lapply(res,function(x) sprintf('data%s',paste0(toupper(substr(colnames(x),1,1)),collapse = ''))))
# Define as N the number of columns/permutation/dataframes
N <- length(res)
# Merge res and dataB for all permutation of dataframes
res <- lapply(res,function(x) cbind(x,dataB))
# Change the name of res so that all data frames are named data
names(res) <- rep("data", N)
# APPLY FOR LOOP TO ALL DATAFRAMES
for (j in NameList){
runCalc <- function(data, y){
data <- data %>%
group_by_at(1) %>%
mutate(Income_1 = weighted.mean(income, weight))
data <- data %>%
group_by_at(2) %>%
mutate(Income_2 = weighted.mean(income, weight))
gini <- c(Gini_RSV(data$Income_1, data$weight), Gini_RSV(data$Income_2,data$weight))
Gini <- data.frame(gini)
colnames(Gini) <- c("Income_1","Income_2")
rownames(Gini) <- c(paste0("Gini_", y))
return(Gini)
}
runOtherCalc <- function(df, y){
Contrib <- (1/5) * df$Income_1 + df$Income_2
Contrib <- data.frame(Contrib)
colnames(Contrib) <- c("myresult")
rownames(Contrib) <- c(paste0("Contrib_", y)
return(Contrib)
}
# Run runCalc over dataframe data by year
df1_List <- lapply(unique(data$year), function(i) {
byperiod <- subset(data, year == i)
runCalc(byperiod, i)
})
# runCalc returns df which then passes to runOtherCalc, again by year
df1_OtherList <- lapply(unique(data$year), function(i)
byperiod <- subset(data, year == i)
df <- runCalc(byperiod, i)
runOtherCalc(df, i)
})
# Run runCalc over dataframe data by subdata
df2_List <- lapply(unique(data$subdata), function(i) {
byperiod <- subset(data, subdata == i)
runCalc(bysubdata, i)
})
# runCalc returns df which then passes to runOtherCalc, again by subdata
df2_OtherList <- lapply(unique(data$subdata), function(i)
bysubdata <- subset(data, subdata == i)
df <- runCalc(bysubdata, i)
runOtherCalc(df, i)
})
# Return all results in separate frames, then append by row in 2 frames
Gini_df1 <- do.call(rbind, df1_List)
Contrib_df1 <- do.call(rbind,df1_OtherList)
Gini_df2 <- do.call(rbind, df1_List)
Contrib_df2 <- do.call(rbind,df1_OtherList)
Gini <- rbind(Gini_df1, Gini_df2)
Contrib <- rbind(Contrib_df1, Contrib_df2)
}
Admittedly, the R error you receive below is a bit cryptic but usually it means you are running an operation on an object that does not exist.
Error: object of type 'closure' is not subsettable.
Specifically, it comes with your lapply call as data is not defined anywhere globally (only within the runCalc method) and as above you remove it with rm(data).
dfList <- lapply(unique(data$year), function(i) {
byperiod <- subset(data, year == i)
runCalc(byperiod, i)
})
By, the way the use of lapply...unique...subset can be replaced with the underused grouping base R function, by().
Gathering from your text and code, I believe you intend to run a year grouping on each dataframe of your list, res. Then consider two by calls, wrapped in a larger function that receives as a parameter a dataframe, df. Then run lapply across all items of list to return a new list of nested dataframe pairs.
# SECONDARY FUNCTIONS
runCalc <- function(data) {
data <- data %>%
group_by_at(1) %>%
mutate(Income_1 = weighted.mean(income, weight))
data <- data %>%
group_by_at(2) %>%
mutate(Income_2 = weighted.mean(income, weight))
Gini <- data.frame(
year = data$year[[1]],
Income_1 = unname(Gini_RSV(data$Income_1, data$weight)),
Income_2 = unname(Gini_RSV(data$Income_2, data$weight)),
row.names = paste0("Gini_", data$year[[1]])
)
return(Gini)
}
runOtherCalc <- function(df){
Contrib <- data.frame(
myresult = (1/5) * df$Income_1 + df$Income_2,
row.names = paste0("Contrib_", df$year[[1]])
)
return(Contrib)
}
# PRIMARY FUNCTION
runDfOperations <- function(df) {
gList <- by(df, df$year, runCalc)
gTmp <- do.call(rbind, gList)
cList <- by(gTmp, gTmp$year, runOtherCalc)
cTmp <- do.call(rbind, cList)
gtmp$year <- NULL
return(list(gTmp, cTmp))
}
# RETURNS NESTED LIST OF TWO DFs FOR EACH ORIGINAL DF
new_res <- lapply(res, runDfOperations)
# SEPARATE LISTS IF NEEDED (EQUAL LENGTH)
Gini <- lapply(new_res, "[[", 1)
Contrib <- lapply(new_res, "[[", 2)

R data frame manipulation/transformation using ldply

I am trying to write some R code which will take the iris dataset and do a log transform of the numeric columns as per some criterion, say if skewness > 0.2. I have tried to use ldply, but it doesn't quite give me the output I want. It is giving me a transposed data frame, the variable names are missing and the non-numeric column entries are messed up.
Before posting this question I searched and found the following related topics but didn't quite meet what exactly I was looking for
Selecting only numeric columns from a data frame
extract only numeric columns from data frame
data
Below is the code. Appreciate the help!
data(iris)
df <- iris
df <- ldply(names(df), function(x)
{
if (class(df[[x]])=="numeric")
{
tmp <- df[[x]][!is.na(df[[x]])]
if (abs(skewness(tmp)) > 0.2)
{
df[[x]] <- log10( 1 + df[[x]] )
}
else df[[x]] <- df[[x]]
}
else df[[x]] <- df[[x]]
#df[[x]] <- data.frame(df[[x]])
#df2 <- cbind(df2, df[[x]])
#return(NULL)
}
)
Try with lapply:
#Skewness package
library(e1071)
lapply(iris, function(x) {
if(is.numeric(x)){
if(abs(skewness(x, na.rm = T))>0.2){
log10(1 + x)} else x
}
else x
})
We can use lapply
library(e1071)
lapply(iris, function(x) if(is.numeric(x) & abs(skewness(x, na.rm = TRUE)) > 0.2)
log10(1+x) else x)
We can also loop by the columns of interest after creating a logical index
i1 <- sapply(iris, is.numeric)
i2 <- sapply(iris[i1], function(x) abs(skewness(x, na.rm = TRUE)) > 0.2)
iris[i1][i2] <- lapply(iris[i1][i2], function(x) log10(1+x))

Subsetting data from R data frame using incremental variable names

I have an R data frame, df, with column names V1, V2, V3...V1000. I need to subset df by selecting every 20th column, that is, V1, V21, V41, V61 through end of columns.
I think this can be done using dplyr's select(df, num_range("V", val)), but am stumped how to iterate val through 1000 columns, stepping by 20.
Any suggestions?
Use the seq function with dplyr's select and num_range as below:
library(dplyr)
df <- as.data.frame(matrix(rnorm(3000), nrow = 3))
df %>% select(num_range("V", seq(1, 1000, by = 20)))
You can try,
df[seq(1, ncol(df), 20)]
u can use some function like this.Here nskip=20 as u want to skip 20 columns
FOO <- function(data, nSubsets, nskip)
{
outList <- vector("list", length = nSubsets)
totcol <- ncol(data)
for (i in seq_len(nSubsets))
{
colsToGrab<- seq(i, totcol, nSkip)
outList[[i]] <- data[,colsToGrab ]
}
return(outList)
}

Resources