I have a dataframe called covars with three ethnicities. How do I apply function Get_STATs so I can get the output for each ethnicity?
Right, now I am running it like this:
tt <- covars[covars$ETHNICITY == "HISPANIC",]
Get_STATs(tt)
tt <- covars[covars$ETHNICITY == "WHITE",]
Get_STATs(tt)
tt <- covars[covars$ETHNICITY == "ASIAN",]
Get_STATs(tt)
I tried to run it like this
aggregate(covars, by = list(covars$ETHNICITY), FUN = Get_STATs)
which generates error rror: $ operator is invalid for atomic vectors
aggregate runs on each column separately, we may need by
do.call(rbind, by(covars, covars$ETHNICITY, FUN = Get_STATs))
Or split into a list and loop over the list and apply the function
do.call(rbind, lapply(split(covars, covars$ETHNICITY), Get_STATs))
If we need the ETHNICITY names as well
lst1 <- split(covars, covars$ETHNICITY)
do.call(rbind, Map(cbind, ETHNICITY = names(lst1), lapply(lst1, Get_STATs)))
Depending on the Get_STATs function, you can use dplyr:
tt <-
covars %>%
group_by(ETHNICITY) %>%
Get_STATs()
Related
I have a list with dataframes:
df1 <- data.frame(id = seq(1:10), name = LETTERS[1:10])
df2 <- data.frame(id = seq(11:20), name = LETTERS[11:20])
mylist <- list(df1, df2)
I want to remove rows from each dataframe in the list based on a condition (in this case, the value stored in column id). I create an empty vector where I will store the ids:
ids_to_remove <- c()
Then I apply my function:
sapply(mylist, function(df) {
rows_above_th <- df[(df$id > 8),] # select the rows from each df above a threshold
a <- rows_above_th$id # obtain the ids of the rows above the threshold
ids_to_remove <- append(ids_to_remove, a) # append each id to the vector
},
simplify = T
)
However, with or without simplify = T, this returns a matrix, while my desired output (ids_to_remove) would be a vector containing the ids, like this:
ids_to_remove <- c(9,10,9,10)
Because lastly I would use it in this way on single dataframes:
for(i in 1:length(ids_to_remove)){
mylist[[1]] <- mylist[[1]] %>%
filter(!id == ids_to_remove[i])
}
And like this on the whole list (which is not working and I don´t get why):
i = 1
lapply(mylist,
function(df) {
for(i in 1:length(ids_to_remove)){
df <- df %>%
filter(!id == ids_to_remove[i])
i = i + 1
}
} )
I get the errors may be in the append part of the sapply and maybe in the indexing of the lapply. I played around a bit but couldn´t still find the errors (or a better way to do this).
EDIT: original data has 70 dataframes (in a list) for a total of 2 million rows
If you are using sapply/lapply you want to avoid trying to change the values of global variables. Instead, you should return the values you want. For example generate a vector if IDs to remove for each item in the list as a list
ids_to_remove <- lapply(mylist, function(df) {
rows_above_th <- df[(df$id > 8),] # select the rows from each df above a threshold
rows_above_th$id # obtain the ids of the rows above the threshold
})
And then you can use that list with your data list and mapply to iterate the two lists together
mapply(function(data, ids) {
data %>% dplyr::filter(!id %in% ids)
}, mylist, ids_to_remove, SIMPLIFY=FALSE)
Using base R
Map(\(x, y) subset(x, !id %in% y), mylist, ids_to_remove)
I have the following 3 data frames, each of which has columns with names. I want to combine them and retain the column names. When I use the patch I found for combining dataframes, it drops that name on any dataframes that don't have at least 2 columns. How can I retain the names?
x<-data.frame(mean(1:10))
names(x)[names(x) == 'mean.1.10.'] <- 'var.name'
y<-data.frame(1:4)
names(y)[names(y) == 'X1.4'] <- 'var.name2'
z<-data.frame(matrix(1:10,5,2))
names(z)[names(z) == 'X1'] <- 'var.name3'
names(z)[names(z) == 'X2'] <- 'var.name4'
list_datf <- list(x, y, z)
n_r <- seq_len(max(sapply(list_datf, nrow)))
NEW <- do.call(cbind, lapply(list_datf, `[`, n_r, ))
You need to include drop = FALSE in the indexing step so that the things you're binding together retain all of their dimensions. I couldn't figure out a way to do this by passing drop = FALSE as an extra argument to [, so I resorted to using an anonymous function instead.
NEW <- do.call(cbind, lapply(list_datf, function(x) x[n_r, , drop = FALSE]))
Alternatively, you could convert your components to tibbles, which (unlike data frames) never drop "unneeded" dimensions:
NEW <- do.call(cbind, lapply(list_datf, function(x) tibble::as_tibble(x)[n_r, ]))
If you want to go full tidyverse:
library(dplyr)
list_datf %>% purrr::map(~ tibble::as_tibble(.)[n_r, ]) %>% bind_cols()
I have a list of data frames and a character vector with strings. The number of dataframes and the number of strings in the chr are the same.
I'd like to populate a specific column in each dataframe in the list with the string at the corresponding position in the character vector
dfs<-list(mtcars[,1:4], iris[,1:4])
dfs <- lapply(dfs, function(x) transform(x, mycol=""))
z <- c("red", "blue")
As the final output I'd like
dfs[[1]]$mycol to be populated with red and
dfs[[2]]$mycol to be populated with blue
Conceptually, I think I need to do something like this:
dfs <- lapply(dfs, function(n) dfs[[n]]$mycol <- z[n]), but I get the error
Error in z[n] : invalid subscript type 'list'
The real data is a list of 97 elements
You can also try creating directly mycol with mapply():
#Data
dfs<-list(mtcars[,1:4], iris[,1:4])
z <- c("red", "blue")
#Code
L <- mapply(function(x,y) {x$mycol<-y;return(x)},x=dfs,y=z,SIMPLIFY = F)
This is what you're after
lapply(1:n, function(x) transform(dfs[x], mycol = z[x]))
When you want to perform an apply by passing an index on several objects, simply apply on 1:n then pass this as an argument to the different objects within the anonymous function.
EDIT transform only works with all objects contained in the same environment. So it throws the error object not found with the previous code because dfs exists the .GlobalEnv while x exists only in the function environments.
The below code works
lapply(c(1:n), function(x) {
toreplace <<- z[x] # forcing to parent envir using <<-
base::transform(dfs[x], mycol = toreplace)
})
We could use Map with transform in base R
Map(transform, dfs, mycol = z)
Or map2 from purrr
library(purrr)
library(dplyr)
map2(dfs, z, ~ .x %>%
mutate(mycol = .y))
Is it a way I can get the data info from global environment into a summary table?
For example, I have a lot of data set named TXXX in my global environment, like
I would like to table that looks like this
Is it possible to also get all the variable list for each data using programing?
it will looks like this:
Any way I can do that by programming? Thanks.
We can use mget to get all the objects that starts with 'T' followed by 3 digit number in to a list , then loo over the list get the number of rows, 'Obs' and number of columns 'Variable'), rbind the list elements after creating the column 'Data' as the names of the list
lst1 <- lapply(mget(ls(pattern = "^T\\d{3}$")),
function(x) data.frame(Obs = nrow(x),
Variable = ncol(x)))
out <- do.call(rbind, Map(cbind, Data = names(lst1), lst1))
row.names(out) <- NULL
If we need the column names, we could use rowr to cbind the column names when the lengths are not the same
lst1 <- lapply(mget(ls(pattern = "^T\\d{3}$")), names)
library(versions)
available.versions('rowr') # // check for available version. Not in CRAN
install.versions('rowr', '1.1.2') # // install a version
library(rowr) # // load the package
do.call(cbind.fill, c(lst1, fill = NA))
Or without installing rowr
mx <- max(lengths(lst1))
do.call(cbind, lapply(lst1, `length<-`, mx))
Or using tidyverse
library(dplyr)
library(purrr)
mget(ls(pattern = '^T\\d{3}$')) %>%
map_dfr(~ tibble(Obs = nrow(.x), Variable = ncol(.x)), .id = 'Data')
I have around 10 DFs and would like to perform the following calculations on all of them and then have out as 10 new DFs.
I have been able to get this to work for 1 DF, but rather than copying the code and changing the names, 10 times, I wanted to see if there is a way to do this in. Ideally, I end up with 1 DF and 10 different columns, but am happy with anything
The calculations I am trying to do are:
temp <- merge (x=DF1, y=temp1, by = c("name"), all.x= TRUE)
asset_column <-grep("^Assets_", names(DF1))
return_column <-grep("^Return_", names(DF1))
OutputDF <-
stack(colSums(t(t(temp[asset_column])/colSums(temp[asset_column],
na.rm=TRUE)) * US_only[return_column],na.rm =TRUE))
OutputDF['values'] = OutputDF['values']/100
If these are repeatable calculations in a list, loop through the list with lapply and do the same code where we specify the first dataset from the anonymous function call (function(x) x)
out <- lapply(lst1, function(x) {
temp <- merge (x, y=temp1, by = c("name"), all.x= TRUE)
asset_column <-grep("^Assets_", names(x))
return_column <-grep("^Return_", names(x))
OutputDF <-
stack(colSums(t(t(temp[asset_column])/colSums(temp[asset_column],
na.rm=TRUE)) * US_only[return_column],na.rm =TRUE))
OutputDF['values'] = OutputDF['values']/100
OutputDF
})
Here, the output is also a list of data.frames which can be kept in the list as such or extract with [[