R - How to create multiple datasets based on levels of factor in multiple columns? - r
I'm kinda new to R and still looking for ways to make my code more elegant. I want to create multiple datasets in a more efficient way, each based on a particular value over different columns.
This is my dataset:
df<-data.frame(A=c(1,2,2,3,4,5,1,1,2,3),
B=c(4,4,2,3,4,2,1,5,2,2),
C=c(3,3,3,3,4,2,5,1,2,3),
D=c(1,2,5,5,5,4,5,5,2,3),
E=c(1,4,2,3,4,2,5,1,2,3),
dummy1=c("yes","yes","no","no","no","no","yes","no","yes","yes"),
dummy2=c("high","low","low","low","high","high","high","low","low","high"))
And I need each column to be a factor:
df[colnames(df)] <- lapply(df[colnames(df)], factor)
Now, what I want to obtain is one dataframe called "Likert_rank_yes" that contains all the observations that in the column "dummy1" have "yes", one dataframe called "Likert_rank_no" that contains all the observations that in the column "dummy1" have "no", one dataframe called "Likert_rank_high" that contains all the observations that in the column "dummy2" have "high" and so on for all my other dummies.
I want to loop or streamline the process in some way, so that there are few commands to run to get all the datasets I need.
The first two dataframes should look something like this:
Dataframe called "Likert_rank_yes" that contains all the observations that in the column "dummy1" have "yes"
Dataframe called "Likert_rank_no" that contains all the observations that in the column "dummy1" have "no"
I have to do this with several dummies with multiple levels and would like to automate/loop the process or make it more efficient, so that I don't have to subset and rename every dataframe for each dummy level. Ideally I would also need to drop the last column in each df created (the one containing the dummy considered).
I tried splitting like below but it seems it is not possible using multiple values, I just get 4 dfs (yes AND high observations, yes AND low obs, no AND high obs etc.) like so:
Splitting with a list of columns doesn't work
list_df <- split(df[c(1:5)], list(df$dummy1,df$dummy2), sep=".")
Can you help? Thanks in advance!
You need two lapplys:
vals <- colnames(df)[1:5]
dummies <- colnames(df)[-(1:5)]
step1 <- lapply(dummies, function(x) df[, c(vals, x)])
step2 <- lapply(step1, function(x) split(x, x[, 6]))
names(step2) <- dummies
step2
# $dummy1
# $dummy1$no
# A B C D E dummy1
# 3 2 2 3 5 2 no
# 4 3 3 3 5 3 no
# 5 4 4 4 5 4 no
# 6 5 2 2 4 2 no
# 8 1 5 1 5 1 no
#
# $dummy1$yes
# A B C D E dummy1
# 1 1 4 3 1 1 yes
# 2 2 4 3 2 4 yes
# 7 1 1 5 5 5 yes
# 9 2 2 2 2 2 yes
# 10 3 2 3 3 3 yes
#
#
# $dummy2
# $dummy2$high
# A B C D E dummy2
# 1 1 4 3 1 1 high
# 5 4 4 4 5 4 high
# 6 5 2 2 4 2 high
# 7 1 1 5 5 5 high
# 10 3 2 3 3 3 high
#
# $dummy2$low
# A B C D E dummy2
# 2 2 4 3 2 4 low
# 3 2 2 3 5 2 low
# 4 3 3 3 5 3 low
# 8 1 5 1 5 1 low
# 9 2 2 2 2 2 low
For the first data set ("dummy1" and "no") use step2$dummy1$no or step2[[1]][[1]] or step2[["dummy1"]][["no"]].
For programming purposes it is usually better to keep the list intact since it makes it simple to write code that processes all of the data frames in the list without having to specify them individually.
You are very close:
tbls <- unlist(step2, recursive=FALSE)
list2env(tbls, envir=.GlobalEnv)
ls()
# [1] "df" "dummies" "dummy1.no" "dummy1.yes" "dummy2.high" "dummy2.low" "step1" "step2" "tbls" "vals"
This will create the same set of tables.
Related
How to extract a list of columns name based on the means of their data?
I'm pretty new to R and hope i'll make myself clear enough. I have a table of several columns which are factors. I want to make a score for each of these columns. Then I want to calculate the mean of each score, and display the list of columns ranked by their mean scores, is that possible ? Table would be: head(musico[,69:73]) AVIS1 AVIS2 AVIS3 AVIS4 AVIS5 1 2 1 2 3 2 2 2 5 2 3 2 3 3 2 5 5 1 4 1 2 5 5 5 5 1 5 1 3 1 6 4 1 4 5 4 I want to make a score for each: musico$score1<-0 musico$score1[musico$AVIS1==1]<-1 musico$score1[musico$AVIS1==2]<-0.5 then do the mean of each column score: mean of score1, mean of score2, ...: mean(musico$score1), mean(musico$score2), ... My goal is to have a list of titles (avis1, avis2,...) ranked by their mean score. Any advice appreciated !
Here's one way using base although it is somewhat unclear what you want. What does score1 have to do with AVIS1? I think you may be missing some of the data from musico. Based on the example provided, here's a base R solution. vapply loops through the data.frame and produces the mean for each column. Then the stack and order are only there to make the output a dataframe that looks nice. music <- read.table(text = " AVIS1 AVIS2 AVIS3 AVIS4 AVIS5 1 2 1 2 3 2 2 2 5 2 3 2 3 3 2 5 5 1 4 1 2 5 5 5 5 1 5 1 3 1 6 4 1 4 5 4", header = TRUE) means <- vapply(music, mean, 1) stack(means[order(means, decreasing = TRUE)]) values ind 4 4.000000 AVIS4 3 3.166667 AVIS3 2 2.666667 AVIS2 5 2.500000 AVIS5 1 2.166667 AVIS1
This is how I would do it by first introducing a scores vector to be used as a lookup. I assume that scores are decreasing by 0.5 and that the number of scores needed are according to the maximum number of levels found in your columns (i.e. 6 seen in AVIS1). Then using tidyr you can organise your data set such that you have to variables (i.e. AVIS and Value) containing the respective levels. Then add a score variable with the mutate function from dplyr in which the position of the score in the score vector matches the value in the Value variable. From here you can find the mean scores corresponding to the AVIS levels, arrange them accordingly and put them in a list. music <- read.table(text = " AVIS1 AVIS2 AVIS3 AVIS4 AVIS5 1 2 1 2 3 2 2 2 5 2 3 2 3 3 2 5 5 1 4 1 2 5 5 5 5 1 5 1 3 1 6 4 1 4 5 4", header = TRUE) # your data scores <- seq(1, by = -0.5, length.out = 6) # vector of scores library(tidyr) library(dplyr) music2 <- music %>% gather(AVIS, Value) %>% # here you tidy the data mutate(score = scores[Value]) %>% # match score to value group_by(AVIS) %>% # group AVIS levels summarise(score.mean = mean(score)) %>% # find mean scores for AVIS levels arrange(desc(score.mean)) list <- list(AVIS = music2$AVIS) # here is the list > list$AVIS [1] "AVIS1" "AVIS5" "AVIS2" "AVIS3" "AVIS4"
Merge multiple data frames with partially matching rows
I have data frames with lists of elements such as NAMES. There are different names in dataframes, but most of them match together. I'd like to combine all of them in one list in which I'd see whether some names are missing from any of df. DATA sample for df1: X x 1 1 rh_Structure/Focus_S 2 2 rh_Structure/Focus_C 3 3 lh_Structure/Focus_S 4 4 lh_Structure/Focus_C 5 5 RH_Type-Function-S 6 6 RH_REFERENT-S and for df2 X x 1 1 rh_Structure/Focus_S 2 2 rh_Structure/Focus_C 3 3 lh_Structure/Focus_S 4 4 lh_Structure/Focus_C 5 5 UCZESTNIK 6 6 COACH and expected result would be: NAME. df1 df2 1 COACH NA 6 2 lh_Structure/Focus_C 4 4 3 lh_Structure/Focus_S 3 3 4 RH_REFERENT-S 6 NA 5 rh_Structure/Focus_C 2 2 6 rh_Structure/Focus_S 1 1 7 RH_Type-Function-S 5 NA 8 UCZESTNIK NA 5 I can do that with merge.data.frame(df1,df2,by = "x", all=T), but the I can't do it with more df with similar structure. Any help would be appreciated.
It might be easier to work with this in a long form. Just rbind all the datasets below one another with a flag for which dataset they came from. Then it's relatively straightforward to get a tabulation of all the missing values (and as an added bonus, you can see if you have any duplicates in any of the source datasets):
dfs <- c("df1","df2")
dfall <- do.call(rbind, Map(cbind, mget(dfs), src=dfs))
table(dfall$x, dfall$src)
# df1 df2
# COACH 0 1
# lh_Structure/Focus_C 1 1
# lh_Structure/Focus_S 1 1
# RH_REFERENT-S 1 0
# rh_Structure/Focus_C 1 1
# rh_Structure/Focus_S 1 1
# RH_Type-Function-S 1 0
# UCZESTNIK 0 1
How to merge columns in R with different levels of values
I have been given a dataset that I am attempting to perform logistic regression on. However, to do so, I need to merge some columns in R. For instance in the carevaluations data set, I am given (BuyingPrice_low, BuyingPrice_medium, BuyingPrice_high, BuyingPrice_vhigh, MaintenancePrice_low MaintenancePrice_medium MaintenancePrice_high MaintenancePrice_vhigh) How would I combine the columns buying price_low, medium, etc. into one column called "BuyingPrice" with the order and their respective data in each column and the same with the maintenanceprice column?
library(dplyr) df <- data.frame(Buy_low=rep(c(0,1), 10), Buy_high=rep(c(0,1), 10)) one_column <- df %>% gather(var, value) head(one_column) var value 1 Buy_low 0 2 Buy_low 1 3 Buy_low 0 4 Buy_low 1 5 Buy_low 0 6 Buy_low 1
It can be done with stack in base R : df1 <- data.frame(a=1:3,b=4:6,c=7:9) stack(df1) # values ind # 1 1 a # 2 2 a # 3 3 a # 4 4 b # 5 5 b # 6 6 b # 7 7 c # 8 8 c # 9 9 c
How to remove outiers from multi columns of a data frame
I would like to get a data frame that contains only data that is within 2 SD per each numeric column. I know how to do it for a single column but how can I do it for a bunch of columns at once? Here is the toy data frame: df <- read.table(text = "target birds wolfs Country 3 21 7 a 3 8 4 b 1 2 8 c 1 2 3 a 1 8 3 a 6 1 2 a 6 7 1 b 6 1 5 c",header = TRUE) Here is the code line for getting only the data that is under 2 SD for a single column(birds).How can I do it for all numeric columns at once? df[!(abs(df$birds - mean(df$birds))/sd(df$birds)) > 2,] target birds wolfs Country 2 3 8 4 b 3 1 2 8 c 4 1 2 3 a 5 1 8 3 a 6 6 1 2 a 7 6 7 1 b 8 6 1 5 c
We can use lapply to loop over the dataset columns and subset the numeric vectors (by using a if/else condition) based on the mean and sd. lapply(df, function(x) if(is.numeric(x)) x[!(abs((x-mean(x))/sd(x))>2)] else x) EDIT: I was under the impression that we need to remove the outliers for each column separately. But, if we need to keep only the rows that have no outliers for the numeric columns, we can loop through the columns with lapply as before, instead of returning 'x', we return the sequence of 'x' and then get the intersect of the list element with Reduce. The numeric index can be used for subsetting the rows. lst <- lapply(df, function(x) if(is.numeric(x)) seq_along(x)[!(abs((x-mean(x))/sd(x))>2)] else seq_along(x)) df[Reduce(intersect,lst),]
I'm guessing that you are trying to filter your data set by checking that all of the numeric columns are within 2 SD (?) In that case I would suggest to create two filters. 1 one that will indicate numeric columns, the second one that will check that all of them within 2 SD. For the second condition, we can use the built in scale function indx <- sapply(df, is.numeric) indx2 <- rowSums(abs(scale(df[indx])) <= 2) == sum(indx) df[indx2,] # target birds wolfs Country # 2 3 8 4 b # 3 1 2 8 c # 4 1 2 3 a # 5 1 8 3 a # 6 6 1 2 a # 7 6 7 1 b # 8 6 1 5 c
Calculating the occurrences of numbers in the subsets of a data.frame
I have a data frame in R which is similar to the follows. Actually my real ’df’ dataframe is much bigger than this one here but I really do not want to confuse anybody so that is why I try to simplify things as much as possible.
So here’s the data frame.
id <-c(1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,3,3,3,3,3,3,3,3,3,3)
a <-c(3,1,3,3,1,3,3,3,3,1,3,2,1,2,1,3,3,2,1,1,1,3,1,3,3,3,2,1,1,3)
b <-c(3,2,1,1,1,1,1,1,1,1,1,2,1,3,2,1,1,1,2,1,3,1,2,2,1,3,3,2,3,2)
c <-c(1,3,2,3,2,1,2,3,3,2,2,3,1,2,3,3,3,1,1,2,3,3,1,2,2,3,2,2,3,2)
d <-c(3,3,3,1,3,2,2,1,2,3,2,2,2,1,3,1,2,2,3,2,3,2,3,2,1,1,1,1,1,2)
e <-c(2,3,1,2,1,2,3,3,1,1,2,1,1,3,3,2,1,1,3,3,2,2,3,3,3,2,3,2,1,3)
df <-data.frame(id,a,b,c,d,e)
df
Basically what I would like to do is to get the occurrences of numbers for each column (a,b,c,d,e) and for each id group (1,2,3) (for this latter grouping see my column ’id’).
So, for column ’a’ and for id number ’1’ (for the latter see column ’id’) the code would be something like this:
as.numeric(table(df[1:10,2]))
##The results are:
[1] 3 7
Just to briefly explain my results: in column ’a’ (and regarding only those records which have number ’1’ in column ’id’) we can say that number '1' occured 3 times and number '3' occured 7 times.
Again, just to show you another example. For column ’a’ and for id number ’2’ (for the latter grouping see again column ’id’):
as.numeric(table(df[11:20,2]))
##After running the codes the results are:
[1] 4 3 3
Let me explain a little again: in column ’a’ and regarding only those observations which have number ’2’ in column ’id’) we can say that number '1' occured 4 times, number '2' occured 3 times and number '3' occured 3 times.
So this is what I would like to do. Calculating the occurrences of numbers for each custom-defined subsets (and then collecting these values into a data frame). I know it is not a difficult task but the PROBLEM is that I’m gonna have to change the input ’df’ dataframe on a regular basis and hence both the overall number of rows and columns might change over time…
What I have done so far is that I have separated the ’df’ dataframe by columns, like this:
for (z in (2:ncol(df))) assign(paste("df",z,sep="."),df[,z])
So df.2 will refer to df$a, df.3 will equal df$b, df.4 will equal df$c etc. But I’m really stuck now and I don’t know how to move forward…
Is there a proper, ”automatic” way to solve this problem?
How about - > library(reshape) > dftab <- table(melt(df,'id')) > dftab , , value = 1 variable id a b c d e 1 3 8 2 2 4 2 4 6 3 2 4 3 4 2 1 5 1 , , value = 2 variable id a b c d e 1 0 1 4 3 3 2 3 3 3 6 2 3 1 4 5 3 4 , , value = 3 variable id a b c d e 1 7 1 4 5 3 2 3 1 4 2 4 3 5 4 4 2 5 So to get the number of '3's in column 'a' and group '1' you could just do > dftab[3,'a',1] [1] 4
A combination of tapply and apply can create the data you want: tapply(df$id,df$id,function(x) apply(df[id==x,-1],2,table)) However, when a grouping doesn't have all the elements in it, as in 1a, the result will be a list for that id group rather than a nice table (matrix). $`1` $`1`$a 1 3 3 7 $`1`$b 1 2 3 8 1 1 $`1`$c 1 2 3 2 4 4 $`1`$d 1 2 3 2 3 5 $`1`$e 1 2 3 4 3 3 $`2` a b c d e 1 4 6 3 2 4 2 3 3 3 6 2 3 3 1 4 2 4 $`3` a b c d e 1 4 2 1 5 1 2 1 4 5 3 4 3 5 4 4 2 5
I'm sure someone will have a more elegant solution than this, but you can cobble it together with a simple function and dlply from the plyr package.
ColTables <- function(df) {
counts <- list()
for(a in names(df)[names(df) != "id"]) {
counts[[a]] <- table(df[a])
}
return(counts)
}
results <- dlply(df, "id", ColTables)
This gets you back a list - the first "layer" of the list will be the id variable; the second the table results for each column for that id variable. For example:
> results[['2']]['a']
$a
1 2 3
4 3 3
For id variable = 2, column = a, per your above example.
A way to do it is using the aggregate function, but you have to add a column to your dataframe
> df$freq <- 0
> aggregate(freq~a+id,df,length)
a id freq
1 1 1 3
2 3 1 7
3 1 2 4
4 2 2 3
5 3 2 3
6 1 3 4
7 2 3 1
8 3 3 5
Of course you can write a function to do it, so it's easier to do it frequently, and you don't have to add a column to your actual data frame
> frequency <- function(df,groups) {
+ relevant <- df[,groups]
+ relevant$freq <- 0
+ aggregate(freq~.,relevant,length)
+ }
> frequency(df,c("b","id"))
b id freq
1 1 1 8
2 2 1 1
3 3 1 1
4 1 2 6
5 2 2 3
6 3 2 1
7 1 3 2
8 2 3 4
9 3 3 4
You didn't say how you'd like the data. The by function might give you the output you like. by(df, df$id, function(x) lapply(x[,-1], table))