My data is like:
a <- data.frame(a1=c(2,2,1,1,2,2,3,3),
a2=c(5,4,2,2,5,5,6,6),
a3=c(3,1,5,5,7,7,8,8))
Then, i sort the data like:
aa <- a %>%
arrange(desc(a3),desc(a2),desc(a1))
The data looks like:
> aa
a1 a2 a3
1 3 6 8
2 3 6 8
3 2 5 7
4 2 5 7
5 1 2 5
6 1 2 5
7 2 5 3
8 2 4 1
Now i need to group the data by a3, a2 and a1. So, in aa, the rows 1 and 2 will be in one group, and row 3 and 4 will be in one group as well. Now I need to give every group an index, which starts from 1. So, the data should look like below:
> aa
a1 a2 a3 Index
1 3 6 8 1
2 3 6 8 1
3 2 5 7 2
4 2 5 7 2
5 1 2 5 3
6 1 2 5 3
7 2 5 3 4
8 2 4 1 5
So in summarizing, I need to arrange the data in the descending order first, then group it, then give every group an index starting from 1. Could anyone help me out here?
We could potentially use group_indices, but that would also have a reordering issue. Instead, an option is to paste (or str_c - from stringr) on the columns of interest and then match with unique values of pasted string
library(dplyr)
library(stringr)
aa %>%
mutate(Index = str_c(a1, a2, a3),
Index = match(Index, unique(Index)))
Or instead of arrangeing separately, use it with across
library(tidyr)
a %>%
arrange(across(a1:a3, desc)) %>%
unite(Index, everything(), remove = FALSE) %>%
mutate(Index = match(Index, unique(Index)))
Or with .GRP in data.table
library(dplyr)
setDT(aa)[, Index := .GRP, .(a1, a2, a3)]
aa
# a1 a2 a3 Index
#1: 3 6 8 1
#2: 3 6 8 1
#3: 2 5 7 2
#4: 2 5 7 2
#5: 1 2 5 3
#6: 1 2 5 3
#7: 2 5 3 4
#8: 2 4 1 5
Base R:
a_ordered <- with(a, a[rev(order(a1, a2, a3)), ])
a_ordered$idx <- with(a_ordered,
cumsum(abs(c(
0,
diff(as.integer(factor(paste0(
a1, a2, a3
))))
))) + 1)
Data:
a <- data.frame(
a1 = c(2, 2, 1, 1, 2, 2, 3, 3),
a2 = c(5, 4, 2, 2, 5, 5, 6, 6),
a3 = c(3, 1, 5, 5, 7, 7, 8, 8)
)
Related
I have a large list with thousands of data frames included in it. These data frames have multiple columns each. Thereby, I want to automatically bind in each of these data frames the columns into one column. This means that they are appended below each other as shown below. Thereafter, I would transform the list to a data frame which would have varying column lengths due to the different number of columns within each element in the original list.
From this:
y1 y2
1 4
2 5
3 6
To this:
y1
1
2
3
4
5
6
This should be done for each element in the list, whereby the solution needs to take into account that there are thousands of different data frames, which cannot be mentioned individually (example):
df1 = data.frame(
X1 = c(1, 2, 3),
X1.2 = c(4, 5, 6)
)
df2 = data.frame(
X2 = c(7, 8, 9),
X2.2 = c(1, 4, 6)
)
df3 = data.frame(
X3 = c(3, 4, 1),
X3.2 = c(8, 3, 5),
X3.3 = c(3, 1, 9)
)
listOfDataframe = list(df1, df2, df3)
Final output:
df_final = data.frame(
X1 = c(1, 2, 3, 4, 5, 6),
X2 = c(7, 8, 9, 1, 4, 6),
X3 = c(3, 4, 1, 8, 3, 5, 3, 1, 9)
)
Another problem underlying this question is that there will be a differing number of rows, which I do not know how to account for in the data frame, as the columns need to have the same length.
Thank you in advance for your help, it is highly appreciated.
Structure of list within R:
We can unlist after looping over the list with lapply
lst1 <- lapply(listOfDataframe, \(x)
setNames(data.frame(unlist(x, use.names = FALSE)), names(x)[1]))
-output
lst1
[[1]]
X1
1 1
2 2
3 3
4 4
5 5
6 6
[[2]]
X2
1 7
2 8
3 9
4 1
5 4
6 6
[[3]]
X3
1 3
2 4
3 1
4 8
5 3
6 5
7 3
8 1
9 9
If we need to convert the list to a single data.frame, use cbind.na from qPCR
do.call(qpcR:::cbind.na, lst1)
X1 X2 X3
1 1 7 3
2 2 8 4
3 3 9 1
4 4 1 8
5 5 4 3
6 6 6 5
7 NA NA 3
8 NA NA 1
9 NA NA 9
Here is a tidyverse solution:
library(dplyr)
library(purrr)
listOfDataframe %>%
map(~.x %>% stack(.)) %>%
map(~.x %>% select(-ind))
[[1]]
values
1 1
2 2
3 3
4 4
5 5
6 6
[[2]]
values
1 7
2 8
3 9
4 1
5 4
6 6
[[3]]
values
1 3
2 4
3 1
4 8
5 3
6 5
7 3
8 1
9 9
If I have a dataframe like this:
df <- data.frame(c1=1:6, c2=2:7)
I can happily replace values in c2 that are larger then 4 doing
df$c2[df$c2 > 4] <- 10
yielding the desired output
c1 c2
1 1 2
2 2 3
3 3 4
4 4 10
5 5 10
6 6 10
However, I want to select the column by its name using a string, in this case "c2", as the column selection should not be hard-coded but it is context dependent.
The best I could come up with is
df[,c('c2')][df[,c('c2')] > 4] <- 1000
yielding
c1 c2
1 1 2
2 2 3
3 3 4
4 4 1000
5 5 1000
6 6 1000
It works, but I find it rather ugly. Is there a better way of doing the same thing?
Maybe using replace
df['c2'] <- replace(df['c2'], df['c2'] > 4, 100)
df
# c1 c2
#1 1 2
#2 2 3
#3 3 4
#4 4 100
#5 5 100
#6 6 100
Or something similar as your attempt
df['c2'][df['c2'] > 4] <- 100
If one is open to packages, we can use purrr's modify_at or dplyr's mutate_at
purrr::modify_at(df,"c2",
function(x)
ifelse(x>4,100,x))
With dplyr:
mutate_at(df,"c2",
function(x)
ifelse(x>4,100,x))
Using transform and ifelse
transform(df, c2 = ifelse(c2 > 4, 100, c2))
# c1 c2
#1 1 2
#2 2 3
#3 3 4
#4 4 100
#5 5 100
#6 6 100
If we need to pass a string, one option with dplyr, would be convert to symbol and evaluate
library(dplyr)
df %>%
mutate(!! "c2" := replace(!! rlang::sym("c2"),
!! rlang::sym("c2") > 4, 100))
# c1 c2
#1 1 2
#2 2 3
#3 3 4
#4 4 100
#5 5 100
#6 6 100
df[df$c2 > 4, 'c2'] <- 10
# or
df$c2 <- with(df, replace(c2, c2 > 4, 10))
Using package data.table you could do:
library(data.table)
setDT(df)
df[c2 > 4, c2 := 10]
I have data on several thousand US basketball players over multiple years.
Each basketball player has a unique ID. It is known for what team and on which position they play in a given year, much like the mock data df below:
df <- data.frame(id = c(rep(1:4, times=2), 1),
year = c(1, 1, 2, 2, 3, 4, 4, 4,5),
team = c(1,2,3,4, 2,2,4,4,2),
position = c(1,2,3,4,1,1,4,4,4))
> df
id year team position
1 1 1 1 1
2 2 1 2 2
3 3 2 3 3
4 4 2 4 4
5 1 3 2 1
6 2 4 2 1
7 3 4 4 4
8 4 4 4 4
9 1 5 2 4
What is an efficient way to manipulate df into new_df below?
> new_df
id move time position.1 position.2 year.1 year.2
1 1 0 2 1 1 1 3
2 2 1 3 2 1 1 4
3 3 0 2 3 4 2 4
4 4 1 2 4 4 2 4
5 1 0 2 1 4 3 5
In new_df the first occurrence of the basketball player is compared to the second occurrence, recorded whether the player switched teams and how long it took the player to make the switch.
Note:
In the real data some basketball players occur more than twice and can play for multiple teams and on multiple positions.
In such a case a new row in new_df is added that compares each additional occurrence of a player with only the previous occurrence.
Edit: I think this is not a rather simple reshape exercise, because of the reasons mentioned in the previous two sentences. To clarify this, I've added an additional occurrence of player ID 1 to the mock data.
Any help is most welcome and appreciated!
s=table(df$id)
df$time=rep(1:max(s),each=length(s))
df1 = reshape(df,idvar = "id",dir="wide")
transform(df1, move=+(team.1==team.2),time=year.2-year.1)
id year.1 team.1 position.1 year.2 team.2 position.2 move time
1 1 1 1 1 3 2 1 0 2
2 2 1 2 2 4 2 1 1 3
3 3 2 3 3 4 4 4 0 2
4 4 2 4 4 4 4 4 1 2
The below code should help you get till the point where the data is transposed
You'll have to create the move and time variables
df <- data.frame(id = rep(1:4, times=2),
year = c(1, 1, 2, 2, 3, 4, 4, 4),
team = c(1, 2, 3, 4, 2, 2, 4, 4),
position = c(1, 2, 3, 4, 1, 1, 4, 4))
library(reshape2)
library(data.table)
setDT(df) #convert to data.table
df[,rno:=rank(year,ties="min"),by=.(id)] #gives the occurance
#creating the transposed dataset
Dcast_DT<-dcast(df,id~rno,value.var = c("year","team","position"))
This piece of code did the trick, using data.table
#transform to data.table
dt <- as.data.table(df)
#sort on year
setorder(dt, year, na.last=TRUE)
#indicate the names of the new columns
new_cols= c("time", "move", "prev_team", "prev_year", "prev_position")
#set up the new variables
dtt[ , (new_cols) := list(year - shift(year),team!= shift(team), shift(team), shift(year), shift(position)), by = id]
# select only repeating occurrences
dtt <- dtt[!is.na(dtt$time),]
#outcome
dtt
id year team position time move prev_team prev_year prev_position
1: 1 3 2 1 2 TRUE 1 1 1
2: 2 4 2 1 3 FALSE 2 1 2
3: 3 4 4 4 2 TRUE 3 2 3
4: 4 4 4 4 2 FALSE 4 2 4
5: 1 5 2 4 2 FALSE 2 3 1
Say I have a longitudinal data set as below
ID <- c(1, 1, 2, 2, 3, 3, 4, 4)
time <- c(1, 2, 1, 2, 1, 2, 1, 2)
value <- c(7, 5, 9, 2, NA, 3, 7, NA)
mydata <- data.frame(ID, time, value)
ID time value
1 1 1 7
2 1 2 5
3 2 1 9
4 2 2 2
5 3 1 NA
6 3 2 3
7 4 1 7
8 4 2 NA
In this data-set, we have 4 cases with data at two time-points (let's say pre and post treatment)
Something I want to do is set criteria to delete any case that are not complete for both time-points. In this example, I would want to delete ID3 (who is missing timepoint 1), and ID4 (who is missing timepoint 2). Like below:
ID time value
1 1 1 7
2 1 2 5
3 2 1 9
4 2 2 2
I am not having much luck. I've tried variants of complete.cases() or which() to no avail
I'm still new to R, and would be hugely appreciative if anyone could help me out
Edit: Thank you Ronak for answering my question. Upon reflection of my real data, I have encountered a second problem. My actual data is more reflected by the below:
ID <- c(1, 1, 2, 2, 3, 3, 4, 4, 5, 6, 7, 8)
time <- c(1, 2, 1, 2, 1, 2, 1, 2, 1, 1, 1, 1)
value <- c(7, 5, 9, 2, NA, 3, 7, NA, 8, 9, 7, 6)
mydata <- data.frame(ID, time, value)
ID time value
1 1 1 7
2 1 2 5
3 2 1 9
4 2 2 2
5 3 1 NA
6 3 2 3
7 4 1 7
8 4 2 NA
9 5 1 8
10 6 1 9
11 7 1 7
12 8 1 6
Where I would also want to remove cases 5, 6, 7 and 8. These IDs have an entry for Time 1, but not Time 2. Hopefully this makes sense
Thanks a heap
If you switch your data to wide format (where each time point is represented as its own column), then you can use na.omit. Using dplyr and tidyr functions:
library(dplyr)
mydata <- mydata %>%
tidyr::spread(key=time, value=value) %>% # reformat to wide
na.omit() %>% # delete cases with missingness on any variable (i.e. any time point)
tidyr::gather(key="time", value="value", -ID) # put it back in long format
> mydata
ID time value
1 1 1 7
2 2 1 9
3 1 2 5
4 2 2 2
Note that this will work (it will keep only cases with complete data for both time 1 and time 2) even when you have a time point missing without an explicit NA present in the data, like this:
> mydata
ID time value
1 1 1 7
2 1 2 5
3 2 1 9
4 2 2 2
5 3 1 NA
6 3 2 3
7 4 1 7
8 4 2 NA
9 5 1 8
10 6 1 9
11 7 1 7
12 8 1 6
You can do this easily with sqldf.
library(sqldf)
sqldf(' select * from (select ID, count(*) as cnt from mydata where value is not null group by id having cnt >1 ) t1 inner join mydata t2 on t1.ID=t2.ID')
You would select those id having a count greater than 1 and who doesn't have NA in their values and then join back with the original data.
#Ronak already provided
mydata[!mydata$ID %in% mydata$ID[is.na(mydata$value)], ]
For the second part, you can just group over each id and filter on their frequency
k2 <- data.frame(table(mydata$ID))
k2$Var1[k2$Freq > 1]
and then do something like
mydata[mydata$ID %in% k2$Var1[k2$Freq > 1],]
See the updated answer
# Eliminates ID cases with NA
mydata = mydata[!mydata$ID %in% mydata[!complete.cases(mydata) ,]$ID, ]
library(plyr)
# counts all the IDs
cnt = count(mydata, "ID")
# Eliminates any ID that doesn't have 2 observations
mydata[mydata$ID %in% cnt[cnt$freq == 2, ]$ID, ]
ID time value
1 1 1 7
2 1 2 5
3 2 1 9
4 2 2 2
This question already has answers here:
How to get frequencies then add it as a variable in an array?
(3 answers)
Closed 8 years ago.
Suppose I have the following data frame:
userID <- c(1, 1, 3, 5, 3, 5)
A <- c(2, 3, 2, 1, 2, 1)
B <- c(2, 3, 1, 0, 1, 0)
df <- data.frame(userID, A, B)
df
# userID A B
# 1 1 2 2
# 2 1 3 3
# 3 3 2 1
# 4 5 1 0
# 5 3 2 1
# 6 5 1 0
I would like to create a data frame with the same columns but with an added final column that counts up the number of unique tuples / combinations of the other columns. The output should look like the following:
userID A B count
1 2 2 1
1 3 3 1
3 2 1 2
5 1 0 2
The meaning is the the tuple / combination of (1, 2, 2) occurs with count=1, while the tuple of (3, 2, 1) occurs twice so has count=2. I would prefer not to use any external packages.
1) aggregate
ag <- aggregate(count ~ ., cbind(count = 1, df), length)
ag[do.call("order", ag), ] # sort the rows
giving:
userID A B count
3 1 2 2 1
4 1 3 3 1
2 3 2 1 2
1 5 1 0 2
The last line of code which sorts the rows could be omitted if the order of the rows is unimportant.
The remaining solutions use the indicated packages:
2) sqldf
library(sqldf)
Names <- toString(names(df))
fn$sqldf("select *, count(*) count from df group by $Names order by $Names")
giving:
userID A B count
1 1 2 2 1
2 1 3 3 1
3 3 2 1 2
4 5 1 0 2
The order by clause could be omitted if the order is unimportant.
3) dplyr
library(dplyr)
df %>% regroup(as.list(names(df))) %>% summarise(count = n())
giving:
Source: local data frame [4 x 4]
Groups: userID, A
userID A B count
1 1 2 2 1
2 1 3 3 1
3 3 2 1 2
4 5 1 0 2
4) data.table
library(data.table)
data.table(df)[, list(count = .N), by = names(df)]
giving:
userID A B count
1: 1 2 2 1
2: 1 3 3 1
3: 3 2 1 2
4: 5 1 0 2
ADDED additional solutions. Also some small improvements.
Here's a fairly straightforward way (ave to the rescue!):
unique(cbind(df,
count = ave(rep(1, nrow(df)),
do.call(paste, df),
FUN = length)))
# userID A B count
# 1 1 2 2 1
# 2 1 3 3 1
# 3 3 2 1 2
# 4 5 1 0 2
Here's a variation of the above:
unique(within(df, {
counter <- rep(1, nrow(df))
count <- ave(counter, df, FUN = length)
rm(counter)
}))
# userID A B count
# 1 1 2 2 1
# 2 1 3 3 1
# 3 3 2 1 2
# 4 5 1 0 2
userID <- c(1, 1, 3, 5, 3, 5)
A <- c(2, 3, 2, 1, 2, 1)
B <- c(2, 3, 1, 0, 1, 0)
df <- data.frame(userID, A, B)
Make a quick factor of the tuples:
df$AB <- as.factor(paste(df$userID,df$A,df$B, sep=""))
No external packages just taking advantage of summary() and storing it as a DF then merging the counts on the original data:
df2 <- as.data.frame(summary(df$AB))
df2 <- data.frame(x=row.names(df2), y=df2[1])
names(df2) <- c("AB", "count")
df <- merge(df, df2, by="AB", all.x=TRUE)
df$AB <- NULL
Almost final output, just has dupes:
df
userID A B count
1 1 2 2 1
2 1 3 3 1
3 3 2 1 2
4 3 2 1 2
5 5 1 0 2
6 5 1 0 2
Lastly, clean up dupes:
df <- df[!duplicated(df), ]
Here you go:
df
userID A B count
1 1 2 2 1
2 1 3 3 1
3 3 2 1 2
5 5 1 0 2
Been a while not doing that with sql or plyr. if you can use dplyr or a package later on do it. Bioconductor has a lot of great sequencing packages if it starts to get more complex.
Hope this helps.
This should do the trick, even if it is a little bit ugly:
vec <- table(apply(df,1,paste,collapse=""))
df2 <- data.frame(do.call(rbind,strsplit(names(vec),"")))
names(df2) <- names(df)
df2$count <- vec
# userID A B count
#1 1 2 2 1
#2 1 3 3 1
#3 3 2 1 2
#4 5 1 0 2