I have a dataframe which contains 63 columns and 50 rows. I have given below a toy dataset.
>df
rs_1 rs_2 rs_3 rs_4 ... rs_60 A.Ag B.Ag C.Ag
0 0 1 2 ... 1 02:/01 02:/07 03:07/04:01
1 2 1 2 ... 0 02:/01 02:/07 03:07/04:01
2 1 1 2 ... 2 02:/01 02:/07 03:07/04:01
0 0 1 0 ... 2 02:/01 02:/07 03:07/04:01
Now I need to find the highest frequencies of the columns (A.Ag, B.Ag and C.Ag) for each rs_* =0, 1 and 2 separately. The desire outcome would be for example rs_*=0
rs_id Code A.Ag Code B.Ag Code C.Ag
rs_1 02:/01 2 02:/07 5 03:07 5
rs_2 02:/01 3 01:/05 2 05:00 4
could you please help me with this? I tried with the following function
for (i in 1:60){
if (file[,i]==0)
{
temp1 = data.frame(sort(table(file[,61]), decreasing = TRUE)) #onlr for A.Ag coulmn
temp1$Var1 = names(file)[i]
res_types = rbind(res_types, temp1)
}
}
I got the number of frequencies and rs_id. But could not get the code. Can anyone help me with this?
The desire outcome will be
rs_id Code Combination A.A Combination B.Ag Combination C.Ag
rs_1 0 1:01/1:01 7 13:02/13:02 2 03:04/03:04 3
rs_1 0 1:01/11:01 5 13:02/49:01 2 03:04/15:02 3
rs_1 0 1:01/2:01 4 13:02/57:01 2 03:04/7:01 3
rs_1 1 1:01/2:05 3 13:02/8:01 4 06:02/06:02 3
rs_1 1 1:01/24:02 3 14:01/14:02 3 06:02/15:02 3
rs_1 1 1:01/24:02 3 14:01/14:02 2 06:02/15:02 3
rs_2 0 1:01/31:01 3 15:01/15:01 1 06:02/3:03 4
rs_2 0 11:01/2:01 4 15:01/18:01 1 06:02/3:04 1
It might be easier to do this using data.table package. Explanation inline.
library(data.table)
#convert into a long format
longDat <- melt(dat, measure.vars=patterns("^rs"), variable.name="rs_id",
value.name="val_id")
#for each group of rs_id (rs_1, ..., rs_60) and val_id in (0,1,2),
#count the frequency of each code
longDat[,
unlist(
lapply(c("A.Ag","B.Ag","C.Ag"),
function(x) setNames(aggregate(get(x), list(get(x)), length), c("Code", x))
),
recursive=FALSE),
by=c("rs_id", "val_id")]
Is this what you are looking for? Does this help?
data:
library(data.table)
dat <- fread("rs_1,rs_2,rs_3,rs_4,rs_60,A.Ag,B.Ag,C.Ag
0,0,1,2,1,02:/01,02:/07,03:07/04:01
1,2,1,2,0,02:/01,02:/07,03:07/04:01
2,1,1,2,2,02:/01,02:/07,03:07/04:01
0,0,1,0,2,02:/01,02:/07,03:07/04:01")
edit: OP request to retrieve top 3 for each rs_id, val_id and *.Ag
It is prob more readable to do it one *.Ag at a time, count and then take top 3 and then finally merge all the results as follows:
library(data.table)
#convert into a long format
longDat <- melt(dat, measure.vars=patterns("^rs"), variable.name="rs_id",
value.name="val_id")
ids <- c("rs_id", "val_id")
Reduce(function(dt1,dt2) merge(dt1,dt2,by=ids,all=TRUE),
lapply(c("A.Ag","B.Ag","C.Ag"), function(x) {
res <- longDat[, list(.N), by=c(ids, x)][order(-N)]
setnames(res[, head(.SD ,3L), by=ids], c(x, "N"), c(paste0(x,"_Code"), x))
}))
Related
My dataset has columns and values like this. The column names all start with a common string, Col_a_**
ID Col_a_01 Col_a_02 Col_a_03
1 1 2 1
2 1 NA 0
3 NA 0 2
4 1 0 1
5 0 0 2
My goal is to replace the missing values with the mode values for that column.
The expected dataset to be like this
ID Col_a_01 Col_a_02 Col_a_03
1 1 2 1
2 1 0** 0
3 1** 0 2
4 1 0 1
5 0 0 2
The NA in the first column is replaced by 1 because the mode of the 1st column is 1. The NA in the second column is replaced by 0 because the mode for the 2nd column is 0.
I can do this like this below
getmode <- function(v) {
uniqv <- unique(v)
uniqv[which.max(tabulate(match(v, uniqv)))]
}
df$Col_a_01[is.na(Col_a_01)==TRUE] <- getmode(df$Col_a_01)
df$Col_a_03[is.na(Col_a_02)==TRUE] <- getmode(df$Col_a_02)
df$Col_a_03[is.na(Col_a_03)==TRUE] <- getmode(df$Col_a_03)
But this becomes unwieldy if I have 100 columns starting with the similar names ending in 1,2,3..100. I am curious if there is an easier and more elegant way to accomplish this. Thanks in advance.
You can change the NA values with ifelse/replace, to apply a function to multiple columns use across in dplyr.
library(dplyr)
df <- df %>%
mutate(across(starts_with('Col_a'), ~replace(., is.na(.), getmode(.))))
In base R , use lapply -
cols <- grep('Col_a', names(df))
df[cols] <- lapply(df[cols], function(x) replace(x, is.na(x), getmode(x)))
We can use na.aggregate with FUN specified as getmode
library(zoo)
library(dplyr)
df1 <- df1 %>%
mutate(across(starts_with('Col_a'), na.aggregate, FUN = getmode))
-output
df1
ID Col_a_01 Col_a_02 Col_a_03
1 1 1 2 1
2 2 1 0 0
3 3 1 0 2
4 4 1 0 1
5 5 0 0 2
Or it can be simply
na.aggregate(df1, FUN = getmode)
ID Col_a_01 Col_a_02 Col_a_03
1 1 1 2 1
2 2 1 0 0
3 3 1 0 2
4 4 1 0 1
5 5 0 0 2
I have a data frame as
df<- as.data.frame(expand.grid(0:1, 0:4, 0:3,0:7, 2:7))
I want to get all unique combinations using 2 variables of the given 5 variables in the data frame df
Apply a function f (extracting unique couple) to each couple of columns:
f<-function(col,df)
{
return(unique(df[,col]))
}
#All combinantions
comb_col<-combn(colnames(df),2)
Your output
apply(comb_col,2,f,df=df)
[[1]]
Var1 Var2
1 0 0
2 1 0
3 0 1
4 1 1
5 0 2
6 1 2
7 0 3
8 1 3
9 0 4
10 1 4
[[2]]
Var1 Var3
1 0 0
2 1 0
11 0 1
12 1 1
21 0 2
22 1 2
31 0 3
32 1 3
...
You can use distinct function from dplyr package:
df <- as.data.frame(expand.grid(0:1, 0:4, 0:3,0:7, 2:7))
library(dplyr)
df %>%
distinct(Var1, Var2)
Also you have an option to keep the rest of your columns with .keep_all = TRUE parameter.
If you want to get all the possible combinations:
# Generate matrix with all combinations of variables
comb <- combn(names(df), 2)
# Generate a list with all unique values in your data.frame
apply(comb, 2, function(x) df %>% distinct_(.dots = x))
I am trying to split one column in a data frame in to multiple columns which hold the values from the original column as new column names. Then if there was an occurrence for that respective column in the original give it a 1 in the new column or 0 if no match. I realize this is not the best way to explain so, for example:
df <- data.frame(subject = c(1:4), Location = c('A', 'A/B', 'B/C/D', 'A/B/C/D'))
# subject Location
# 1 1 A
# 2 2 A/B
# 3 3 B/C/D
# 4 4 A/B/C/D
and would like to expand it to wide format, something such as, with 1's and 0's (or T and F):
# subject A B C D
# 1 1 1 0 0 0
# 2 2 1 1 0 0
# 3 3 0 1 1 1
# 4 4 1 1 1 1
I have looked into tidyr and the separate function and reshape2 and the cast function but seem to getting hung up on giving logical values. Any help on the issue would be greatly appreciated. Thank you.
You may try cSplit_e from package splitstackshape:
library(splitstackshape)
cSplit_e(data = df, split.col = "Location", sep = "/",
type = "character", drop = TRUE, fill = 0)
# subject Location_A Location_B Location_C Location_D
# 1 1 1 0 0 0
# 2 2 1 1 0 0
# 3 3 0 1 1 1
# 4 4 1 1 1 1
You could take the following step-by-step approach.
## get the unique values after splitting
u <- unique(unlist(strsplit(as.character(df$Location), "/")))
## compare 'u' with 'Location'
m <- vapply(u, grepl, logical(length(u)), x = df$Location)
## coerce to integer representation
m[] <- as.integer(m)
## bind 'm' to 'subject'
cbind(df["subject"], m)
# subject A B C D
# 1 1 1 0 0 0
# 2 2 1 1 0 0
# 3 3 0 1 1 1
# 4 4 1 1 1 1
I am currently working on a Multistate Analysis dataset in "long" form (one row for each individual's observation; each individual is repeatedly measured up to 5 times).
The idea is that each individual can recurrently transition across the levels of the time-varying state variable s = 1, 2, 3, 4. All the other variables that I have (here cohort) are fixed within any given id.
After some analyses, I need to reshape the dataset in "wide" form, according to the specific sequence of visited states. Here is an example of the initial long data:
dat <- read.table(text = "
id cohort s
1 1 2
1 1 2
1 1 1
1 1 4
2 3 1
2 3 1
2 3 3
3 2 1
3 2 2
3 2 3
3 2 3
3 2 4",
header=TRUE)
The final "wide" dataset should take into account the specific individual sequence of visited states, recorded into the newly created variables s1, s2, s3, s4, s5, where s1 is the first state visited by the individual and so on.
According to the above example, the wide dataset looks like:
id cohort s1 s2 s3 s4 s5
1 1 2 2 1 4 0
2 3 1 1 3 0 0
3 2 1 2 3 3 4
I tried to use reshape(), and also to focus on transposing s, but without the intended result. Actually, my knowledge of the R functions is quite limited.. Can you give any suggestion? Thanks.
EDIT: obtaining a different kind of wide dataset
Thank you all for your help, I have a related question if I can. Especially when each individual is observed for a long time and there are few transitions across states, it is very useful to reshape the initial sample dat in this alternative way:
id cohort s1 s2 s3 s4 s5 dur1 dur2 dur3 dur4 dur5
1 1 2 1 4 0 0 2 1 1 0 0
2 3 1 3 0 0 0 2 1 0 0 0
3 2 1 2 3 4 0 1 1 2 1 0
In practice now s1-s5 are the distinct visited states, and dur1-dur5 the time spent in each respective distinct visited state.
Can you please give a hand for reaching this data structure? I believe it is necessary to create all the dur- and s- variables in an intermediate sample before using reshape(). Otherwise maybe it is possible to directly adopt -reshape2-?
dat <- read.table(text = "
id cohort s
1 1 2
1 1 2
1 1 1
1 1 4
2 3 1
2 3 1
2 3 3
3 2 1
3 2 2
3 2 3
3 2 3
3 2 4",
header=TRUE)
df <- data.frame(
dat,
period = sequence(rle(dat$id)$lengths)
)
wide <- reshape(df, v.names = "s", idvar = c("id", "cohort"),
timevar = "period", direction = "wide")
wide[is.na(wide)] = 0
wide
Gives:
id cohort s.1 s.2 s.3 s.4 s.5
1 1 1 2 2 1 4 0
5 2 3 1 1 3 0 0
8 3 2 1 2 3 3 4
then using the following line gives your names:
names(wide) <- c('id','cohort', paste('s', seq_along(1:5), sep=''))
# id cohort s1 s2 s3 s4 s5
# 1 1 1 2 2 1 4 0
# 5 2 3 1 1 3 0 0
# 8 3 2 1 2 3 3 4
If you use sep='' in the wide statement you do not have to rename the variables:
wide <- reshape(df, v.names = "s", idvar = c("id", "cohort"),
timevar = "period", direction = "wide", sep='')
I suspect there are ways to avoid creating the period variable and avoid replacing NA directly in the wide statement, but I have not figured those out yet.
ok...
library(plyr)
library(reshape2)
dat2 <- ddply(dat,.(id,cohort), function(x)
data.frame(s=x$s,name=paste0("s",seq_along(x$s))))
dat2 <- ddply(dat2,.(id,cohort), function(x)
dcast(x, id + cohort ~ name, value.var= "s" ,fill= 0)
)
dat2[is.na(dat2)] <- 0
dat2
# id cohort s1 s2 s3 s4 s5
# 1 1 1 2 2 1 4 0
# 2 2 3 1 1 3 0 0
# 3 3 2 1 2 3 3 4
This seem right? I admit the first ddply is hardly elegant.
Try this:
library(reshape2)
dat$seq <- ave(dat$id, dat$id, FUN = function(x) paste0("s", seq_along(x)))
dat.s <- dcast(dat, id + cohort ~ seq, value.var = "s", fill = 0)
which gives this:
> dat.s
id cohort s1 s2 s3 s4 s5
1 1 1 2 2 1 4 0
2 2 3 1 1 3 0 0
3 3 2 1 2 3 3 4
If you did not mind using just 1, 2, ..., 5 as column names then you could shorten the ave line to just:
dat$seq <- ave(dat$id, dat$id, FUN = seq_along)
Regarding the second question that was added later try this:
library(plyr)
dur.fn <- function(x) {
r <- rle(x$s)$length
data.frame(id = x$id[1], dur.value = r, dur.seq = paste0("dur", seq_along(r)))
}
dat.dur.long <- ddply(dat, .(id), dur.fn)
dat.dur <- dcast(dat.dur.long, id ~ dur.seq, c, value.var = "dur.value", fill = 0)
cbind(dat.s, dat.dur[-1])
which gives:
id cohort s1 s2 s3 s4 s5 dur1 dur2 dur3 dur4
1 1 1 2 2 1 4 0 2 1 1 0
2 2 3 1 1 3 0 0 2 1 0 0
3 3 2 1 2 3 3 4 1 1 2 1
I am trying to reshape the following dataset with reshape(), without much results.
The starting dataset is in "wide" form, with each id described through one row. The dataset is intended to be adopted for carry out Multistate analyses (a generalization of Survival Analysis).
Each person is recorded for a given overall time span. During this period the subject can experience a number of transitions among states (for simplicity let us fix to two the maximum number of distinct states that can be visited). The first visited state is s1 = 1, 2, 3, 4. The person stays within the state for dur1 time periods, and the same applies for the second visited state s2:
id cohort s1 dur1 s2 dur2
1 1 3 4 2 5
2 0 1 4 4 3
The dataset in long format which I woud like to obtain is:
id cohort s
1 1 3
1 1 3
1 1 3
1 1 3
1 1 2
1 1 2
1 1 2
1 1 2
1 1 2
2 0 1
2 0 1
2 0 1
2 0 1
2 0 4
2 0 4
2 0 4
In practice, each id has dur1 + dur2 rows, and s1 and s2 are melted in a single variable s.
How would you do this transformation? Also, how would you cmoe back to the original dataset "wide" form?
Many thanks!
dat <- cbind(id=c(1,2), cohort=c(1, 0), s1=c(3, 1), dur1=c(4, 4), s2=c(2, 4), dur2=c(5, 3))
You can use reshape() for the first step, but then you need to do some more work. Also, reshape() needs a data.frame() as its input, but your sample data is a matrix.
Here's how to proceed:
reshape() your data from wide to long:
dat2 <- reshape(data.frame(dat), direction = "long",
idvar = c("id", "cohort"),
varying = 3:ncol(dat), sep = "")
dat2
# id cohort time s dur
# 1.1.1 1 1 1 3 4
# 2.0.1 2 0 1 1 4
# 1.1.2 1 1 2 2 5
# 2.0.2 2 0 2 4 3
"Expand" the resulting data.frame using rep()
dat3 <- dat2[rep(seq_len(nrow(dat2)), dat2$dur), c("id", "cohort", "s")]
dat3[order(dat3$id), ]
# id cohort s
# 1.1.1 1 1 3
# 1.1.1.1 1 1 3
# 1.1.1.2 1 1 3
# 1.1.1.3 1 1 3
# 1.1.2 1 1 2
# 1.1.2.1 1 1 2
# 1.1.2.2 1 1 2
# 1.1.2.3 1 1 2
# 1.1.2.4 1 1 2
# 2.0.1 2 0 1
# 2.0.1.1 2 0 1
# 2.0.1.2 2 0 1
# 2.0.1.3 2 0 1
# 2.0.2 2 0 4
# 2.0.2.1 2 0 4
# 2.0.2.2 2 0 4
You can get rid of the funky row names too by using rownames(dat3) <- NULL.
Update: Retaining the ability to revert to the original form
In the example above, since we dropped the "time" and "dur" variables, it isn't possible to directly revert to the original dataset. If you feel this is something you would need to do, I suggest keeping those columns in and creating another data.frame with the subset of the columns that you need if required.
Here's how:
Use aggregate() to get back to "dat2":
aggregate(cbind(s, dur) ~ ., dat3, unique)
# id cohort time s dur
# 1 2 0 1 1 4
# 2 1 1 1 3 4
# 3 2 0 2 4 3
# 4 1 1 2 2 5
Wrap reshape() around that to get back to "dat1". Here, in one step:
reshape(aggregate(cbind(s, dur) ~ ., dat3, unique),
direction = "wide", idvar = c("id", "cohort"))
# id cohort s.1 dur.1 s.2 dur.2
# 1 2 0 1 4 4 3
# 2 1 1 3 4 2 5
There are probably better ways, but this might work.
df <- read.table(text = '
id cohort s1 dur1 s2 dur2
1 1 3 4 2 5
2 0 1 4 4 3',
header=TRUE)
hist <- matrix(0, nrow=2, ncol=9)
hist
for(i in 1:nrow(df)) {
hist[i,] <- c(rep(df[i,3], df[i,4]), rep(df[i,5], df[i,6]), rep(0, (9 - df[i,4] - df[i,6])))
}
hist
hist2 <- cbind(df[,1:2], hist)
colnames(hist2) <- c('id', 'cohort', paste('x', seq_along(1:9), sep=''))
library(reshape2)
hist3 <- melt(hist2, id.vars=c('id', 'cohort'), variable.name='x', value.name='state')
hist4 <- hist3[order(hist3$id, hist3$cohort),]
hist4
hist4 <- hist4[ , !names(hist4) %in% c("x")]
hist4 <- hist4[!(hist4[,2]==0 & hist4[,3]==0),]
Gives:
id cohort state
1 1 1 3
3 1 1 3
5 1 1 3
7 1 1 3
9 1 1 2
11 1 1 2
13 1 1 2
15 1 1 2
17 1 1 2
2 2 0 1
4 2 0 1
6 2 0 1
8 2 0 1
10 2 0 4
12 2 0 4
14 2 0 4
Of course, if you have more than two states per id then this would have to be modified (and it might have to be modified if you have more than two cohorts). For example, I suppose with 9 sample periods one person could be in the following sequence of states:
1 3 2 4 3 4 1 1 2