I have data like this:
test <- data.frame(id = c(1,2,1,5,5,5,6),
time = c(0,1,4,5,6,7,9),
cond = c("a","a","b","a","b","b","b"),
value = c(5,3,2,4,0,3,1),
stringsAsFactors=F)
setDT(test)[,order := order(time),id][order(id,order)]
id time cond value order
1 0 a 5 1
2 1 a 3 1
1 4 b 2 2
5 5 a 4 1
5 6 b 0 2
5 7 b 3 3
6 9 b 1 1
The data.table function creates a column "order" which is the order of time based on the group id.
I would like to create a column which returns the previous value but only where the condition is "b". If the condition is "a" return the current value and if the condition is "b" and the previous is "b" then skip to the next non "b". If the first condition of a group is "b" Then return NA.
Desired output would be like this:
id time cond value order prev
1 0 a 5 1 5
2 1 a 3 1 3
1 4 b 2 2 5
5 5 a 4 1 4
5 6 b 0 2 4
5 7 b 3 3 4
6 9 b 1 1 NA
I've tried some functions like this but only returned NAs.
test[, prev := shift(value[cond == 'b']), .(id,order)]
If I understood the problem correctly, one option could be:
library(data.table)
setDT(test)[, order := order(time), id][order(id, order)]
test[, prev := {
frst <- ifelse(cond[1] == "a", value[1],
ifelse(cond[1] == "b", NA, cond[1]))
prev <- as.integer(ifelse(cond == "b" & shift(cond) == "b",
NA,
c(frst, shift(value)[-1])))
}, by = id][cond == "b", prev := zoo::na.locf(prev), by = id]
Output:
id time cond value order prev
1: 1 0 a 5 1 5
2: 1 4 b 2 2 5
3: 2 1 a 3 1 3
4: 5 5 a 4 1 4
5: 5 6 b 0 2 4
6: 5 7 b 3 3 4
7: 6 9 b 1 1 NA
If you assign the non-b values first, zoo:na.locf can do the rest (fill the b (NA) values downwards).
library(zoo)
test[cond != 'b', prev := value]
test[, prev := na.locf(prev), id]
test
# id time cond value order prev
# 1: 1 0 a 5 1 5
# 2: 2 1 a 3 1 3
# 3: 1 4 b 2 2 5
# 4: 5 5 a 4 1 4
# 5: 5 6 b 0 2 4
# 6: 5 7 b 3 3 4
# 7: 6 9 b 1 1 NA
Related
Say I have the following data.table:
library(data.table)
DT <- data.table(
ID = rep(c(1,2,3),4),
day = c(rep(1,3),rep(2,3),rep(3,3),rep(4,3)),
Status = c(rep('A',3),'A','B','B','A','C','B','A','D','C')
)
What I would like to achieve is that for each ID, find number of observations (in this case if sorted by days, the number of day it takes to hit a specific Status. So if I need to do this for Status C, the result would be:
0 for ID 1 (since doesn't contain status C), 3 for ID 2, and 4 for ID 3.
The only way came to my mind was to write a function and do nested for loops, but I am sure there should be much better/faster/more efficient ways.
Appreciate any help.
A possible data.table approach adding one column for the number of days to reach each status (0 if never reached):
library(data.table)
## status id's
status_ids <- unique(DT$Status)
status_cols <- paste("status", status_ids, sep = "_")
## add one column for each status id
setorder(DT, ID, day)
DT[, (status_cols) := lapply(status_ids, \(s) ifelse(any(Status == s), min(day[Status == s]), 0)), by = "ID"]
DT
#> ID day Status status_A status_B status_C status_D
#> 1: 1 1 A 1 0 0 0
#> 2: 1 2 A 1 0 0 0
#> 3: 1 3 A 1 0 0 0
#> 4: 1 4 A 1 0 0 0
#> 5: 2 1 A 1 2 3 4
#> 6: 2 2 B 1 2 3 4
#> 7: 2 3 C 1 2 3 4
#> 8: 2 4 D 1 2 3 4
#> 9: 3 1 A 1 2 4 0
#> 10: 3 2 B 1 2 4 0
#> 11: 3 3 B 1 2 4 0
#> 12: 3 4 C 1 2 4 0
You can split by ID and return the first match of day.
sapply(split(DT[,2:3], DT$ID), \(x) x$day[match("C", x$Status)])
# 1 2 3
#NA 3 4
Does this work:
library(dplyr)
DT %>% left_join(
DT %>% group_by(ID) %>% summarise(col = row_number()[Status == 'C'])
) %>% replace_na(list(col= 0))
`summarise()` has grouped output by 'ID'. You can override using the
`.groups` argument.
Joining, by = "ID"
ID day Status col
1: 1 1 A 0
2: 2 1 A 3
3: 3 1 A 4
4: 1 2 A 0
5: 2 2 B 3
6: 3 2 B 4
7: 1 3 A 0
8: 2 3 C 3
9: 3 3 B 4
10: 1 4 A 0
11: 2 4 D 3
12: 3 4 C 4
I would like to create an accumulative incremental counter that increases only when a condition is met.
DT <- data.table(id = c(1, 1, 1, 1, 1, 1, 1, 2, 2, 2),
b = c(10L, 5L, 3L, 4L, 2L, 6L, 1L, 3L, 5L, 7L))
I don't get the desired result with rleid because when two conditions are met in consecutive rows, the increment is not performed
> DT[,count := rleid(b>=5),id]
> DT
id b count
1: 1 10 1
2: 1 5 1
3: 1 3 2
4: 1 4 2
5: 1 2 2
6: 1 6 3
7: 1 1 4
8: 2 3 1
9: 2 5 2
10: 2 7 2
The expected result is
> DT
id b count
1: 1 10 1
2: 1 5 2
3: 1 3 2
4: 1 4 2
5: 1 2 2
6: 1 6 3
7: 1 1 3
8: 2 3 1
9: 2 5 2
10: 2 7 3
Here is an option with cumsum. Grouped by 'id', get the cumulative sum of logical expression (b >= 5). For 'id' 2, the first element that is greater than or equal to 5 is at position 2 (in the grouped position), thus the first row will be 0. Inorder to make this 1, an option is to convert it to factor and then coerce to integer so that we get the integer storage values (R indexing starts from 1)
DT[, count := as.integer(factor(cumsum(b >= 5))), id]
-output
DT
id b count
1: 1 10 1
2: 1 5 2
3: 1 3 2
4: 1 4 2
5: 1 2 2
6: 1 6 3
7: 1 1 3
8: 2 3 1
9: 2 5 2
10: 2 7 3
Another data.table option with cumsum
> DT[, count := (v <- cumsum(b >= 5)) - v[1] + 1, id][]
id b count
1: 1 10 1
2: 1 5 2
3: 1 3 2
4: 1 4 2
5: 1 2 2
6: 1 6 3
7: 1 1 3
8: 2 3 1
9: 2 5 2
10: 2 7 3
We can also use accumulate function for this purpose. Here are some notes on this solution:
accumulate takes a two argument function as its .f argument where .x is the previous/ accumulated value and .y is the current value in the sequence of values of vector b
I set the initial value of count as 1 thus remove the first value of b cause we don't need it anymore and check the next value by .y and if the condition is met it will be added by one otherwise it remains as is.
library(dplyr)
library(purrr)
DT %>%
group_by(id) %>%
mutate(count = accumulate(b[-1], .init = 1,
~ if(.y >= 5) {
.x + 1
} else {
.x
}))
# A tibble: 10 x 3
# Groups: id [2]
id b count
<dbl> <int> <dbl>
1 1 10 1
2 1 5 2
3 1 3 2
4 1 4 2
5 1 2 2
6 1 6 3
7 1 1 3
8 2 3 1
9 2 5 2
10 2 7 3
Say I have something like this:
ID = c("a","a","a","a","a", "b","b","b","b","b")
Group = c("1","2","3","4","5", "1","2","3","4","5")
Value = c(3, 4,2,4,3, 6, 1, 8, 9, 10)
df<-data.frame(ID,Group,Value)
I want to subtract group=5 from group=3 within the ID, with an output column which has this difference for each ID like so:
ID Group Value Want
1 a 1 3 1
2 a 2 4 1
3 a 3 2 1
4 a 4 4 1
5 a 5 3 1
6 b 1 6 2
7 b 2 1 2
8 b 3 8 2
9 b 4 9 2
10 b 5 10 2
Also, if that calculation cannot be done (i.e. group 5 is missing), NA values for the 'want' column would be ideal.
As there is only one unique 'Group' per 'ID', we can do subsetting
library(dplyr)
df %>%
group_by(ID) %>%
mutate(want = Value[Group == 5] - Value[Group == 3])
# A tibble: 10 x 4
# Groups: ID [2]
# ID Group Value want
# <fct> <fct> <dbl> <dbl>
# 1 a 1 3 1
# 2 a 2 4 1
# 3 a 3 2 1
# 4 a 4 4 1
# 5 a 5 3 1
# 6 b 1 6 2
# 7 b 2 1 2
# 8 b 3 8 2
# 9 b 4 9 2
#10 b 5 10 2
The above can be made more error-proof if we convert to numeric index and get the first element. When there are no TRUE, by using [1], it returns NA
df %>%
slice(-10) %>%
group_by(ID) %>%
mutate(want = Value[which(Group == 5)[1]] - Value[which(Group == 3)[1]])
Or use match which returns an index of NA if there are no matches, and anything with NA index returns NA which will subsequently return NA in subtraction (NA -3)
df %>%
slice(-10) %>% # removing the last row where Group is 10
group_by(ID) %>%
mutate(want = Value[match(5, Group)] - Value[match(3, Group)])
Here is a base R solution
dfout <- Reduce(rbind,
lapply(split(df,df$ID),
function(x) within(x, Want <-diff(subset(Value, Group %in% c("3","5"))))))
such that
> dfout
ID Group Value Want
1 a 1 3 1
2 a 2 4 1
3 a 3 2 1
4 a 4 4 1
5 a 5 3 1
6 b 1 6 2
7 b 2 1 2
8 b 3 8 2
9 b 4 9 2
10 b 5 10 2
A data.table method:
library(data.table)
setDT(df)[, want := (Value[Group == 5] - Value[Group == 3]), by = .(ID)]
df
# ID Group Value want
# 1: a 1 3 1
# 2: a 2 4 1
# 3: a 3 2 1
# 4: a 4 4 1
# 5: a 5 3 1
# 6: b 1 6 2
# 7: b 2 1 2
# 8: b 3 8 2
# 9: b 4 9 2
# 10: b 5 10 2
Here is a solution using base R.
unsplit(
lapply(
split(df, df$ID),
function(d) {
x5 = d$Value[d$Group == "5"]
x5 = ifelse(length(x5) == 1, x5, NA)
x3 = d$Value[d$Group == "3"]
x3 = ifelse(length(x3) == 1, x3, NA)
d$Want = x5 - x3
d
}),
df$ID)
Part 1 I have the following data table. I want to make a new column which contains the number of occurrences of each id where there is any style value, except for NA. The main problem is that I don't know how to deal with the NA. Currently, when NA is present I get a frequency of 1.
id style
1 A
1 A
2 A
2 B
3 NA
4 A
4 C
5 NA
I tried using the following, but it still counts NA values
dt[, allele_count := .N, by = list(pat_id, style)]
The desired data table would be as follows:
id style count
1 A 2
1 A 2
2 A 2
2 B 2
3 NA 0
4 A 4
4 B 4
4 B 4
4 C 4
5 NA 0
Part2 I would also like to be able to add another column which would have the number of occurrences of each id with a certain style value.
id style count2
1 A 2
1 A 2
2 A 1
2 B 1
3 NA 0
4 A 1
4 B 2
4 B 2
4 C 1
5 NA 0
Bonus Question: Instead of looking at how many times an id occurs with a given style value as in Part2 , How can you calculate the number of different style values for each id, as follows.
id style count3
1 A 1
1 A 1
2 A 2
2 B 2
3 NA 0
4 A 3
4 B 3
4 B 3
4 C 3
5 NA 0
Here's one possibility. Basically we use a row subset to assign the new columns, then replace the NA values in all three new columns with zero at the end.
nna <- !is.na(dt$style) ## so we don't have to call it four times
dt[nna, count := .N, by = id][nna, count2 := .N, by = .(id, style)][
nna, count3 := uniqueN(style), by = id][!nna, names(dt)[3:5] := 0L]
which results in
id style count count2 count3
1: 1 A 2 2 1
2: 1 A 2 2 1
3: 2 A 2 1 2
4: 2 B 2 1 2
5: 3 NA 0 0 0
6: 4 A 2 1 2
7: 4 C 2 1 2
8: 5 NA 0 0 0
Or you can simplify this down to the following, then reorder the columns if necessary.
dt[nna, c("count", "count3") := .(.N, uniqueN(style)), by = id][
nna, count2 := .N, by = .(id, style)][!nna, names(dt)[3:5] := 0L]
Note that this method is very similar to the other posted answer. I am not sure which of the two is the preferred method, row subset or if() statement.
This is a possible solution :
library(data.table)
dt <- data.table(id=c(1,1,2,2,3,4,4,5),
style=c('A','A','A','B',NA,'A','C',NA))
# count = number of ids having ALL styles defined
dt[, count := if(any(is.na(style))) 0L else .N, by = id]
# count2 = number of id-style occurrences (0 if style = NA)
dt[, count2 := if(is.na(style)) 0L else .N, by = .(id, style)]
> dt
id style count count2
1: 1 A 2 2
2: 1 A 2 2
3: 2 A 2 1
4: 2 B 2 1
5: 3 NA 0 0
6: 4 A 2 1
7: 4 C 2 1
8: 5 NA 0 0
BONUS:
dt[, count3 := uniqueN(na.omit(style)), by = id]
> dt
id style count count2 count3
1: 1 A 2 2 1
2: 1 A 2 2 1
3: 2 A 2 1 2
4: 2 B 2 1 2
5: 3 NA 0 0 0
6: 4 A 2 1 2
7: 4 C 2 1 2
8: 5 NA 0 0 0
I have a data set with observations that are both grouped and ordered (by rank). I'd like to add a third variable that is a count of the number of observations for each grouping variable. I'm aware of ways to group and count variables but I can't find a way to re-insert these counts back into the original data set, which has more rows. I'd like to get the variable C in the example table below.
A B C
1 1 3
1 2 3
1 3 3
2 1 4
2 2 4
2 3 4
2 4 4
Here's one way using ave:
DF <- within(DF, {C <- ave(A, A, FUN=length)})
# A B C
# 1 1 1 3
# 2 1 2 3
# 3 1 3 3
# 4 2 1 4
# 5 2 2 4
# 6 2 3 4
# 7 2 4 4
Here is one approach using data.table that makes use of .N, which is described in the help file to "data.table" as .N is an integer, length 1, containing the number of rows in the group.
> library(data.table)
> DT <- data.table(A = rep(c(1, 2), times = c(3, 4)), B = c(1:3, 1:4))
> DT
A B
1: 1 1
2: 1 2
3: 1 3
4: 2 1
5: 2 2
6: 2 3
7: 2 4
> DT[, C := .N, by = "A"]
> DT
A B C
1: 1 1 3
2: 1 2 3
3: 1 3 3
4: 2 1 4
5: 2 2 4
6: 2 3 4
7: 2 4 4