Lets say I have these data set:
library(data.table)
mydata <- data.table(year=1991:2000,
z=c(0,0,1,1,1,1,1,0,0,0))
If I call the dataset, it will look something like this:
mydata
year z
1: 1991 0
2: 1992 0
3: 1993 1
4: 1994 1
5: 1995 1
6: 1996 1
7: 1997 1
8: 1998 0
9: 1999 0
10: 2000 0
What I need is:
A transition variable, call it c. If I had these dataset, it would look something like this:
year z c
1: 1991 0 0
2: 1992 0 0
3: 1993 1 1
4: 1994 1 NA
5: 1995 1 NA
6: 1996 1 NA
7: 1997 1 NA
8: 1998 0 0
9: 1999 0 0
10: 2000 0 0
Essentially, c marks when there has been a transition in variable z, from z=0 to z=1. When it does, c puts a 1 just once and then starts putting NA's until it returns to the original state (z=0). Then, it starts putting zeroes.
I have another id variable, but that would complicate the example. I think I can manage that part.
** EDITED **: In fact, it does not matter whether I have an id variable or not.
It sounds easy, but not being a R expert myself, it's killing me!
You can use rleid to create a group variable, and then replace duplicated 1 in z with NA using ifelse statement:
mydata[, c := ifelse(duplicated(z) & z == 1, NA_integer_, z), by = rleid(z)][]
# year z c
# 1: 1991 0 0
# 2: 1992 0 0
# 3: 1993 1 1
# 4: 1994 1 NA
# 5: 1995 1 NA
# 6: 1996 1 NA
# 7: 1997 1 NA
# 8: 1998 0 0
# 9: 1999 0 0
#10: 2000 0 0
Another attempt:
mydata[, c := z]
mydata[c==1, c := replace(c,-1,NA), by=rleid(z)]
# year z c
# 1: 1991 0 0
# 2: 1992 0 0
# 3: 1993 1 1
# 4: 1994 1 NA
# 5: 1995 1 NA
# 6: 1996 1 NA
# 7: 1997 1 NA
# 8: 1998 0 0
# 9: 1999 0 0
#10: 2000 0 0
library(data.table)
mydata <- data.table(year=1991:2000, z=c(0,0,1,1,1,1,1,0,0,0))
mydata[,c:=ifelse(z!=shift(z, type="lag"), 1, 0)]
mydata[1,]$c = 0
check this function in data.table shift(x, n=1L, fill=NA, type=c("lag", "lead"), give.names=FALSE) Shift function shifts x in both "lead" and "lag" direction. N is the number of steps. In the first comparison, NA is generated that has been changed to 0 in the last line. You can read ?shift in your session and read more about this function.
Related
I am looking for a way to fill in a column in R based on values in a different column. Below is what my data looks like.
year
action
player
end
2001
1
Mike
2003
2002
0
Mike
NA
2003
0
Mike
NA
2004
0
Mike
NA
2001
0
Alan
NA
2002
0
Alan
NA
2003
1
Alan
2004
2004
0
Alan
NA
I would like to either change the "action" column or create a new column such that it reflects the duration between the "year" and "end" variables. Below is what it would look like:
year
action
player
end
2001
1
Mike
2003
2002
1
Mike
NA
2003
1
Mike
NA
2004
0
Mike
NA
2001
0
Alan
NA
2002
0
Alan
NA
2003
1
Alan
2004
2004
1
Alan
NA
I have tried to do this with the following loop:
i <- 0
z <- 0
for (i in 1:nrow(df)){
i <- z + i + 1
if (df[i, 2] == 0) {}
else {df[i, 5] = (df[i, 4] - df[i, 1])}
z <- df[i,5]
for (z in i:nrow(df)){df[i, 2] = 1}
}
Here, my i value is skyrocketing, breaking the loop. I am not sure why that is occuring. I'd be interested to either know how to fix my approach or how to do this in a smarter fashion.
There's no need for explicit loops here.
First group your data frame by player. Then find the rows where the cumulative sum (cumsum) of action is greater than 0 and the year is less than or equal to the end year of the group. If the row meets these conditions, set action to 1, otherwise to 0.
Using the dplyr package you could achieve this in a couple of lines:
library(dplyr)
df %>%
group_by(player) %>%
mutate(action = as.numeric(cumsum(action) > 0 & year <= na.omit(end)[1]))
#> # A tibble: 8 x 4
#> # Groups: player [2]
#> year action player end
#> <int> <dbl> <chr> <int>
#> 1 2001 1 Mike 2003
#> 2 2002 1 Mike NA
#> 3 2003 1 Mike NA
#> 4 2004 0 Mike NA
#> 5 2001 0 Alan NA
#> 6 2002 0 Alan NA
#> 7 2003 1 Alan 2004
#> 8 2004 1 Alan NA
I need to calculate a penalized cumulative sum.
Individuals "A", "B" and "C" were supposed to get tested every other year. Every time they get tested, they accumulate 1 point. However, when they miss a test, their cumulative score gets deducted in 1.
I have the following code:
data.frame(year = rep(1990:1995, 3), person.id = c(rep("A", 6), rep("B", 6), rep("C", 6)), needs.testing = rep(c("Yes", "No"), 9), test.compliance = c(c(1,0,1,0,1,0), c(1,0,1,0,0,0), c(1,0,0,0,0,0)), penalized.compliance.cum.sum = c(c(1,1,2,2,3,3), c(1,1,2,2,1,1), c(1,1,0,0,-1,-1)))
...which gives the following:
year person.id needs.testing test.compliance penalized.compliance.cum.sum
1 1990 A Yes 1 1
2 1991 A No 0 1
3 1992 A Yes 1 2
4 1993 A No 0 2
5 1994 A Yes 1 3
6 1995 A No 0 3
7 1990 B Yes 1 1
8 1991 B No 0 1
9 1992 B Yes 1 2
10 1993 B No 0 2
11 1994 B Yes 0 1
12 1995 B No 0 1
13 1990 C Yes 1 1
14 1991 C No 0 1
15 1992 C Yes 0 0
16 1993 C No 0 0
17 1994 C Yes 0 -1
18 1995 C No 0 -1
As it is evident, "A" fully complied. "B" somewhat complied (in year 1994 he's supposed to get tested, but he missed the test, and consequently his cumulative sum gets deducted from 2 to 1). Finally, "C" complies just once (in year 1990, and every time she needs to get tested, she misses the test).
What I need is some code to get the "penalized.compliance.cum.sum" variable.
Please note:
Tests are every other year.
The "penalized.compliance.cum.sum" variable keeps adding the previous score.
But starts deducting only if the individual misses the test on the testing year (denoted in the "needs.testing" variable).
For instance, individual "C" complies in year 1990. In 1991 she doesn't need to get tested, and hence keeps her score of 1. Then, she misses the 1992 test, and 1 is subtracted from her cumulative score, getting a score of 0 in 1992. Then she keeps missing test getting a -1 at the end of the study.
Also, I need to assign different penalties (i.e. different numbers). In this example, it's just 1. However, I need to be able to penalize using other numbers such as 0.5, 0.1, and others.
Thanks!
Using case_when
library(dplyr)
df1 %>%
group_by(person.id) %>%
mutate(res = cumsum(case_when(needs.testing == "Yes" ~ 1- 2 *(test.compliance < 1), TRUE ~ 0)))
base R
do.call(rbind, by(dat, dat$person.id,
function(z) transform(z, res = cumsum(ifelse(needs.testing == "Yes", 1-2*(test.compliance < 1), 0)))
))
# year person.id needs.testing test.compliance penalized.compliance.cum.sum res
# A.1 1990 A Yes 1 1 1
# A.2 1991 A No 0 1 1
# A.3 1992 A Yes 1 2 2
# A.4 1993 A No 0 2 2
# A.5 1994 A Yes 1 3 3
# A.6 1995 A No 0 3 3
# B.7 1990 B Yes 1 1 1
# B.8 1991 B No 0 1 1
# B.9 1992 B Yes 1 2 2
# B.10 1993 B No 0 2 2
# B.11 1994 B Yes 0 1 1
# B.12 1995 B No 0 1 1
# C.13 1990 C Yes 1 1 1
# C.14 1991 C No 0 1 1
# C.15 1992 C Yes 0 0 0
# C.16 1993 C No 0 0 0
# C.17 1994 C Yes 0 -1 -1
# C.18 1995 C No 0 -1 -1
by splits a frame up by the INDICES (dat$person.id here), where in the function z is the data for just that group. This allows us to operate on the data without fearing the person changing in a vector.
by returns a list, and the canonical base-R way to combine lists into a frame is either rbind(a, b) when only two frames, or do.call(rbind, list(...)) when there may be more than two frames in the list.
The 1-2*(.) is just a trick to waffle between +1 and -1 based on test.compliance.
This has the side-effect of potentially changing the order of the rows. For instance, if it were ordered first by year then person.id, then the by-group calculations will still be good, but the output will be grouped by person.id (and ordered by year within the group). Minor, but note it if you need order to be something.
dplyr
library(dplyr)
dat %>%
group_by(person.id) %>%
mutate(res = cumsum(if_else(needs.testing == "Yes", 1-2*(test.compliance < 1), 0))) %>%
ungroup()
data.table
library(data.table)
datDT <- as.data.table(dat)
datDT[, res := cumsum(fifelse(needs.testing == "Yes", 1-2*(test.compliance < 1), 0)), by = .(person.id)]
This might do the trick for you?
df <- data.frame(year = rep(1990:1995, 3), person.id = c(rep("A", 6), rep("B", 6), rep("C", 6)), needs.testing = rep(c("Yes", "No"), 9), test.compliance = c(c(1,0,1,0,1,0), c(1,0,1,0,0,0), c(1,0,0,0,0,0)), penalized.compliance.cum.sum = c(c(1,1,2,2,3,3), c(1,1,2,2,1,1), c(1,1,0,0,-1,-1)))
library("dplyr")
penalty <- -1
df %>%
group_by(person.id) %>%
mutate(cumsum = cumsum(ifelse(needs.testing == "Yes" & test.compliance == 0, penalty, test.compliance)))
## A tibble: 18 x 6
## Groups: person.id [3]
# year person.id needs.testing test.compliance penalized.compliance.cum.sum cumsum
# <int> <chr> <chr> <dbl> <dbl> <dbl>
# 1 1990 A Yes 1 1 1
# 2 1991 A No 0 1 1
# 3 1992 A Yes 1 2 2
# 4 1993 A No 0 2 2
# 5 1994 A Yes 1 3 3
# 6 1995 A No 0 3 3
# 7 1990 B Yes 1 1 1
# 8 1991 B No 0 1 1
# 9 1992 B Yes 1 2 2
#10 1993 B No 0 2 2
#11 1994 B Yes 0 1 1
#12 1995 B No 0 1 1
#13 1990 C Yes 1 1 1
#14 1991 C No 0 1 1
#15 1992 C Yes 0 0 0
#16 1993 C No 0 0 0
#17 1994 C Yes 0 -1 -1
#18 1995 C No 0 -1 -1
You can then easily adjust the penalty variable to be whatever penalty you want.
I have several hundred data sets that cover several hundred variables for the period from 1875 to 2020. However, there are not the same number of entries for each year, or even none at all, so I would like to adjust the data sets.
Specifically, I would like to have the same number of rows for each year, with the added series for each year containing only NAs. If the year with the most entries has 5 rows in the data set, then all years should have 5 rows in the data set. If a year is not yet included in the data set, it would have to be added with the corresponding number of rows and NAs for all variables.
Due to the size of the data sets I would like to work with data.tables, but I have no idea how to solve this problem in an efficient way using data.table coding. My previous attempts were mainly loop combinations, which made the processing extremely slow. For your orientation, here is a minimal example of the data set structure. Any kind of help is deeply appreciated.
First <- 1875; Last <- 2020
Year <- c(1979,1979,1979,1982,1987,1987,1987,1988,1989,1990,1993,1995,1997,1997,1998,1999,2000)
Sums <- c(0.30,1.47,4.05,1.30,1.42,1.86,1.29,0.97,1.54,0.46,0.67,0.98,1.73,0.74,1.70,0.95,0.90)
Days <- c(3,4,3,5,3,3,3,3,7,3,8,10,3,3,3,3,3)
Data <- data.table(Year=Year, Sums=Sums, Days=Days)
Ideally, the procedure would output a data.table with a similar pattern. For reasons of readability, the data set does not start with 1875, but 1975.
Year Sums Days
1: 1979 0.30 3 # 1979 has the most observations in the data.table
2: 1979 1.47 4
3: 1979 4.05 3
4: 1982 1.30 5
5: 1982 1.42 3
6: 1982 NA NA # New observation
7: 1987 1.86 3
8: 1987 1.29 3
9: 1987 0.97 3
10: 1988 1.54 7
11: 1988 NA NA # New observation
12: 1988 NA NA # New observation
13: 1989 0.46 3
14: 1989 NA NA # New observation
15: 1989 NA NA # New observation
16: 1990 0.67 8
17: 1990 NA NA # New obeservation
18: 1990 NA NA # New obeservation
19: 1991 NA NA # New observation for 1991; year wasn't included previously
20: 1991 NA NA # New observation for 1991; year wasn't included previously
21: 1991 NA NA # New observation; year wasn't included
22: 1992 NA NA # New observation; year wasn't included
23: 1992 NA NA # New observation; year wasn't included
24: 1992 NA NA # New observation; year wasn't included
25: 1993 0.98 10
26: 1993 NA NA # New observation
27: 1993 NA NA # New observation
28: 1994 NA NA # New observation; year wasn't included
29: 1994 NA NA # New observation; year wasn't included
30: 1994 NA NA # New observation; year wasn't included
31: 1995 1.73 3
32: 1995 NA NA # New obeservations
33: 1995 NA NA # New obeservations
..................
Another data.table option:
Data[, ri := rowid(Year)][
CJ(Year=seq(min(Year), max(Year), by=1L), ri=seq.int(max(ri))), on=.NATURAL]
Or for a specific range (First to Last):
Data[, ri := rowid(Year)][
CJ(Year=First:Last, ri=seq.int(max(ri))), on=.NATURAL]
n <- max(table(Data$Year))
setkey(Data, Year)
Data2 <- Data[J(First:Last), .SD[1:n], by = .EACHI]
Or without setting key (thanks to chinsoon12):
Data2 <- Data[J(Year = First:Last), on = .NATURAL, .SD[1:n], by = .EACHI]
Example output:
Data2[Year %between% c(1996L, 1999L)]
# Year Sums Days
# 1: 1996 NA NA
# 2: 1996 NA NA
# 3: 1996 NA NA
# 4: 1997 1.73 3
# 5: 1997 0.74 3
# 6: 1997 NA NA
# 7: 1998 1.70 3
# 8: 1998 NA NA
# 9: 1998 NA NA
# 10: 1999 0.95 3
# 11: 1999 NA NA
# 12: 1999 NA NA
We can find the most number of rows for a particular year using table function. We can then use complete to include all the incomplete observations from First to Last year with each year having n rows.
library(dplyr)
library(tidyr)
n <- max(table(Data$Year))
Data %>%
group_by(Year) %>%
mutate(row = row_number()) %>%
ungroup %>%
complete(Year = First:Last, row = 1:n, fill = list(Sums = 0, Days = 0))
# A tibble: 438 x 4
# Year row Sums Days
# <dbl> <int> <dbl> <dbl>
# 1 1875 1 0 0
# 2 1875 2 0 0
# 3 1875 3 0 0
# 4 1876 1 0 0
# 5 1876 2 0 0
# 6 1876 3 0 0
# 7 1877 1 0 0
# 8 1877 2 0 0
# 9 1877 3 0 0
#10 1878 1 0 0
# … with 428 more rows
i have a data frame like that
library(data.table)
mydata<-
data.table(comname=c("hon","hon","hon","acer","acer","acer","acer","acer","acer"),
oversea=c(1,0,1,1,0,1,1,1,0),
year=c(1991,1992,1993,1981,1982,1983,1983,1984,1985),
hopecount=c(0,0,1,0,0,1,1,2,2))
comname oversea year hopecount
1: hon 1 1991 0
2: hon 0 1992 0
3: hon 1 1993 1
4: acer 1 1981 0
5: acer 0 1982 0
6: acer 1 1983 1
7: acer 1 1983 1
8: acer 1 1984 2
9: acer 0 1985 2
i count the occurrence of comname conditional on oversea==1 and start with zero:
mydata[oversea==1, mycount := match(year, unique(year))-1, comname];mydata
i hope to get mycount=hopecount ,but mycount will be NA when oversea==0
is there any way to let oversea==0 "no count" and fill in "previous count time" instead of "NA".
just like the form of hopecount
thx alot ^^"
We can use na.locf from zoo to replace the NA elements with the previous non-NA adjacent element
library(data.table)
library(zoo)
mydata[oversea==1, hopecount2 := match(year, unique(year))-1, comname
][, hopecount2 := na.locf(hopecount2), comname]
identical(mydata$hopecount, mydata$hopecount2)
#[1] TRUE
I am trying to create a "long" data frame of indicator ("dummy") variables out of a very peculiar type of "wide" data frame in R that has interval ranges of years defining my data.
What I have looks like this:
f=data.frame(name=c("A","B","C"),
year.start=c(1990,1994,1993),year.end=c(1994,1995,1993))
name year.start year.end
1 A 1990 1994
2 B 1994 1995
3 C 1993 1993
Update: I have changed the value of year.start for A to 1990 from the initial example of 1993 to address some of the answers below which rely on unique values instead of intervals.
What I would like is a long data frame that would look like this, with an entry for each of the possible years in the original data frame, eg, 1990 through 1995 where 1 = present and 0 = absent.
name year indicator
A 1990 1
A 1991 1
A 1992 1
A 1993 1
A 1994 1
A 1995 0
B 1990 0
B 1991 0
B 1992 0
B 1993 0
B 1994 1
B 1995 1
C 1990 0
C 1991 0
C 1992 0
C 1993 1
C 1994 0
C 1995 0
Try as I might, I don't see how I can do this with Hadley Wickham's reshape2 package.
Thanks!
Someone else might have suggestion for reshape2, but here is a base R solution:
years <- factor(unlist(f[-1]), levels=seq(min(f[-1]), max(f[-1]), by=1))
result <- data.frame(table(years, rep(f[[1]], length.out=length(years))))
# years Var2 Freq
# 1 1990 A 1
# 2 1991 A 0
# 3 1992 A 0
# 4 1993 A 0
# 5 1994 A 1
# 6 1995 A 0
# 7 1990 B 0
# 8 1991 B 0
# 9 1992 B 0
# 10 1993 B 0
# 11 1994 B 1
# 12 1995 B 1
# 13 1990 C 0
# 14 1991 C 0
# 15 1992 C 0
# 16 1993 C 2
# 17 1994 C 0
# 18 1995 C 0
here is a step-by-step breakdown, using data.table
library(data.table)
f <- as.data.table(f)
## ALL OF NAME-YEAR COMBINATIONS
ALL <- f[, CJ(name=name, year=seq(min(year.start), max(year.end)))]
## WHICH COMBINATIONS EXIST
PRESENT <- f[, list(year = seq(year.start, year.end)), by=name]
## SETKEYS FOR MERGING
setkey(ALL, name, year)
setkey(PRESENT, name, year)
## INITIALIZE INDICATOR TO ZERO, THEN SET TO 1 FOR THOSE PRESENT
ALL[, indicator := 0]
ALL[PRESENT, indicator := 1]
ALL
name year indicator
1: A 1993 1
2: A 1994 1
3: A 1995 0
4: B 1993 0
5: B 1994 1
6: B 1995 1
7: C 1993 1
8: C 1994 0
9: C 1995 0
Here's another solution, similar to the ones above, which aims to be straightforward:
zz <- cbind(name=f[1],year=rep(min(f[-1]):max(f[-1]),each=nrow(f)))
zz$indicator <- as.numeric((f$name==zz$name &
f$year.start<=zz$year &
f$year.end >=zz$year))
result <- zz[order(zz$name,zz$year),]
The first line builds a template with all the names and all the years. The second line sets indicator based on whether it is present in the range. The third line just reorders the result.
Another base R solution
f=data.frame(name=c("A","B","C"),
year.start=c(1993,1994,1993),year.end=c(1994,1995,1993), stringsAsFactors=F)
x <- expand.grid(unique(f$name),min(f1$year):max(f1$year))
names(x) <- c("name", "year")
x$indicator <- sapply(1:nrow(x), function(i) sum(x$name[i]==f$name & x$year[i] >= f$year.start & x$year[i] <= f$year.end))
x[order(x$name),]