I recently tried to match adjacent identical rows in a dataframe based on two variables (Condition1 and Outcome1 below). I have seen people doing this with all rows but not with adjacent rows, which is why I developed the following three-step work-around (which I hope did not overthink things):
-I lagged the variables based on which I wanted the matching to be done.
-I compared the variables and lagged-variables
-I deleted all rows in which both ware identical (and removed the remaining unnecessary columns).
Case <- c("Case 1", "Case 2", "Case 3", "Case 4", "Case 5")
Condition1 <- c(0, 1, 0, 0, 1)
Outcome1 <- c(0, 0, 0, 0, 1)
mwa.df <- data.frame(Case, Condition1, Outcome1)
new.df <- mwa.df
Condition_lag <- c(new.df$Condition1[-1],0)
Outcome_lag <- c(new.df$Outcome1[-1],0)
new.df <- cbind(new.df, Condition_lag, Outcome_lag)
new.df$Comp <- 0
new.df$Comp[new.df$Outcome1 == new.df$Outcome_lag & new.df$Condition1 == new.df$Condition_lag] <- 1
new.df <- subset(new.df, Comp == 0)
new.df <- subset(new.df, select = -c(Condition_lag, Outcome_lag, Comp))
This worked just fine. But when I tried to create a function for this because I had to do this operation with a large number of data frames, I encountered the problem that the lag did not work (i.e. the condition_lag <- c(new.df$condition[-1],0) and outcome_lag <- c(new.df$outcome[-1],0) operations were not carried out). The function code was:
FLC.Dframe <- function(old.df, condition, outcome){
new.df <- old.df
condition_lag <- c(new.df$condition[-1],0)
outcome_lag <- c(new.df$outcome[-1],0)
new.df <- cbind(new.df, condition_lag, outcome_lag)
new.df$comp <- 0
new.df$comp[new.df$outcome == new.df$outcome_lag & new.df$condition == new.df$condition_lag] <- 1
new.df <- subset(new.df, comp == 0)
new.df <- subset(new.df, select = -c(condition_lag, outcome_lag, comp))
return(new.df)
}
As for using the function, I wrote new.df <- FLC.Dframe(mwa.df, Condition1, Outcome1).
Could someone help me with this? Many thanks in advance.
Just generate run-length ids and remove the duplicates.
with(mwa.df, mwa.df[!duplicated(data.table::rleid(Condition1, Outcome1)), ])
Output
Case Condition1 Outcome1
1 Case 1 0 0
2 Case 2 1 0
3 Case 3 0 0
5 Case 5 1 1
If you want a function, then
FLC.Dframe <- function(df, cols) df[!duplicated(data.table::rleidv(df[, cols])), ]
Call this function like this
> FLC.Dframe(mwa.df, c("Condition1", "Outcome1"))
Case Condition1 Outcome1
1 Case 1 0 0
2 Case 2 1 0
3 Case 3 0 0
5 Case 5 1 1
The main problem with your function concerns the incorrect usage of $. This operator treats RHS input as is. For example, in this line new.df$condition the $ operator attempts to find in new.df a column named "condition", but not "Condition1", which is the value of condition. If you rewrite your function as follows, then it should work.
FLC.Dframe <- function(old.df, condition, outcome){
new.df <- old.df
condition_lag <- c(new.df[[condition]][-1],0)
outcome_lag <- c(new.df[[outcome]][-1],0)
new.df <- cbind(new.df, condition_lag, outcome_lag)
new.df$comp <- 0
new.df$comp[new.df[[outcome]] == new.df[["outcome_lag"]] & new.df[[condition]] == new.df[["condition_lag"]]] <- 1
new.df <- subset(new.df, comp == 0)
new.df <- subset(new.df, select = -c(condition_lag, outcome_lag, comp))
return(new.df)
}
You also need to call it like this (note that you need to use characters as inputs)
> FLC.Dframe(mwa.df, "Condition1", "Outcome1")
Case Condition1 Outcome1
1 Case 1 0 0
2 Case 2 1 0
4 Case 4 0 0
5 Case 5 1 1
Related
Sorry if this is a trivial question or doesn't make sense, this is my first post. I'm coming from Excel where I've worked with if statements and index match functions and am trying to do something similar in R to pull data from two columns but not necessarily the same row to get a value in a third column, my example is this
df<-data.frame(ID=c(1,5,4,2,3),A=c(1,0,1,1,1),B=c(0,0,1,0,0))
desired output: df<-data.frame(ID=c(1,5,4,2,3),A=c(1,0,1,1,1),B=c(0,0,1,0,0),C=c(0,0,0,0,1))
What I want is to create a third column "C" that essentially follows this format:
Ifelse(A[ID]=1 & B[ID+1]=1 , C[ID]=1 , C[ID]=0)
Essentially if A=1 in ID "x" and B=1 in ID "x+1" then in the new column C in ID "x" =1 otherwise =0. I could order everything by ID if that makes things easier but doing it by the ID column would be ideal.
So far I've tried ifelse statements but I imagine there is probably a better way of doing this
Using dplyr, we can use lead to get next element after arranging the data by ID.
library(dplyr)
df %>%
arrange(ID) %>%
mutate(C = as.integer(A == 1 & lead(B) == 1))
# ID A B C
#1 1 1 0 0
#2 2 1 0 0
#3 3 1 0 1
#4 4 1 1 0
#5 5 0 0 0
In base R, we can do
df1 <- df[order(df$ID),]
df1$C <- with(df1, c(A[-nrow(df)] == 1 & tail(B, -1) == 1, 0))
Without arranging the data, we can probably do
transform(df, C = as.integer(A[ID] == 1 & B[match(ID + 1, ID)] == 1))
Using the lead function I got this to work
df <- df [order(df$ID), ]
df$C <- ifelse (df$A == 1 & lead (df$B) == 1, 1, 0)
I am trying to use the tidyverse (purrr) package to run a for loop across my dataset. I want to check whether some number of conditions are true across certain columns along the dataset. Note, I am trying to become more familiar with tidyverse and its functions rather than rely on Base R.
Here is the code that I want to write a for loop for.
nrow(subset(data, flwr_clstr1>1 & bud_clstr1==0))
nrow(subset(data, flwr_clstr2>1 & bud_clstr2==0))
nrow(subset(data, flwr_clstr3>1 & bud_clstr3==0))
I have columns of data (in this case, it would be flwr_clstr) that are similar, but differ by the last digit. Also, if there is another way to use tidyverse to check these 'conditions', that would be great too.
Here is my attempt at the for loop.
check1 <- vector("double", ncol(data_phen))
for (i in seq_along(data_phen)) {
check[[i]] <- nrow(subset(data, flwr_clstr[[i]]>1 & bud_clstr[[i]]==0))
}
It would be easier to help if you could provide a reproducible example, however I created a sample of what your data might look like based on my understanding.
We can use map2_int from purrr since we are trying to count number of rows in each pair of columns
library(dplyr)
library(purrr)
map2_int(data %>% select(starts_with("flwr_clstr")),
data %>% select(starts_with("bud_clstr")),
~sum(.x > 1 & .y == 0)) %>% unname()
#[1] 2 3 1
However, base R isn't that bad either. This can be solved using mapply
col1 <- grep("^flwr_clstr", names(data))
col2 <- grep("^bud_clstr", names(data))
mapply(function(x, y) sum(x > 1 & y == 0), data[col1], data[col2])
data
Assuming you have equal number of columns for both "flwr_clstr.." and "bud_clstr.."
data <- data.frame(flwr_clstr1 = c(2, 1, 2, 1, 0), flwr_clstr2 = c(2, 2, 2, 1, 0),
flwr_clstr3 = c(1, 1, 2, 1, 1), bud_clstr1 = 0, bud_clstr2 = 0,bud_clstr3 = 0)
which looks like
data
# flwr_clstr1 flwr_clstr2 flwr_clstr3 bud_clstr1 bud_clstr2 bud_clstr3
#1 2 2 1 0 0 0
#2 1 2 1 0 0 0
#3 2 2 2 0 0 0
#4 1 1 1 0 0 0
#5 0 0 1 0 0 0
Suppose, you're given the following dataframe:
a <- data.frame(var = c(",1,2,3,", ",2,3,5,", ",1,3,5,5,"))
What I am looking for is to create the variables flag_1, ..., flag_7 in a containing the information of how many times the respective values occur. For a, I would expect the following result:
var flag_1 flag_2 flag_3 flag_4 flag_5
",1,2,3," 1. 1. 1. 0. 0.
",2,3,5," 0. 1. 1. 0. 1.
",1,3,5,5," 1. 0. 1. 0. 2.
I managed to get the result using a nested for-loop and an if-condition but there must be a nicer (more aesthetic and better performing) solution.
One option would be to do strsplit, get the table and then cbind with original data
cbind(a, do.call(rbind, lapply(strsplit(as.character(a$var), ","),
function(x) table(factor(x[nzchar(x)], levels = 1:5, labels = paste0("flag_", 1:5))))))
# var flag_1 flag_2 flag_3 flag_4 flag_5
#1 ,1,2,3, 1 1 1 0 0
#2 ,2,3,5, 0 1 1 0 1
#3 ,1,3,5,5, 1 0 1 0 2
Another option is with tidyverse
library(tidyverse)
str_extract_all(a$var, "[0-9]") %>%
map(~ as.integer(.x) %>%
as_tibble) %>%
bind_rows(.id = 'grp') %>%
count(grp, value = factor(value, levels = min(value):max(value))) %>%
spread(value, n, drop = FALSE, fill = 0) %>%
select(-grp) %>%
bind_cols(a, .) %>%
rename_at(vars(matches("^[0-9]+$")), ~ paste0("flag_", .))
# var flag_1 flag_2 flag_3 flag_4 flag_5
#1 ,1,2,3, 1 1 1 0 0
#2 ,2,3,5, 0 1 1 0 1
#3 ,1,3,5,5, 1 0 1 0 2
First, don't make the strings into factors. Nothing good comes from that.
a <- data.frame(var = c(",1,2,3,", ",2,3,5,", ",1,3,5,5,"),
stringsAsFactors = FALSE)
To get from strings to your table is simple enough if we take it in small steps. Here, I've written (or renamed) a function per step and then gone through the steps using lapply one at a time. You can string it all together in a pipeline if like, but it would be roughly these steps.
First, I extract the numbers from the strings. That involves splitting on commas, getting rid of empty strings, you have those because you can begin and end a string with a comma, but otherwise, that step wouldn't be necessary. Then we need to translate the strings into numbers, count how often we see each (we can do that with the as.numeric and table functions, respectively), and then it is just a question of mapping the observed counts into a table that also includes those we haven't observed.
pick_indices <- function(str) unlist(strsplit(str, split = ","))
remove_empty <- function(chrs) chrs[nchar(chrs) > 0]
get_indices <- as.numeric
to_counts <- table
to_flag_vect <- function(counts, len) {
vec <- rep(0, len)
names(vec) <- 1:len
vec[names(counts)] <- counts
vec
}
strings <- lapply(a$var, pick_indices)
cleaned <- lapply(strings, remove_empty)
indices <- lapply(cleaned, get_indices)
counts <- lapply(indices, to_counts)
flags <- lapply(counts, to_flag_vect, len = 5)
We now have the flag-counts in a list, so to make it into the table you want, with the column names you want, we simply do this:
tbl <- do.call(rbind, flags)
colnames(tbl) <- paste0("flag_", 1:5)
tbl
Done.
Split and unlist the values into a factor with appropriate levels
x = strsplit(a$var, ",")
xp = factor(unlist(x), levels = seq_len(5))
Create an index that maps the values of xp to the rows they came from
i = rep(seq_along(x), lengths(x))
use xtabs() to cross-tabulate the entries by row
xt = xtabs(~ i + xp)
and cbind() the matrix representation of the result to the original
> cbind(a, unclass(xt))
var 1 2 3 4 5
1 ,1,2,3, 1 1 1 0 0
2 ,2,3,5, 0 1 1 0 1
3 ,1,3,5,5, 1 0 1 0 2
I have to pre-process a big matrix. To make my example easier to understand I will use the following matrix:
Raw data
Where col = people and row = skills
In R my matrix is:
test <- matrix(c(18,12,15,0,13,0,14,0,12),ncol=3, nrow=3)
Aim
In my case I need to process row by row. So there is 3 steps. For each row I have to :
Put 0 if ij=ij (So all diagonals equals zero)
Put 0 if one of the ij=0
Otherwise I have to add ij+ij
I will show the 3 steps to be more clear.
Step 1 (row1)
The data are the row 1
The result is:
Step 2 (row2)
The data are the row 2
The result is:
Step 3 (row3)
The data are the row 3
The result is:
Create a maximum matrix
Then the maximum matching are :
So my final matrix should be:
Question
Can someone tell me how to succeed to achieve this in R?
And of course the same process should work if my matrix has more row and columns...
Thanks a lot :)
Here is my implementation in R. The code doesn't execute the steps exactly in the way you specified them. I focused on your final matrix and assumed that this is the main result you're interested in.
test <- matrix(c(18,12,15,0,13,0,14,0,12),ncol=3, nrow=3)
rownames(test) <- paste("Skill", 1:dim(test)[1], sep="")
colnames(test) <- paste("People", 1:dim(test)[2], sep="")
test
# Pairwise combinations
comb.mat <- combn(1:dim(test)[2], 2)
pairwise.mat <- data.frame(matrix(t(comb.mat), ncol=2))
pairwise.mat$max.score <- 0
names(pairwise.mat) <- c("Person1", "Person2", "Max.Score")
for ( i in 1:dim(comb.mat)[2] ) { # Loop over the rows
first.person <- comb.mat[1,i]
second.person <- comb.mat[2,i]
temp.mat <- test[, c(first.person, second.person)]
temp.mat[temp.mat == 0] <- NA
temp.rowSums <- rowSums(temp.mat, na.rm=FALSE)
temp.rowSums[is.na(temp.rowSums)] <- 0
max.sum <- max(temp.rowSums)
previous.val <- pairwise.mat$Max.Score[pairwise.mat$Person1 == first.person & pairwise.mat$Person2 == second.person]
pairwise.mat$Max.Score[pairwise.mat$Person1 == first.person & pairwise.mat$Person2 == second.person] <- max.sum*(max.sum > previous.val)
}
pairwise.mat
Person1 Person2 Max.Score
1 1 2 25
2 1 3 32
3 2 3 0
person.mat <- matrix(NA, nrow=dim(test)[2], ncol=dim(test)[2])
rownames(person.mat) <- colnames(person.mat) <- paste("People", 1:dim(test)[2], sep="")
diag(person.mat) <- 0
person.mat[cbind(pairwise.mat[,1], pairwise.mat[,2])] <- pairwise.mat$Max.Score
person.mat[lower.tri(person.mat, diag=F)] <- t(person.mat)[lower.tri(person.mat, diag=F)]
person.mat
People1 People2 People3
People1 0 25 32
People2 25 0 0
People3 32 0 0
I am developing a censored dependent variable for use in survival analysis. My goal is to find the last time ("time") that someone answers a question in a survey (e.g. the point where "q.time" is coded as "1", and "q.time+1" and q at all subsequent times are coded as "0").
By this logic, the last question answered should be coded as "1" (q.time). The first question that is NOT answered (q.time+1) should be coded as "0". And all questions subsequent to the first question NOT answered should be coded as "NA". I then want to remove ALL rows where the DV=NA from my dataset.
A very generous coworker has helped me to develop the following code, but he's on vacation now and it needs a little more lovin'. Code is as follows:
library(plyr) # for ddply
library(stats) # for reshape(...)
# From above
dat <- data.frame(
id=c(1, 2, 3, 4),
q.1=c(1, 1, 0, 0),
q.2=c(1, 0, 1, 0),
dv.1=c(1, 1, 1, 1),
dv.2=c(1, 1, 0, 1))
# From above
long <- reshape(dat,
direction='long',
varying=c('q.1', 'q.2', 'dv.1', 'dv.2'))
ddply(long, .(id), function(df) {
# figure out the dropoff time
answered <- subset(df, q == 1)
last.q = max(answered$time)
subs <- subset(df, time <= last.q + 1)
# set all the dv as desired
new.dv <- rep(last.q,1)
if (last.q < max(df$time)) new.dv <- c(0,last.q)
subs$dv <- new.dv
subs
})
Unfortunately, this yields the error message:
"Error in `$<-.data.frame`(`*tmp*`, "dv", value = c(0, -Inf)) :
replacement has 2 rows, data has 0"
Any ideas? The problem seems to be located in the "rep" command, but I'm a newbie to R. Thank you so much!
UPDATE: SEE EXPLANATIONS BELOW, and then REFER TO FOLLOW-UP QUESTION
Hi there-I completely followed you, and really appreciate the time you took to help me out. I went back into my data and coded in a dummy Q where all respondents have a value of "1" - but, discovered where the error really may be. In my real data set, I have 30 questions (i.e., 30 times in long form). After I altered the dataset so FOR SURE q==1 for all id variables, the error message changed to saying
"Error in `$<-.data.frame`(`*tmp*`, "newvar", value = c(0, 29)) : replacement has 2 rows, data has 31"
If the problem is with the number of rows assigned to subs, then is the source of the error coming from...
subs <- subset(df, time <= last.q + 1)
i.e., $time <= last.q + 1$ is setting the number of rows to the value EQUAL to last.q+1?
UPDATE 2: What, ideally, I'd like my new variable to look like!
id time q dv
1 1 1 1
1 2 1 1
1 3 1 1
1 4 1 1
1 5 0 0
1 6 0 NA
2 1 1 1
2 2 1 1
2 3 0 0
2 4 0 NA
2 5 0 NA
2 6 0 NA
Please note that "q" can vary between "0" or "1" over time (See the observation for id=1 at time=2), but due to the nature of survival analysis, "dv" cannot. What I need to do is create a variable that finds the LAST time that "q" changes between "1" and "0", and then is censored accordingly. After Step 4, my data should look like this:
id time q dv
1 1 1 1
1 2 1 1
1 3 1 1
1 4 1 1
2 1 1 1
2 2 1 1
2 3 0 0
.(id) in plyr is equivalent to
> dum<-split(long,long$id)
> dum[[4]]
id time q dv
4.1 4 1 0 1
4.2 4 2 0 1
your problem is in your 4th split. You reference
answered <- subset(df, q == 1)
in your function. This is an empty set as there are no dum[[4]]$q taking value 1
If you just want to ignore this split then something like
ans<-ddply(long, .(id), function(df) {
# figure out the dropoff time
answered <- subset(df, q == 1)
if(length(answered$q)==0){return()}
last.q = max(answered$time)
subs <- subset(df, time <= last.q + 1)
# set all the dv as desired
new.dv <- rep(last.q,1)
if (last.q < max(df$time)) new.dv <- c(0,last.q)
subs$dv <- new.dv
subs
})
> ans
id time q dv
1 1 1 1 2
2 1 2 1 2
3 2 1 1 0
4 2 2 0 1
5 3 1 0 2
6 3 2 1 2
would be the result
In short: The error is because there is no q == 1 when id == 4.
A good way to check what's going on here is to rewrite the function separately, and manually test each chunk that ddply is processing.
So first rewrite your code like this:
myfun <- function(df) {
# figure out the dropoff time
answered <- subset(df, q == 1)
last.q = max(answered$time)
subs <- subset(df, time <= last.q + 1)
# set all the dv as desired
new.dv <- rep(last.q,1)
if (last.q < max(df$time)) new.dv <- c(0,last.q)
subs$dv <- new.dv
subs
}
ddply(long, .(id), myfun )
That still gives an error of course, but at least now we can manually check what ddply is doing.
ddply(long, .(id), myfun ) really means:
Take the dataframe called long
Create a number of subset dataframes (one for each distinct id)
Apply the function myfun to each subsetted dataframe
Reassemble the results into a single dataframe
So let's attempt to do manually what ddply is doing automatically.
> myfun(subset(long, id == 1))
id time q dv
1.1 1 1 1 2
1.2 1 2 1 2
> myfun(subset(long, id == 2))
id time q dv
2.1 2 1 1 0
2.2 2 2 0 1
> myfun(subset(long, id == 3))
id time q dv
3.1 3 1 0 2
3.2 3 2 1 2
> myfun(subset(long, id == 4))
Error in `$<-.data.frame`(`*tmp*`, "dv", value = c(0, -Inf)) :
replacement has 2 rows, data has 0
In addition: Warning message:
In max(answered$time) : no non-missing arguments to max; returning -Inf
>
So it seems like the error is coming from the step where ddply applies the function for id == 4.
Now let's take the code outside of the function so we can examine each chunk.
> #################
> # set the problem chunk to "df" so we
> # can examine what the function does
> # step by step
> ################
> df <- subset(long, id == 4)
>
> ###################
> # run the bits of function separately
> ###################
> answered <- subset(df, q == 1)
> answered
[1] id time q dv
<0 rows> (or 0-length row.names)
> last.q = max(answered$time)
Warning message:
In max(answered$time) : no non-missing arguments to max; returning -Inf
> last.q
[1] -Inf
> subs <- subset(df, time <= last.q + 1)
> subs
[1] id time q dv
<0 rows> (or 0-length row.names)
> # set all the dv as desired
> new.dv <- rep(last.q,1)
> new.dv
[1] -Inf
> if (last.q < max(df$time)) new.dv <- c(0,last.q)
> subs$dv <- new.dv
Error in `$<-.data.frame`(`*tmp*`, "dv", value = c(0, -Inf)) :
replacement has 2 rows, data has 0
> subs
[1] id time q dv
<0 rows> (or 0-length row.names)
>
So the error that you're getting comes from subs$dv <- new.dv because new.dv has length two (i.e. two values - (0, -Inf)) but sub$dv is length 0. That wouldn't be a problem if dv were a simple vector, but because it's in the sub dataframe whose columns all have two rows, then sub$dv must also have two rows.
The reason sub has zero rows is because there is no q == 1 when id == 4.
Should the final data frame not have anything for id == 4? The answer to your problem really depends on what you want to happen in the case when there are no q==1 for an id. Just let us know, and we can help you with the code.
UPDATE:
The error that you're getting is because subs$dv has 31 values in it and new.dv has two values in it.
In R when you try to assign a longer vector to a shorter vector, it will always complain.
> test <- data.frame(a=rnorm(100),b=rnorm(100))
> test$a <- rnorm(1000)
Error in `$<-.data.frame`(`*tmp*`, "a", value = c(-0.0507065994549323, :
replacement has 1000 rows, data has 100
>
But when you assign a shorter vector to a longer vector, it will only complain if the shorter is not an even multiple of the longer vector. (eg 3 does not go evenly into 100)
> test$a <- rnorm(3)
Error in `$<-.data.frame`(`*tmp*`, "a", value = c(-0.897908251650798, :
replacement has 3 rows, data has 100
But if you tried this, it wouldn't complain since 2 goes into 100 evenly.
> test$a <- rnorm(2)
>
Try this:
> length(test$a)
[1] 100
> length(rnorm(2))
[1] 2
> test$a <- rnorm(2)
> length(test$a)
[1] 100
>
What's it's doing is silently repeating the shorter vector to fill up the longer vector.
And again, what you do to get around the error (i.e. make both vectors the same length) will depend on what you're trying to achieve. Do you make new.dv shorter, or subs$dv longer?
First, to give credit where credit is due, the code below is not mine. It was generated in collaboration with another very generous coworker (and engineer) who helped me work through my problem (for hours!).
I thought that other analysts tasked with constructing a censored variable from survey data might find this code useful, so I am passing the solution along.
library(plyr)
#A function that only selects cases before the last time "q" was coded as "1"
slicedf <- function(df.orig, df=NULL) {
if (is.null(df)) {
return(slicedf(df.orig, df.orig))
}
if (nrow(df) == 0) {
return(df)
}
target <- tail(df, n=1)
#print(df)
#print('--------')
if (target$q == 0) {
return(slicedf(df.orig, df[1:nrow(df) - 1, ]))
}
if (nrow(df.orig) == nrow(df)) {
return(df.orig)
}
return(df.orig[1:(nrow(df) + 1), ])
}
#Applies function to the dataset, and codes over any "0's" before the last "1" as "1"
long <- ddply(long, .(id), function(df) {
df <- slicedf(df)
if(nrow(df) == 0) {
return(df)
}
q <- df$q
if (tail(q, n=1) == 1) {
df$q <- rep(1, length(q))
} else {
df$q <- c(rep(1, length(q) - 1), 0)
}
return(df)
})
Thanks to everyone online who commented for your patience and help.