I have a long form of clinical data that looks something like this:
patientid <- c(100,100,100,101,101,101,102,102,102,104,104,104)
outcome <- c(1,1,1,1,1,NA,1,NA,NA,NA,NA,NA)
time <- c(1,2,3,1,2,3,1,2,3,1,2,3)
Data <- data.frame(patientid=patientid, outcome=outcome, time=time)
A patient should be kept in the database only if they 2 or 3 observations (so patients that have complete data for 0 or only 1 time points should be thrown out. So for this example my desired result is this:
patientid <- c(100,100,100,101,101,101)
outcome <- c(1,1,1,1,1,NA)
time <- c(1,2,3,1,2,3)
Data <- data.frame(patientid=patientid, outcome=outcome, time=time)
Hence patients 102 and 104 are thrown out of the database because of they were missing the outcome variable in 2 or 3 of the time points.
We can create a logical expression on the sum of non-NA elements as a logical vector, grouped by 'patientid' to filter patientid's having more than one non-NA 'outcome'
library(dplyr)
Data %>%
group_by(patientid) %>%
filter(sum(!is.na(outcome)) > 1) %>%
ungroup
-output
# A tibble: 6 x 3
# patientid outcome time
# <dbl> <dbl> <dbl>
#1 100 1 1
#2 100 1 2
#3 100 1 3
#4 101 1 1
#5 101 1 2
#6 101 NA 3
A base R option using subset + ave
subset(
Data,
ave(!is.na(outcome), patientid, FUN = sum) > 1
)
giving
patientid outcome time
1 100 1 1
2 100 1 2
3 100 1 3
4 101 1 1
5 101 1 2
6 101 NA 3
A data.table option
setDT(Data)[, Y := sum(!is.na(outcome)), patientid][Y > 1, ][, Y := NULL][]
or a simpler one (thank #akrun)
setDT(Data)[Data[, .I[sum(!is.na(outcome)) > 1], .(patientid)]$V1]
which gives
patientid outcome time
1: 100 1 1
2: 100 1 2
3: 100 1 3
4: 101 1 1
5: 101 1 2
6: 101 NA 3
library(dplyr)
Data %>%
group_by(patientid) %>%
mutate(observation = sum(outcome, na.rm = TRUE)) %>% # create new variable (observation) and count the observation per patient
filter(observation >=2) %>%
ungroup
output:
# A tibble: 6 x 4
patientid outcome time observation
<dbl> <dbl> <dbl> <dbl>
1 100 1 1 3
2 100 1 2 3
3 100 1 3 3
4 101 1 1 2
5 101 1 2 2
6 101 NA 3 2
Related
I have a data frame with three variables and some missing values in one of the variables that looks like this:
subject <- c(1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2)
part <- c(0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3,0,0,0,0,1,1,1,1,2,2,2,2,3,3,3,3)
sad <- c(1,7,7,4,NA,NA,2,2,NA,2,3,NA,NA,2,2,1,NA,5,NA,6,6,NA,NA,3,3,NA,NA,5,3,NA,7,2)
df1 <- data.frame(subject,part,sad)
I have created a new data frame with the mean values of 'sad' per subject and part using a loop, like this:
columns<-c("sad.m",
"part",
"subject")
df2<-matrix(data=NA,nrow=1,ncol=length(columns))
df2<-data.frame(df2)
names(df2)<-columns
tn<-unique(df1$subject)
row=1
for (s in tn){
for (i in 0:3){
TN<-df1[df1$subject==s&df1$part==i,]
df2[row,"sad.m"]<-mean(as.numeric(TN$sad), na.rm = TRUE)
df2[row,"part"]<-i
df2[row,"subject"]<-s
row=row+1
}
}
Now I want to include an additional variable 'missing' in that indicates the percentage of rows per subject and part with missing values, so that I get df3:
subject <- c(1,1,1,1,2,2,2,2)
part<-c(0,1,2,3,0,1,2,3)
sad.m<-df2$sad.m
missing <- c(0,50,50,25,50,50,50,25)
df3 <- data.frame(subject,part,sad.m,missing)
I'd really appreciate any help on how to go about this!
It's best to try and avoid loops in R where possible, since they can get messy and tend to be quite slow. For this sort of thing, dplyr library is perfect and well worth learning. It can save you a lot of time.
You can create a data frame with both variables by first grouping by subject and part, and then performing a summary of the grouped data frame:
df2 = df1 %>%
dplyr::group_by(subject, part) %>%
dplyr::summarise(
sad_mean = mean(na.omit(sad)),
na_count = (sum(is.na(sad) / n()) * 100)
)
df2
# A tibble: 8 x 4
# Groups: subject [2]
subject part sad_mean na_count
<dbl> <dbl> <dbl> <dbl>
1 1 0 4.75 0
2 1 1 2 50
3 1 2 2.5 50
4 1 3 1.67 25
5 2 0 5.5 50
6 2 1 4.5 50
7 2 2 4 50
8 2 3 4 25
For each subject and part you can calculate mean of sad and calculate ratio of NA value using is.na and mean.
library(dplyr)
df1 %>%
group_by(subject, part) %>%
summarise(sad.m = mean(sad, na.rm = TRUE),
perc_missing = mean(is.na(sad)) * 100)
# subject part sad.m perc_missing
# <dbl> <dbl> <dbl> <dbl>
#1 1 0 4.75 0
#2 1 1 2 50
#3 1 2 2.5 50
#4 1 3 1.67 25
#5 2 0 5.5 50
#6 2 1 4.5 50
#7 2 2 4 50
#8 2 3 4 25
Same logic with data.table :
library(data.table)
setDT(df1)[, .(sad.m = mean(sad, na.rm = TRUE),
perc_missing = mean(is.na(sad)) * 100), .(subject, part)]
Try this dplyr approach to compute df3:
library(dplyr)
#Code
df3 <- df1 %>% group_by(subject,part) %>% summarise(N=100*length(which(is.na(sad)))/length(sad))
Output:
# A tibble: 8 x 3
# Groups: subject [2]
subject part N
<dbl> <dbl> <dbl>
1 1 0 0
2 1 1 50
3 1 2 50
4 1 3 25
5 2 0 50
6 2 1 50
7 2 2 50
8 2 3 25
And for full interaction with df2 you can use left_join():
#Left join
df3 <- df1 %>% group_by(subject,part) %>%
summarise(N=100*length(which(is.na(sad)))/length(sad)) %>%
left_join(df2)
Output:
# A tibble: 8 x 4
# Groups: subject [2]
subject part N sad.m
<dbl> <dbl> <dbl> <dbl>
1 1 0 0 4.75
2 1 1 50 2
3 1 2 50 2.5
4 1 3 25 1.67
5 2 0 50 5.5
6 2 1 50 4.5
7 2 2 50 4
8 2 3 25 4
Apologies if this is a repeat question but I could not find the specific answer I am looking for. I have a dataframe with counts of different species caught on a given trip. A simplified example with 5 trips and 4 species is below:
trip = c(1,1,1,2,2,3,3,3,3,4,5,5)
species = c("a","b","c","b","d","a","b","c","d","c","c","d")
count = c(5,7,3,1,8,10,1,4,3,1,2,10)
dat = cbind.data.frame(trip, species, count)
dat
> dat
trip species count
1 1 a 5
2 1 b 7
3 1 c 3
4 2 b 1
5 2 d 8
6 3 a 10
7 3 b 1
8 3 c 4
9 3 d 3
10 4 c 1
11 5 c 2
12 5 d 10
I am only interested in the counts of species b for each trip. So I want to manipulate this data frame so I end up with one that looks like this:
trip2 = c(1,2,3,4,5)
species2 = c("b","b","b","b","b")
count2 = c(7,1,1,0,0)
dat2 = cbind.data.frame(trip2, species2, count2)
dat2
> dat2
trip2 species2 count2
1 1 b 7
2 2 b 1
3 3 b 1
4 4 b 0
5 5 b 0
I want to keep all trips, including trips where species b was not observed. So I can't just subset the data by species b. I know I can cast the data so species are the columns and then just remove the columns for the other species like so:
library(dplyr)
library(reshape2)
test = dcast(dat, trip ~ species, value.var = "count", fun.aggregate = sum)
test
> test
trip a b c d
1 1 5 7 3 0
2 2 0 1 0 8
3 3 10 1 4 3
4 4 0 0 1 0
5 5 0 0 2 10
However, my real dataset has several hundred species caught on thousands of trips, and if I try to cast that many species to columns R chokes. There are way too many columns. Is there a way to specify in dcast that I only want to cast species b? Or is there another way to do this that doesn't require casting the data? Thank you.
Here is a data.table approach which I suspect will be very fast for you:
library(data.table)
setDT(dat)
result <- dat[,.(species = "b", count = sum(.SD[species == "b",count])),by = trip]
result
trip species count
1: 1 b 7
2: 2 b 1
3: 3 b 1
4: 4 b 0
5: 5 b 0
We can use tidyverse
library(dplyr)
library(tidyr)
dat %>%
filter(species == 'b') %>%
group_by(trip, species) %>%
summarise(count = sum(count)) %>%
ungroup %>%
complete(trip = unique(dat$trip), fill = list(species = 'b', count = 0))
# A tibble: 5 x 3
# trip species count
# <dbl> <chr> <dbl>
#1 1 b 7
#2 2 b 1
#3 3 b 1
#4 4 b 0
#5 5 b 0
I am analysing my data with R for the first time which is a bit challenging. I have a data frame with my data that looks like this:
head(data)
subject group age trial cond acc rt
1 S1 2 1 1 1 1 5045
2 S1 2 1 2 2 1 8034
3 S1 2 1 3 1 1 6236
4 S1 2 1 4 2 1 8087
5 S1 2 1 5 3 0 8756
6 S1 2 1 6 1 1 6619
I would like to compute a mean and standard deviation for each subject in each condition for rt and a sum for each subject in each condition for acc. All the other variables are should remain the same (group and age are subject-specific, and trial can be disregarded).
I have tried using aggregate but that seemed kind of complicated because I had to do it in several steps and re-add information...
I'd be thankful for any help =)
Edit: I realise that I wasn't being clear. I want trial to be disregarded and end up with one row per subject per condition:
head(data_new)
subject group age cond rt_mean rt_sd acc_sum
1 S1 2 1 1 7581 100 5
2 S2 2 1 2 8034 150 4
Sorry about the confusion!
If you don't mind using the data.table package:
library(data.table)
data <- data.table(data)
data[, ':=' (rt_mean = mean(rt), rt_sd = sd(rt), acc_sum = sum(acc)), by = .(subject, cond)]
data
subject group age trial cond acc rt rt_mean rt_sd acc_sum
1: S1 2 1 1 1 1 5045 5966.667 820.83758 3
2: S1 2 1 2 2 1 8034 8060.500 37.47666 2
3: S1 2 1 3 1 1 6236 5966.667 820.83758 3
4: S1 2 1 4 2 1 8087 8060.500 37.47666 2
5: S1 2 1 5 3 0 8756 8756.000 NA 0
6: S1 2 1 6 1 1 6619 5966.667 820.83758 3
Edit:
If you want to get rid of some of the variables and duplicated rows, you need only a small modification - remove the := assignment operator (instead of adding new colums, it will now create a new data.table), add the variables you want to keep and use the unique function:
unique(dt[, .(group, age, rt_mean = mean(rt), rt_sd = sd(rt), acc_sum = sum(acc)), by = .(subject, cond)])
subject cond group age rt_mean rt_sd acc_sum
1: S1 1 2 1 5966.667 820.83758 3
2: S1 2 2 1 8060.500 37.47666 2
3: S1 3 2 1 8756.000 NA 0
If you additionally want to get rid of rows with missing values, use the na.omit function.
The package dplyr is made for this:
library(dplyr)
d %>%
group_by(subject, cond) %>% # we group by the two values
summarise(
mean_rt = mean(rt, na.rm=T),
sd_rt = sd(rt, na.rm=T),
sum_acc = sum(acc, na.rm=T) # here we apply each function to summarise values
)
# A tibble: 3 x 5
# Groups: subject [?]
subject cond mean_rt sd_rt sum_acc
<fct> <int> <dbl> <dbl> <int>
1 S1 1 5967. 821. 3
2 S1 2 8060. 37.5 2
3 S1 3 8756 NA 0
# NA for the last sd_rt is because you can't have
# sd for a single obs.
Basically you need to group_by the columns (one or more) that you need to use as grouping, then inside summarise, you apply each function you need (mean, sd, sum, ecc) to each variable (rt, acc, ecc).
Change summarise with mutate if you want to keep all variables:
d %>%
select(-trial) %>% # use select with -var_name to eliminate columns
group_by(subject, cond) %>%
mutate(
mean_rt = mean(rt, na.rm=T),
sd_rt = sd(rt, na.rm=T),
sum_acc = sum(acc, na.rm=T)
) %>%
ungroup()
# A tibble: 6 x 9
subject group age cond acc rt mean_rt sd_rt sum_acc
<fct> <int> <int> <int> <int> <int> <dbl> <dbl> <int>
1 S1 2 1 1 1 5045 5967. 821. 3
2 S1 2 1 2 1 8034 8060. 37.5 2
3 S1 2 1 1 1 6236 5967. 821. 3
4 S1 2 1 2 1 8087 8060. 37.5 2
5 S1 2 1 3 0 8756 8756 NA 0
6 S1 2 1 1 1 6619 5967. 821. 3
Update based on op request, maybe this is what you need:
d %>%
group_by(subject, cond, group, age) %>%
summarise(
mean_rt = mean(rt, na.rm=T),
sd_rt = sd(rt, na.rm=T),
sum_acc = sum(acc, na.rm=T)
)
# A tibble: 3 x 7
# Groups: subject, cond, group [?]
subject cond group age mean_rt sd_rt sum_acc
<fct> <int> <int> <int> <dbl> <dbl> <int>
1 S1 1 2 1 5967. 821. 3
2 S1 2 2 1 8060. 37.5 2
3 S1 3 2 1 8756 NA 0
Data used:
tt <- "subject group age trial cond acc rt
S1 2 1 1 1 1 5045
S1 2 1 2 2 1 8034
S1 2 1 3 1 1 6236
S1 2 1 4 2 1 8087
S1 2 1 5 3 0 8756
S1 2 1 6 1 1 6619"
d <- read.table(text=tt, header=T)
If you want to compute for example the mean of rt for subject S1 under condition 1, you can use mean(data[data$subject == "S1" & data$cond == 1, 7]).
I hope this gives you an idea how you can filter your values.
I am having troubles finding how to find individual values from the running mean in an R dataframe.
I have an R dataframe:
x ID Mean
1 1 1
1 2 5
2 1 3
2 2 6
Where the mean is the mean for the x measurements for the specific ID in the dataframe.
To find the individual values at each x value rather than the mean, I was thinking that I needed to apply a recursive function on the dataframe and group by the ID. How could I do this in a dataframe while grouping by one of the values when any apply function wouldn't have access to the previous entry in the dataframe?
When completed and appended to the dataframe, I am hoping it to look like this:
x ID Mean IndivValues
1 1 1 1
1 2 5 5
2 1 3 5
2 2 6 7
It's much easier to calculate this from totals -> to individual observation, as below:
Example data.frame:
df <- read.table(text='
x ID Mean
1 1 1
1 2 5
2 1 3
2 2 6
', header=T)
Solution:
library(dplyr); library(magrittr)
df %>%
group_by(id) %>%
mutate(
total = mean * x,
ind_value = total - lag(total, default=0) )
## A tibble: 4 x 5
## Groups: ID [2]
# x ID Mean total ind_value
# <int> <int> <int> <int> <int>
#1 1 1 1 1 1
#2 1 2 5 5 5
#3 2 1 3 6 5
#4 2 2 6 12 7
I want to make a summary by ID, DRUG, FED summarising the sum of the CONC for DVID =1 and DVID==2
df<-
ID DRUG FED DVID CONC
1 1 1 1 20
1 1 1 2 40
2 2 0 1 30
2 2 0 2 100
I tried using this:
df2 <- df %>%
group_by(ID,DRUG,FED) %>%
summarise(SumCOnc=CONC+lag(CONC))
However I am getting this error:
Error: expecting a single value
I don't get the error when I use mutate. Is there a way to go around it so I use summarise in the case described above?
The output should basically be this:
ID DRUG FED SumConc
1 1 1 60
2 2 0 130
This seems pretty straightforward: just use sum(), don't mess around with lag() ...
Get data:
df<- read.table(header=TRUE,
text="
ID DRUG FED DVID CONC
1 1 1 1 20
1 1 1 2 40
2 2 0 1 30
2 2 0 2 100
")
Process:
library(dplyr)
df %>%
group_by(ID,DRUG,FED) %>%
summarise(SumConc=sum(CONC))
## ID DRUG FED SumConc
## 1 1 1 1 60
## 2 2 2 0 130
A simple Base R approach would be using aggregate
aggregate(CONC ~ ID + DRUG + FED, df, sum)
# ID DRUG FED CONC
#1 2 2 0 130
#2 1 1 1 60
Or from data.table
library(data.table)
setDT(df)[, .(SumConc = sum(CONC)), .(ID, DRUG, FED)]
# ID DRUG FED SumConc
#1: 1 1 1 60
#2: 2 2 0 130