I need a function that sums the above N+1 rows in dataframes (data tables) by groups.
An equivalent function for a vector, would be something like below. (Please forgive me if the function below is inefficient)
Function1<-function(x,N){
y<-vector(length=length(x))
for (i in 1:length(x))
if (i<=N)
y[i]<-sum(x[1:i])
else if (i>N)
y[i]<-sum(x[(i-N):i])
return(y)}
Function1(c(1,2,3,4,5,6),3)
#[1] 1 3 6 10 14 18 # Sums previous (above) 4 values (rows)
I wanted to use this function with sapply, like below..
sapply(X=DF<-data.frame(A=c(1:10), B=2), FUN=Function1(N=3))
but couldn't.. because I could not figure out how to set a default for the x in my function. Thus, I built another function for data.frames.
Function2<-function(x, N)
if(is.data.frame(x)) {
y<-data.frame()
for(j in 1:ncol(x))
for(i in 1:nrow(x))
if (i<=N) {
y[i,j]<-sum(x[1:i,j])
} else if (i>N) {
y[i,j]<-sum(x[(i-N):i,j])}
return(y)}
DF<-data.frame(A=c(1:10), B=2)
Function2(DF, 2)
# V1 V2
1 1 2
2 3 4
3 6 6
4 9 6
5 12 6
6 15 6
7 18 6
8 21 6
9 24 6
10 27 6
However, I still need to perform this by groups. For example, for the following data frame with a character column.
DF<-data.frame(Name=rep(c("A","B"),each=5), A=c(1:10), B=2)
I would like to apply my function by group "Name" -- which would result in.
A 1 2
A 3 4
A 6 6
A 9 6
A 12 6
B 6 2
B 13 4
B 21 6
B 24 6
B 27 6
#Perform function2 separately for group A and B.
I was hoping to use function with the data.table package (by=Groups), but couldn't figure out how.
What would be the best way to do this?
(Also, it would be really nice, if I could learn how to make my Function1 to work in sapply)
With data.table, we group by 'Name', loop through the columns of interest specified in .SDcols (here all the columns are of interest so we are not specifying it) and apply the Function1
library(data.table)
setDT(DF)[, lapply(.SD, Function1, 2), Name]
# Name A B
# 1: A 1 2
# 2: A 3 4
# 3: A 6 6
# 4: A 9 6
# 5: A 12 6
# 6: B 6 2
# 7: B 13 4
# 8: B 21 6
# 9: B 24 6
#10: B 27 6
Related
I have a function that I repeat, changing the argument each time, using apply/sapply/lapply.
Works great.
I want to return a data set, where each row contains two (or more) variables from each iteration of the function.
Instead I get an unusable list.
do <-function(x){
a <- x+1
b <- x+2
cbind(a,b)
}
over <- [1:6]
final <- lapply(over, do)
Any suggestions?
Without changing your function do, you can use sapply and transpose it.
data.frame(t(sapply(over, do)))
# X1 X2
#1 2 3
#2 3 4
#3 4 5
#4 5 6
#5 6 7
#6 7 8
If you want to use do in current form with lapply, we can do
do.call(rbind.data.frame, lapply(over, do))
You could also try
as.data.frame(Reduce(rbind, final))
# a b
# 1 2 3
# 2 3 4
# 3 4 5
# 4 5 6
# 5 6 7
# 6 7 8
See ?Reduce and ?rbind for information about what they'll do.
You could also modify your final expression as
final <- as.data.frame(Reduce(rbind, lapply(over, do)))
#final
# a b
# 1 2 3
# 2 3 4
# 3 4 5
# 4 5 6
# 5 6 7
# 6 7 8
I'm currently stumped making some efficient code. I have a vector of variables (med.vars) that were transformed by the in-year global median. Sometimes the global median is 0, which creates Inf/-Inf values I would like to replace with the pre-transformed variable value (vars). I can't figure out how to do this efficiently with some type of data.table 'dat[,:=lapply(.SD), .SDcols=med.vars] function or a for loop with get(), noquotes(), etc.
dat<-data.table(v1=c(2,10,7),v2=c(5,6,5),v3=c(10,15,20),v1.med=c(1,Inf,5),v2.med=c(5,6,5),v3.med=c(-Inf,2,3))
vars<-c("v1","v2","v3")
med.vars<-c("v1.med","v2.med","v3.med")
v1 v2 v3 v1.med v2.med v3.med
1: 2 5 10 1 5 -Inf
2: 10 6 15 Inf 6 2
3: 7 5 20 5 5 3
In reality these vectors are 50+ vars I pull from names(dat) with grep() and use gsub(".med","",med.vars) to create the second vector of pre-transformed variable names.
I would like to efficiently perform
dat[v1.med==Inf | v1.med==-Inf, v1.med:=v1]
dat[v3.med==Inf | v3.med==-Inf, v3.med:=v3]
for each element, med.vars[i], and its corresponding element, vars[i] such that the resulting data.table is:
v1 v2 v3 v1.med v2.med v3.med
1: 2 5 10 1 5 -10
2: 10 6 15 10 6 2
3: 7 5 20 5 5 3
Thank you for your time
OP mentions efficiency, so maybe move to long form. Then the standard syntax can be used:
DT = melt(dat, meas=list(vars, med.vars), value.name=c("var", "med"))
DT[!is.finite(med), med := sign(med)*var]
variable var med
1: 1 2 1
2: 1 10 10
3: 1 7 5
4: 2 5 5
5: 2 6 6
6: 2 5 5
7: 3 10 -10
8: 3 15 2
9: 3 20 3
As these are corresponding columns, we can make use of Map
dat[, (med.vars) := Map(function(x, y) ifelse(is.finite(y), y,
x * sign(y)), .SD[, vars, with = FALSE],
.SD[, med.vars, with = FALSE])]
dat
# v1 v2 v3 v1.med v2.med v3.med
#1: 2 5 10 1 5 -10
#2: 10 6 15 10 6 2
#3: 7 5 20 5 5 3
Or another option is set by looping through the columns with a for loop
for(j in seq_along(vars)) {
i1 <- !is.finite(dat[[med.vars[j]]])
v1 <- dat[[vars[j]]]
v2 <- dat[[med.vars[j]]]
set(dat, i = which(i1), j = med.vars[j], value = sign(v2[i1]) * v1[i1])
}
This can also be done in base R (on a data.frame)
i1 <- !sapply(dat[med.vars], is.finite)
dat[med.vars][i1] <- dat[vars][i1] * sign(dat[med.vars][i1])
I am generating 5 different prediction and adding those predictions to an existing data frame. My code is:
For j in i{
…
actual.predicted <- data.frame(test_data, predicted)
}
I am trying to concatenate words together to create new column names, in the loop. Specifically, I have a column named “predicted” and I am generating predictions in each iteration of the loop. So, in the first iteration, I want the new column name to be “predicted.1” and for the second iteration, the new column name should be “predicted.2” and so on.
Any thoughts would be greatly appreciated.
You may not even need to use a loop here, but assuming you do, one pattern which might work well here would be to use a list:
results <- list()
for j in i {
# do something involving j
name <- paste0("predicted.", j)
results[[name]] <- data.frame(test_data, predicted)
}
One option is to set the names after assigning new columns
actual.predicted <- data.frame(orig_col = sample(10))
for (j in 1:5){
new_col = sample(10)
actual.predicted <- cbind(actual.predicted, new_col)
names(actual.predicted)[length(actual.predicted)] <- paste0('predicted.',j)
}
actual.predicted
# orig_col predicted.1 predicted.2 predicted.3 predicted.4 predicted.5
# 1 1 4 4 9 1 5
# 2 10 2 3 7 5 9
# 3 8 6 5 4 2 3
# 4 5 9 9 10 7 7
# 5 2 1 10 8 3 10
# 6 9 7 6 6 8 6
# 7 7 8 7 2 4 2
# 8 3 3 1 1 6 8
# 9 6 10 2 3 9 4
# 10 4 5 8 5 10 1
Here is the data.
set.seed(23) data<-data.frame(ID=rep(1:12), group=rep(1:3,times=4), value=(rnorm(12,mean=0.5, sd=0.3)))
ID group value
1 1 1 0.4133934
2 2 2 0.6444651
3 3 3 0.1350871
4 4 1 0.5924411
5 5 2 0.3439465
6 6 3 0.3673059
7 7 1 0.3202062
8 8 2 0.8883733
9 9 3 0.7506174
10 10 1 0.3301955
11 11 2 0.7365258
12 12 3 0.1502212
I want to get z-standardized scores within each group. so I try
library(weights)
data_split<-split(data, data$group) #split the dataframe
stan<-lapply(data_split, function(x) stdz(x$value)) #compute z-scores within group
However, It looks wrong because I want to add a new variable following 'value'
How can I do that? Kindly provide some suggestions(sample code). Any help is greatly appreciated .
Use this instead:
within(data, stan <- ave(value, group, FUN=stdz))
No need to call split nor lapply.
One way using data.table package:
library(data.table)
library(weights)
set.seed(23)
data <- data.table(ID=rep(1:12), group=rep(1:3,times=4), value=(rnorm(12,mean=0.5, sd=0.3)))
setkey(data, ID)
dataNew <- data[, list(ID, stan = stdz(value)), by = 'group']
the result is:
group ID stan
1: 1 1 -0.6159312
2: 1 4 0.9538398
3: 1 7 -1.0782747
4: 1 10 0.7403661
5: 2 2 -1.2683237
6: 2 5 0.7839781
7: 2 8 0.8163844
8: 2 11 -0.3320388
9: 3 3 0.6698418
10: 3 6 0.8674548
11: 3 9 -0.2131335
12: 3 12 -1.3241632
I tried Ferdinand.Kraft's solution but it didn't work for me. I think the stdz function isn't included in the basic R install. Moreover, the within part troubled me in a large dataset with many variables. I think the easiest way is:
data$value.s <- ave(data$value, data$group, FUN=scale)
Add the new column while in your function, and have the function return the whole data frame.
stanL<-lapply(data_split, function(x) {
x$stan <- stdz(x$value)
x
})
stan <- do.call(rbind, stanL)
I have a dataset that looks something like this, with many classes, each with many (5-10) subclasses, each with a value associated with it:
> data.frame(class=rep(letters[1:4], each=4), subclass=c(1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8), value=1:16)
class subclass value
1 a 1 1
2 a 1 2
3 a 2 3
4 a 2 4
5 b 3 5
6 b 3 6
7 b 4 7
8 b 4 8
9 c 5 9
10 c 5 10
11 c 6 11
12 c 6 12
13 d 7 13
14 d 7 14
15 d 8 15
16 d 8 16
I want to first sum the values for each class/subclass, then take the median value for each class among all the subclasses.
I.e., the intermediate step would sum the values for each subclass for each class, and would look like this (note that I don't need to keep the data from this intermediate step):
> data.frame(class=rep(letters[1:4], each=2), subclass=1:8, sum=c(3,7,11,15,19,23,27,31))
class subclass sum
1 a 1 3
2 a 2 7
3 b 3 11
4 b 4 15
5 c 5 19
6 c 6 23
7 d 7 27
8 d 8 31
The second step would take the median for each class among all the subclasses, and would look like this:
> data.frame(class=letters[1:4], median=c(median(c(3,7)), median(c(11,15)), median(c(19,23)), median(c(27,31))))
class median
1 a 5
2 b 13
3 c 21
4 d 29
This is the only data I need to keep. Note that both $class and $subclass will be factor variables, and value will always be a non-missing positive integer. Each class will have a varying number of subclasses.
I'm sure I can do this with some nasty for loops, but I was hoping for a better way that's vectorized and easier to maintain.
Here is another example of using aggregate
temp <- aggregate(df$value,list(class=df$class,subclass=df$subclass),sum)
aggregate(temp$x,list(class=temp$class),median)
Output:
class x
1 a 5
2 b 13
3 c 21
4 d 29
Or if you like a one-liner solution, you can do:
aggregate(value ~ class, median, data=aggregate(value ~ ., sum, data=df))
You could try for your first step:
df_sums <- aggregate(value ~ class + subclass, sum, data=df)
Then:
aggregate(value ~ class, median, data=df_sums)
Here are two other alternatives.
The first uses ave within a within statement where we progressively reduce our source data.frame after adding in our aggregated data. Since this will result in many repeated rows, we can safely use unique as the last step to get the output you want.
unique(within(mydf, {
Sum <- ave(value, class, subclass, FUN = sum)
rm(subclass, value)
Median <- ave(Sum, class, FUN = median)
rm(Sum)
}))
# class Median
# 1 a 5
# 5 b 13
# 9 c 21
# 13 d 29
A second option is to use the "data.table" package and "compound" your statements as below. V1 is the name that will be automatically created by data.table if a name is not specified by the user.
library(data.table)
DT <- data.table(mydf)
DT[, sum(value), by = c("class", "subclass")][, median(V1), by = "class"]
# class V1
# 1: a 5
# 2: b 13
# 3: c 21
# 4: d 29