How to construct a loop function for a massive data frame - r

Question

I'm trying to find outliers in many times series data frames by writing a loop function so that the result would be listed.
Every row usually has numbers or just zeros, but there are sometimes N/A values and they should be eliminated from the function.
For every row there is a name of data (first column) and then different numbers (further columns). For every data I find the mean, standard deviation, Tmin and Tmax and then simple filtering:
mean <- mean(as.vector(as.numeric(x2[1,])))
sd <- sd(as.vector(as.numeric(x2[1,])))
Tmin <- mean - (3*sd)
Tmax <- mean + (3*sd)
x[which(x < Tmin | x > Tmax)]
Sample table:
Name
2015Q1
2015Q2
2015Q3
2015Q4
2016Q1
...
Banking sect.
63.4442
61.73465
67.33452
52.34546
12.3334
...
Money laund. sect.
0
0
0
2.4
0
...
Households
N/A
N/A
1.2
N/A
9.6
...
...
...
...
...
...
...
...
Result after using the loop function should be a table containing only the *Name *column and outlier column and values.
I've tried writing a loop function, but I don't understand loop functions fully. It should be reusable adding new identical template data frames.
for (i in 1:nrow(x)) {
mean <- mean(as.numeric(data.frame(x[i,2:50])))
sd <- sd(as.numeric(data.frame(x[i,2:50])))
Tmin <- mean - (3*sd)
Tmax <- mean + (3*sd)
print(x[which(x < Tmin | x > Tmax)])
i <- i+1
}
Would appreciate any advice.
Thank you

Answer 1

In R, for loops should be avoided like the plague. Use the function rowMeans to compute the mean of each row, and use apply to find the sd of each row. Here's some experimental code:
df <- cbind(
Name=sample(stringi::stri_rand_strings(100, 1, "[A-Z]"), size=100, replace=T),
data.frame(matrix(rnorm(n=1000), nrow=100))
)
means <- rowMeans(as.numeric(df[, 2:ncol(df)], na.rm=T)
stdDevs <- apply(X=df[, 2:ncol(df), MARGIN=1, FUN=sd, na.rm=T)
Tmin <- means - 3*stdDevs
Tmax <- means + 3*stdDevs
outliers <- sapply(df, FUN=function(x){x<Tmin | x>Tmax})
This yields a boolean array, where each entry indicates whether that entry is an outlier. NOTE: the sapply function outputs TRUE for every entry in the Name column.
EDIT: details to get into the format OP desires.
outliers <- data.frame(sapply(df, FUN=function(x){x<Tmin | x>Tmax}))
outliers$Name <- df$Name
outliers <- tidyr::pivot_longer(
outliers, names_to="variable", cols=setdiff(colnames(outliers), "Name")
)
outliers <- outliers[outliers$value == TRUE, ]
New you have a data frame where each row shows a Name-variable combination that is an outlier (according to the +/- three standard deviation rule you set forth).