else if loop running very slowly in R - r

Question

I am performing a research project into the factors that make someone more likely to vote, with a focus on the distance people live from a polling place. I the full voter registration and voter histories for millions of individuals. There are several ways in which someone can vote (in person, absentee, early, or provisional) or not vote (not registered, registered but didn't vote, or ineligible to vote). My data comes with a column (29) for how someone voted in a given election. NULL means not registered, V for in person, etc.
For regression analysis, I want to create a different column for each voter type (1 for yes, 0 for no, column numbers 68-74) and another 1/0 column (number 75) for whether or not someone voted at all. The code I wrote below should do the trick, but it's running impossibly slowly on my computer and hasn't even been able to get to the 1000th row after an hour. It works perfectly, except the speed. I've been approved to use my university's supercomputer*, but I want to figure out a faster algorithm. I have R and STATA both on my laptop and the supercomputer* and would be happy to use either.
dcv.new <- read.csv("VoterHist.csv", header=TRUE)
# I previously set columns 68-75 to default to 0
for(i in 1:nrow(dcv.new))
{
if(is.na(dcv.new[i,29]))
{
dcv.new[i,69] <- 1
}
else if(dcv.new[i,29]=="V")
{
dcv.new[i,68] <- 1
dcv.new[i,75] <- 1
}
else if(dcv.new[i,29]=="A")
{
dcv.new[i,70] <- 1
dcv.new[i,75] <- 1
}
else if(dcv.new[i,29]=="N")
{
dcv.new[i,71] <- 1
}
else if(dcv.new[i,29]=="E")
{
dcv.new[i,72] <- 1
}
else if(dcv.new[i,29]=="Y")
{
dcv.new[i,73] <- 1
}
else if(dcv.new[i,29]=="P")
{
dcv.new[i,74] <- 1
dcv.new[i,75] <- 1
}
else if(dcv.new[i,29]=="X")
{
dcv.new[i,74] <- 1
dcv.new[i,75] <- 1
}
}
*Technically "High performance computing cluster", but let's be honest, supercomputer sounds way cooler.

Answer 1

R is vectorised, in the main, so look for vectorised operations in place of loops. In this case you can vectorise each operation so it works on the entire matrix rather than on individual rows.
Here are the first three of your if else statements:
dcv.new[is.na(dcv.new[,29]), 69] <- 1
dcv.new[dcv.new[,29]=="V", c(68,75)] <- 1
dcv.new[dcv.new[,29]=="A", c(70,75)] <- 1
....
You should get the idea.
Some explanation:
What we are doing is selecting rows from certain columns of dcv.new that meet criteria (such as == "V") and then we assign the value 1 to each of those selected elements of dcv.new in a single operation. R recycles the 1 that we assigned such that it becomes the same length as that required to fill all the selected elements.
Note how we select more than one column at once for updating: dcv.new[x , c(68,75)] updates columns 68 and 75 for rows x only, where x is a logical vector indexing the rows we need to update. The logical vector is produced by statements like dcv.new[,29]=="V". These return a TRUE if an element of dcv.new[,29] equals "V" and FALSE if not.
However...!
In the case of regression, we can let R make the matrix of dummy variables for us, we don't need to do it by hand. Say the column dcv.new[, 29] was named voterType. If we coerce it to be a factor
dcv.new <- transform(dcv.new, voterType = factor(voterType))
when we fit a model using the formula notation we can do:
mod <- lm(response ~ voterType, data = dcv.new)
and R will create the appropriate contrasts to make voterType use the correct degrees of freedom. By default R uses the first level of a factor as the base level and hence model coefficients represent deviations from this reference level. To see what is the reference level for voterType after converting it to a factor do
with(dcv.new, levels(voterType)[1])
Note that most modelling functions that take a formula, like the one shown above, work as I described and show below. You aren't restricted to lm() models.
Here is a small example
set.seed(42)
dcv.new <- data.frame(response = rnorm(20),
voterType = sample(c("V","A","N","E","Y","P","X",NA), 20,
replace = TRUE))
head(dcv.new)
> head(dcv.new)
response voterType
1 1.3709584 E
2 -0.5646982 E
3 0.3631284 V
4 0.6328626 <NA>
5 0.4042683 E
6 -0.1061245 <NA>
The model can then be fitted as
mod <- lm(response ~ voterType, data = dcv.new)
summary(mod)
giving in this case
> mod <- lm(response ~ voterType, data = dcv.new)
> summary(mod)
Call:
lm(formula = response ~ voterType, data = dcv.new)
Residuals:
Min 1Q Median 3Q Max
-2.8241 -0.4075 0.0000 0.5856 1.9030
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -2.656 1.425 -1.864 0.0952 .
voterTypeE 2.612 1.593 1.639 0.1356
voterTypeN 3.040 1.646 1.847 0.0978 .
voterTypeP 2.742 1.646 1.666 0.1300
voterTypeV 2.771 1.745 1.588 0.1468
voterTypeX 2.378 2.015 1.180 0.2684
voterTypeY 3.285 1.745 1.882 0.0925 .
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 1.425 on 9 degrees of freedom
(4 observations deleted due to missingness)
Multiple R-squared: 0.3154, Adjusted R-squared: -0.1411
F-statistic: 0.6909 on 6 and 9 DF, p-value: 0.6635
The magic all happens with the formula code but essentially what happens behind the scenes is that once R has located all the variables named in the formula, it essentially ends up calling something like
model.matrix( ~ voterType, data = dcv.new)
which generates the covariate matrix needed for the underlying matrix algebra and QR decomposition. That code above, for the small example gives:
> model.matrix(~ voterType, data = dcv.new)
(Intercept) voterTypeE voterTypeN voterTypeP voterTypeV voterTypeX
1 1 1 0 0 0 0
2 1 1 0 0 0 0
3 1 0 0 0 1 0
5 1 1 0 0 0 0
8 1 0 0 1 0 0
10 1 0 0 0 0 0
11 1 0 1 0 0 0
12 1 0 1 0 0 0
13 1 1 0 0 0 0
14 1 0 0 0 0 1
15 1 0 0 0 1 0
16 1 0 0 1 0 0
17 1 0 0 1 0 0
18 1 0 0 0 0 0
19 1 0 1 0 0 0
20 1 0 0 0 0 0
voterTypeY
1 0
2 0
3 0
5 0
8 0
10 1
11 0
12 0
13 0
14 0
15 0
16 0
17 0
18 0
19 0
20 1
attr(,"assign")
[1] 0 1 1 1 1 1 1
attr(,"contrasts")
attr(,"contrasts")$voterType
[1] "contr.treatment"
Which is what you are wanting to do with your code. So if you really need it, you could use model.matrix() like I show to also generate the matrix - stripping off the attributes as you don't need them.
In this case the reference level is "A":
> with(dcv.new, levels(voterType)[1])
[1] "A"
which is represented by the (Intercept) column in the output from model.matrix. Note that these treatment contrasts code for deviations from the reference level. You can get dummy values by suppressing the intercept in the formula by adding -1 (0r +0):
> model.matrix(~ voterType - 1, data = dcv.new)
voterTypeA voterTypeE voterTypeN voterTypeP voterTypeV voterTypeX voterTypeY
1 0 1 0 0 0 0 0
2 0 1 0 0 0 0 0
3 0 0 0 0 1 0 0
5 0 1 0 0 0 0 0
8 0 0 0 1 0 0 0
10 0 0 0 0 0 0 1
11 0 0 1 0 0 0 0
12 0 0 1 0 0 0 0
13 0 1 0 0 0 0 0
14 0 0 0 0 0 1 0
15 0 0 0 0 1 0 0
16 0 0 0 1 0 0 0
17 0 0 0 1 0 0 0
18 1 0 0 0 0 0 0
19 0 0 1 0 0 0 0
20 0 0 0 0 0 0 1
attr(,"assign")
[1] 1 1 1 1 1 1 1
attr(,"contrasts")
attr(,"contrasts")$voterType
[1] "contr.treatment"

Answer 2

You should vectorize your code. And forget about so many if's
dcv.new[is.na(dcv.new[,29]),69] <- 1
dcv.new[dcv.new[,29] == "V", c(68, 75)] <- 1
....enter code here
Continue as needed