I am searching for an equivalent function in R of the extremely convenient Stata command simulate. The command basically allows you to declare a program (reg_simulation in the example below) and then invoke such a program from simulate and store desired outputs.
Below is a Stata illustration of the usage of the simulate program, together with my attempt to replicate it using R.
Finally, my main question is: is this how R users will run a Montecarlo simulation? or am I missing something in terms of structure or speed bottlenecks? Thank you a lot in advance.
Stata example
Defining reg_simulation program.
clear all
*Define "reg_simulation" to be used later on by "simulate" command
program reg_simulation, rclass
*Declaring Stata version
version 13
*Droping all variables on memory
drop _all
*Set sample size (n=100)
set obs 100
*Simulate model
gen x1 = rnormal()
gen x2 = rnormal()
gen y = 1 + 0.5 * x1 + 1.5 *x2 + rnormal()
*Estimate OLS
reg y x1 x2
*Store coefficients
matrix B = e(b)
return matrix betas = B
end
Calling reg_simulation from simulate command:
*Seet seed
set seed 1234
*Run the actual simulation 10 times using "reg_simulation"
simulate , reps(10) nodots: reg_simulation
Obtained result (stored data on memory)
_b_x1 _b_x2 _b_cons
.4470155 1.50748 1.043514
.4235979 1.60144 1.048863
.5006762 1.362679 .8828927
.5319981 1.494726 1.103693
.4926634 1.476443 .8611253
.5920001 1.557737 .8391003
.5893909 1.384571 1.312495
.4721891 1.37305 1.017576
.7109139 1.47294 1.055216
.4197589 1.442816 .9404677
R replication of the Stata program above.
Using R I have managed to get the following (not an R expert tho). However, the part that worries me the most is the for-loop structure that loops over each the number of repetitions nreps.
Defining reg_simulation function.
#Defining a function
reg_simulation<- function(obs = 1000){
data <- data.frame(
#Generate data
x1 <-rnorm(obs, 0 , 1) ,
x2 <-rnorm(obs, 0 , 1) ,
y <- 1 + 0.5* x1 + 1.5 * x2 + rnorm(obs, 0 , 1) )
#Estimate OLS
ols <- lm(y ~ x1 + x2, data=data)
return(ols$coefficients)
}
Calling reg_simulation 10 times using a for-loop structure:
#Generate list to store results from simulation
results_list <- list()
# N repetitions
nreps <- 10
for (i in 1:nreps) {
#Set seed internally (to get different values in each run)
set.seed(i)
#Save results into list
results_list[i] <- list(reg_simulation(obs=1000))
}
#unlist results
df_results<- data.frame(t(sapply(results_list,
function(x) x[1:max(lengths(results_list))])))
Obtained result: df_results.
#final results
df_results
# X.Intercept. x1 x2
# 1 1.0162384 0.5490488 1.522017
# 2 1.0663263 0.4989537 1.496758
# 3 0.9862365 0.5144083 1.462388
# 4 1.0137042 0.4767466 1.551139
# 5 0.9996164 0.5020535 1.489724
# 6 1.0351182 0.4372447 1.444495
# 7 0.9975050 0.4809259 1.525741
# 8 1.0286192 0.5253288 1.491966
# 9 1.0107962 0.4659812 1.505793
# 10 0.9765663 0.5317318 1.501162
You're on the right track. Couple of hints/corrections:
Don't use <- inside data.frame()
In R, we construct data frames using = for internal column assignment, i.e. data.frame(x = 1:10, y = 11:20) rather than data.frame(x <- 1:10, y <- 11:20).
(There's more to be said about <- vs =, but I don't want to distract from your main question.)
In your case, you don't actually even need to create a data frame since x1, x2 and y will all be recognized as "global" variables within the scope of the function. I'll post some code at the end of my answer demonstrating this.
When growing a list via a for loop in R, always try to pre-allocate the list first
Always try to pre-allocate the list length and type if you are going to grow a (long) for loop. Reason: That way, R knows how much memory to efficiently allocate to your object. In the case where you are only doing 10 reps, that would mean starting with something like:
results_list <- vector("list", 10)
3. Consider using lapply instead of for
for loops have a bit of bad rep in R. (Somewhat unfairly, but that's a story for another day.) An alternative that many R users would consider is the functional programming approach offered by lapply. I'll hold off on showing you the code for a second, but it will look very similar to a for loop. Just to note quickly, following on from point 2, that one immediate benefit is that you don't need to pre-allocate the list with lapply.
4. Run large loops in parallel
A Monte Carlo simulation is an ideal candidate for running everything in parallel, since each iteration is supposed to be independent of the others. An easy way to go parallel in R is via the future.apply package.
Putting everything together, here's how I'd probably do your simulation. Note that this might be more "advanced" than you possibly need, but since I'm here...
library(data.table) ## optional, but what I'll use to coerce the list into a DT
library(future.apply) ## for parallel stuff
plan(multisession) ## use all available cores
obs <- 1e3
# Defining a function
reg_simulation <- function(...){
x1 <- rnorm(obs, 0 , 1)
x2 <- rnorm(obs, 0 , 1)
y <- 1 + 0.5* x1 + 1.5 * x2 + rnorm(obs, 0 , 1)
#Estimate OLS
ols <- lm(y ~ x1 + x2)
# return(ols$coefficients)
return(as.data.frame(t(ols$coefficients)))
}
# N repetitions
nreps <- 10
## Serial version
# results <- lapply(1:nreps, reg_simulation)
## Parallel version
results <- future_lapply(1:nreps, reg_simulation, future.seed = 1234L)
## Unlist / convert into a data.table
results <- rbindlist(results)
So, following up on the comments, you want to vary your independent variables (x) and also the error term and simulate the coefficients, but you also want to catch errors if any occur. The following would do the trick:
set.seed(42)
#Defining a function
reg_simulation<- function(obs = 1000){
data <- data.frame(
#Generate data
x1 <-rnorm(obs, 0 , 1) ,
x2 <-rnorm(obs, 0 , 1) ,
y <- 1 + 0.5* x1 + 1.5 * x2 + rnorm(obs, 0 , 1) )
#Estimate OLS
tryCatch(
{
ols <- lm(y ~ x1 + x2, data=data)
return(ols$coefficients)
},
error = function(e){
return(c('(Intercept)'=NA, 'x1'=NA, 'x2'=NA))
}
)
}
output <- t(data.frame(replicate(10, reg_simulation())))
output
(Intercept) x1 x2
X1 0.9961328 0.4782010 1.481712
X2 1.0234698 0.4801982 1.556393
X3 1.0336289 0.5239380 1.435468
X4 0.9796523 0.5095907 1.493548
...
Here, tryCatch (see also failwith) catches the error and returns NA as the default value.
Note that you only need to set the seed once because the seed changes automatically with every call to random number generator in a deterministic fashion.
Related
Say we have a cost function:
my_func <- function(x1,x2,x3,x4,x5,x6,x7,x8,x9,x10,x11){
rst <- (x1^2) * x2 + x2 + x3*x4*x5*x6*x7*x8*x9 + x10^3 + (1-x11-x10)
return(rst)
}
The real example has same number of variables, more complex cost function.
Then say I have constraints for each variable:
0 <= x1 <= (1/11)
0 <= x2 <= (1/20)
0 <= x3 <= (1/5)
....
0 <= x11 <= (1/10)
And I want to find the minimum cost for this.
My current solution is, first building sequences for each variable, with a given accuracy (0.001), say for x1:
x1.seq <- seq(0, 1/11, by = 0.001)
....
Then I run 11 for loops, and try every combination, to try and find the minimum:
hold <- NULL # Preallocation of this variable does help, a bit...
for (i in 1:length(x1.seq){
x1 <- x1.seq[i]
for (j in 1:length(x2.seq){
x2 <- x2.seq[j]
....
hold <- c(hold, my_func(x1 = x1, ..., x11 = x11))
....
}
}
min(hold)
Now as all of you probably know, this will run forever given the accuracy is not trivial. So is there a faster way of doing this in R? I know of other approaches like partial derivatives or Lagrange multipliers etc, but they are also time consuming (and rough with my non-liner problem), and I am looking for an R code approach, if one exists.
I will use some non linear optimization package:
nolptr https://cran.r-project.org/web/packages/nloptr/index.html
NlcOptim https://cran.r-project.org/web/packages/NlcOptim/index.html
optimx: https://cran.r-project.org/web/packages/optimx/index.html
Updated:
I'm trying to examine the variability in my parameter estimates from a merMod object by simulating known data and running the model 100 times. I'd like the result to be a data frame that looks like the following:
| simulation | intercept | est.x1 | est.x2 |
| ---------- | --------- | ------ | ------ |
| sim_study1 |.09 |.75 |.25 |
| sim_study2 |.10 |.72 |.21 |
| sim_study3 |NA |NA |NA |
My code to generate multilevel data with a random intercept and 2 predictors is:
# note. this code block runs as expected, and if I run a lmer() call
# on a simulated data set I get values that one would expect.
gen_fake <- function(i, j){
school <- rep(1:j)
person <- rep(1:i) # students nested in schools
# parameters
mu_a_true <- 0.10 # real intercept
sigma_a_true <- 0.10 # varince of intercept
sigma_y_true <- 0.40
b1_true <- .75
b2_true <- .25
# random intercept for schools
a_true <- rnorm(j, mu_a_true, sigma_a_true)
# random data for predictors
x1 <- rnorm(i, 0, 1)
x2 <- rnorm(i, 0, 1)
# outcome
y <- rnorm(i, a_true[school] + b1_true*x1 + b2_true*x2, sigma_y_true)
return (data.frame(y, person, school, x1, x2))
}
I'm attempting to conduct a 100 simulations of a model, while generating new data each time. Note, I'm trying to implement tryCatch within the loop because with more complex models, where the model might not terminate normally, I'd like value returned in the table to be NA for parameters.
My code for this is as follows:
# create an empty data frame with names of parameters (there's probably
# a slicker way to do this within the loop where I can match names from
# the model call)
sim_results <- data.frame(matrix(nrow=100, ncol=3,
dimnames=list(c(),
c("intercept",
"est.x1", "est.x2"))),
stringsAsFactors=F)
# load library for analysis
library(lme4)
# conduct 100 simulations of the model generating fake data for each run
sim_study <- function (i, j, n.sims){
for (sim in 1:n.sims){
fake_dat <- gen_fake(i, j)
tryCatch({
lmer_sim <- lmer(y ~ x1 + x2 + (1|school), data = fake_dat)
}, error = function(e){
return(NA)
}) #return previous value of fm if error
estimates <- rbind(fixef(lmer_sim))
}
sim_results[sim,] <- estimates
}
# run the simulation study
sim_study (1000,5,100)
The issue I am having is that the function only returns 1 row and it isn’t populating the empty data frame I made:
(Intercept) x1 x2
[1,] 0.09659339 0.7746392 0.2325391
I'm unsure of the issue. Finally, any feedback you might have for how to make this work faster would also be appreciated, as I'd like to learn more about that issue. Thanks for any assistance.
This may be a bit of a forehead-slapper, but I think you just misplaced the loop brackets? This works for me:
sim_study <- function (i, j, n.sims){
for (sim in 1:n.sims) {
if (sim %% 10 == 0 ) cat(".\n") ## print progress
fake_dat <- gen_fake(i, j)
tryCatch({
lmer_sim <- lmer(y ~ x1 + x2 + (1|school),
data = fake_dat)
}, error = function(e){
return(rep(NA,3)) ## return vector of correct length
}) #return previous value of fm if error
estimates <- rbind(fixef(lmer_sim))
sim_results[sim,] <- estimates
}
return(sim_results)
}
A few more points:
I'm not sure whether the tryCatch() logic works, since I didn't hit any errors (but I think it ought to be modified to return an object with the current length, as above)
you could replace some of your gen_fake (not the generation of the predictors, but the generation of the response with the built-in ?simulate.merMod(), but I don't think it would actually work any better (or worse)
speeding this up significantly would be a bit of work/hacky. There is a refit() function that works quickly if only the predictor variable has changed, but it doesn't hold in this case. You could use the tricks specified here ...
I am trying to bootstrap lambda1 parameter in LASSO regression (using library penalized) (NOT the coefficients estimates as i KNOW that is does not make sense to calculate e.g. 95% CIs for them, this is the question about lambda1 ONLY).
This is where I am so far:
df <- read.table(header=T, text="group class v1 v2
1 Ala 1 3.98 23.2
2 Ala 2 5.37 18.5
3 C 1 4.73 22.1
4 B 1 4.17 22.3
5 C 2 4.47 22.4
")
Tried this:
X<-df[,c(3,4)] # data, variables in columns, cases in rows
Y<-df[,2] # dichotomous response
for (i 1:100) {
opt1<-optL1(Y,X)
opt1$lambda
}
But got Error: unexpected "}" in "}"
Tried this:
f<-function(X,Y,i){
opt1<-optL1(Y,X,[i])
}
boot(X,f,100)
But got Error in boot (X,f,100): incorrect number of subscripts on matrix... Can somebody help?
Here is what is wrong with the for loop:
1) It needs the syntax for (i in 1:100) {} in order to work;
2) It needs to save opt1$lambda in a proper object;
3) It most likely needs the values (Y,X) to change from one iteration of the loop to another.
The R code which addresses items 1) and 2) above could be written as follows:
lambda <- NULL
for (i in 1:100) {
opt1 <- optL1(Y,X) # opt1 will NOT change
# since Y and X are the SAME
# over each iteration of the for loop
lambda <- c(lambda, opt1$lambda)
}
lambda
In this code, the object lambda which will store the value opt1$lambda produced at each iteration is declared at the top of the for loop with the command lambda -> NULL and then it is augmented after each iteration with the command
lambda <- c(lambda, opt1$lambda).
In general, using the NULL trick is not recommended for a large number of iterations. A better alternative would be this:
lambda <- list('vector', 100)
for (i in 1:100) {
opt1 <- optL1(Y,X) # opt1 will NOT change
# since Y and X are the SAME
# over each iteration of the for loop
lambda[i] <- opt1$lambda
}
lambda <- unlist(lambda)
lambda
With this second alternative, we pre-allocate lambda at the top of the for loop to be a list with 100 components, such that the i-th component will store the value opt1$lambda produced during the i-th iteration. Inside the for loop, we save the value of opt1$lambda in the list named lambda with the command:
lambda[i] <- opt1$lambda.
At the end of the loop, we unlist lambda so that it becomes a regular vector (i.e., column of numbers).
You can alter the function to take in a data.frame, and specific the columns to use for response and covariate inside optL1 :
library(boot)
library(penalized)
f<-function(data,ind){
fit = optL1(data[ind,"class"],data[ind,c("v1","v2")])
fit$lambda
}
df = data.frame(group=sample(c("A","B","C"),100,replace=TRUE),
class=sample(2,100,replace=TRUE),
v1 = rnorm(100),
v2 = rnorm(100)
)
bo = boot(df,f,100)
o
ORDINARY NONPARAMETRIC BOOTSTRAP
Call:
boot(data = df, statistic = f, R = 100)
Bootstrap Statistics :
original bias std. error
t1* 2.887399 0.2768409 1.85466
novice here. I am fitting a negative binomial model on count data where Y is the count of events, D is the treatment, and X is a logarithmic offset:
out <- glm.nb(y ~ d + offset(log(x)),data=d1)
I would like to bootstrap the confidence intervals of the first difference between D=1 and D=0. I've gotten this far, but not sure if it is the correct approach:
holder <- matrix(NA,1200,1)
out <- out <- glm.nb(y ~ d + offset(log(x)),data=d1)
for (i in 1:1200){
q <- sample(1:nrow(d1), 1)
d2 <- d1[q,]
d1_1 <- d1_2 <- d2
d1_1$d <- 1
d1_2$d <- 0
d1pred <- predict(out,d1_1,type="response")
d2pred <- predict(out,d1_2,type="response")
holder[i,1] <- (d1pred[1] - d2pred[1])
}
mean(holder)
Is this the correct way to bootstrap the first difference?
Generally, your approach is ok, but you can do it in more R-ish way. Firstly, if you are serious about bootstrapping you can employ boot library and benefit from more compact code, no loops and many other advanced options.
In your case it can look like:
## Data generation
N <- 100
set.seed(1)
d1 <- data.frame(y=rbinom(N, N, 0.5),
d=rbinom(N, 1, 0.5),
x=rnorm(N, 10, 3))
## Model
out <- glm.nb(y ~ d + offset(log(x)), data=d1)
## Statistic function (what we are bootstrapping)
## Returns difference between D=1 and D=0
diff <- function(x,i,model){
v1 <- v2 <- x[i,]
v1$d <- 1
v2$d <- 0
predict(model,v1,type="response") - predict(model,v2,type="response")
}
## Bootstrapping itself
b <- boot(d1, diff, R=5e3, model=out)
mean(b$t)
Now b$t holds bootstrapped values. See names(b) and/or ?boot for extra information.
Bootstrapping is time consuming operation, and one of the obvious advantage of boot library is support for parallel operations. It's as easy as:
b <- boot(d1, diff, R=5e3, model=out, parallel="multicore", ncpus=2)
If you are on Windows use parallel="snow" instead.
I'm using either dyn or dynlm to predict time series using lagged variables.
However, the predict function in either case only evaluates one time step at a time, taking a constant time of 24 milliseconds per step on my computer, or about 1.8 hours for my dataset, which is super long, given that the entire regression takes about 10 seconds.
So, I'm thinking that perhaps the fastest thing might be just to evaluate the formula by hand?
So, is there some way of evaluating a formula given values in a data.frame or the current envrironment or similar?
I'm thinking of something along the lines of:
evalMagic( load ~ temperature + time, data.frame( temperature = 10, time = 4 ) )
I suppose, as I write this, that we need to handle the coefficients somehow, something like:
evalMagic( load ~ temperature + time, data.frame( temperature = 10, time = 4 ), model$coefficients )
.... so this raises the questions of:
isn't this what predict is supposed to do?
why is predict so slow?
what options do I have to make the prediction a bit faster? After all, it's not inverting any matrices or something, it's just a bit of arithmetic!
I wrote my own lag implementation in the end. It's hacky and not beautiful, but it's a lot faster. It can process 1000 rows in 4 seconds on my crappy laptop.
# lags is a data.frame, eg:
# var amount
# y 1
# y 2
addLags <- function( dataset, lags ) {
N <- nrow(dataset)
print(lags)
if( nrow(lags) > 0 ) {
print(lags)
for( j in 1:nrow(lags) ) {
sourcename <- as.character( lags[j,"var"] )
k <- lags[j,"amount"]
cat("k",k,"sourcename",sourcename,"\n")
lagcolname <- sprintf("%s_%d",sourcename,k)
dataset[,lagcolname] <- c(rep(0,k), dataset[1:(N-k),sourcename])
}
}
dataset
}
lmLagged <- function( formula, train, lags ) {
# get largest lag, and skip that
N <- nrow(train)
skip <- 0
for( j in 1:nrow(lags) ) {
k <- lags[j,"amount"]
skip <- max(k,skip)
}
print(train)
train <- addLags( train, lags )
print(train)
lm( formula, train[(skip+1):N,] )
}
# pass in training data, test data,
# it will step through one by one
# need to give dependent var name
# lags is a data.frame, eg:
# var amount
# y 1
# y 2
predictLagged <- function( model, train, test, dependentvarname, lags ) {
Ntrain <- nrow(train)
Ntest <- nrow(test)
test[,dependentvarname] <- NA
testtraindata <- rbind( train, test )
testtraindata <- addLags( testtraindata, lags )
for( i in 1:Ntest ) {
thistestdata <- testtraindata[Ntrain + i,]
result <- predict(model,newdata=thistestdata)
for( j in 1:nrow(lags) ) {
sourcename <- lags[j,"var"]
k <- lags[j,"amount"]
lagcolname <- sprintf("%s_%d",sourcename,k)
testtraindata[Ntrain + i + k,lagcolname] <- result
}
testtraindata[Ntrain+i,dependentvarname] <- result
}
return( testtraindata[(Ntrain+1):(Ntrain + Ntest),dependentvarname] )
}
library("RUnit")
# size of training data
N <- 6
predictN <- 50
# create training data, which we can get exact fit on
set.seed(1)
x = sample( 100, N )
traindata <- numeric()
traindata[1] <- 1 + 1.1 * x[1]
traindata[2] <- 2 + 1.1 * x[2]
for( i in 3:N ) {
traindata[i] <- 0.5 + 0.3 * traindata[i-2] - 0.8 * traindata[i-1] + 1.1 * x[i]
}
train <- data.frame(x = x, y = traindata, foo = 1)
#train$x <- NULL
# create testing data, bunch of NAs
test <- data.frame( x = sample(100,predictN), y = rep(NA,predictN), foo = 1)
# specify which lags we need to handle
# one row per lag, with name of variable we are lagging, and the distance
# we can then use these in the formula, eg y_1, and y_2
# are y lagged by 1 and 2 respectively
# It's hacky but it kind of works...
lags <- data.frame( var = c("y","y"), amount = c(1,2) )
# fit a model
model <- lmLagged( y ~ x + y_1 + y_2, train, lags )
# look at the model, it's a perfect fit. Nice!
print(model)
print(system.time( test <- predictLagged( model, train, test, "y", lags ) ))
#checkEqualsNumeric( 69.10228, test[56-6], tolerance = 0.0001 )
#checkEquals( 2972.159, test$y[106-6] )
print(test)
# nice plot
plot(test, type='l')
Output:
> source("test/test.regressionlagged.r",echo=F)
Call:
lm(formula = formula, data = train[(skip + 1):N, ])
Coefficients:
(Intercept) x y_1 y_2
0.5 1.1 -0.8 0.3
user system elapsed
0.204 0.000 0.204
[1] -19.108620 131.494916 -42.228519 80.331290 -54.433588 86.846257
[7] -13.807082 77.199543 12.698241 64.101270 56.428457 72.487616
[13] -3.161555 99.575529 8.991110 44.079771 28.433517 3.077118
[19] 30.768361 12.008447 2.323751 36.343533 67.822299 -13.154779
[25] 72.070513 -11.602844 115.003429 -79.583596 164.667906 -102.309403
[31] 193.347894 -176.071136 254.361277 -225.010363 349.216673 -299.076448
[37] 400.626160 -371.223862 453.966938 -420.140709 560.802649 -542.284332
[43] 701.568260 -679.439907 839.222404 -773.509895 897.474637 -935.232679
[49] 1022.328534 -991.232631
There's about 12 hours work in those 91 lines of code. Ok, I confess I played Plants and Zombies for a bit. So, 10 hours. Plus lunch and dinner. Still, quite a lot of work anyway.
If we change predictN to 1000, I get about 4.1 seconds from the system.time call.
I think it's faster because:
we don't use timeseries; I suspect that speeds things up
we don't use dynamic lm libraries, just normal lm; I guess that's slightly faster
we only pass a single row of data into predict for each prediction, which I think is significantly faster, eg using dyn$lm or dynmlm, if one has a lag of 30, one would need to pass 31 rows of data into predict AFAIK
a lot less data.frame/matrix copying, since we just update the lag values in-place on each iteration
Edit: corrected minor buggette where predictLagged returned a multi-column data-frame instead of just a numeric vector
Edit2: corrected less minor bug where you couldn't add more than one variable. Also reconciled the comments and code for lags, and changed the lags structure to "var" and "amount" in place of "name" and "lags". Also, updated the test code to add a second variable.
Edit: there are tons of bugs in this version, which I know, because I've unit-tested it a bit more and fixed them, but copying and pasting is very time-consuming, so I will update this post in a few days, once my deadline is over.
Maybe you're looking for this:
fastlinpred <- function(formula, newdata, coefs) {
X <- model.matrix( formula, data=newdata)
X %*% coefs
}
coefs <- c(1,2,3)
dd <- data.frame( temperature = 10, time = 4 )
fastlinpred( ~ temperature + time,
dd , coefs )
This assumes that the formula has only a RHS (you can get rid of the LHS of a formula by doing form[-2]).
This certainly gets rid of a lot of the overhead of predict.lm, but I don't know if it is as fast as you want. model.matrix has a lot of internal machinery too.