rstan MCMC: Different squence of data resulting in different results, why? - r

Question

I am new to Stan and rstan.
I recently may find a weird issue when I worked on Markov chain Monte Carlo (MCMC). In short, for example, the data has 10 observations, say ID 1 to 10. Now, I permutate it by shifting the 10th row between the original first and second rows, say ID 1, 10, and 2 to 9. Two different scenarios of data will give different estimates, even I fix the same random seed.
To illustrate the issue in a simpler way, I write the following R scripts.
##TEST 01
# generate data
N <- 100
set.seed(123)
Y <- rnorm(N, 1.6, 0.2)
stan_code1 <- "
data {
int <lower=0> N; //number of data
real Y[N]; //data in an (C++) array
}
parameters {
real mu; //mean parameter of a normal distribution
real <lower=0> sigma; //standard deviation parameter of a normal distribution
}
model {
//prior distributions for parameters
mu ~ normal(1.7, 0.3);
sigma ~ cauchy(0, 1);
//likelihood of Y given parameters
for (i in 1:N) {
Y[i] ~ normal(mu, sigma);
}
}
"
# compile model
library(rstan)
model1 <- stan_model(model_code = stan_code1) #usually, take half a minute to run
# pass data to stan and run model
set.seed(123)
fit <- sampling(model1, list(N=N, Y=Y), iter=200, chains=4)
print(fit)
# mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
# mu 1.62 0.00 0.02 1.58 1.61 1.62 1.63 1.66 473 1.00
# sigma 0.18 0.00 0.01 0.16 0.18 0.18 0.19 0.21 141 1.02
# lp__ 117.84 0.07 0.85 115.77 117.37 118.07 118.51 118.78 169 1.01
Yp <- Y[c(1,100,2:99)]
set.seed(123)
fit2 <- sampling(model1, list(N=N, Y=Yp), iter=200, chains=4)
print(fit2)
# mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
# mu 1.62 0.00 0.02 1.59 1.61 1.62 1.63 1.66 480 0.99
# sigma 0.18 0.00 0.01 0.16 0.18 0.18 0.19 0.21 139 1.02
# lp__ 117.79 0.09 0.95 115.72 117.35 118.05 118.49 118.77 124 1.01
As we can see from the above simple case, two results fit and fit2 are different.
And, even stranger, if I write the likelihood before the priors (previousy, the priors are written ahead of the likelihood) in code file, the same random seed and the same data will still give different estimates.
##TEST 01'
# generate data
#N <- 100
set.seed(123)
Y <- rnorm(N, 1.6, 0.2)
stan_code11 <- "
data {
int <lower=0> N; //number of data
real Y[N]; //data in an (C++) array
}
parameters {
real mu; //mean parameter of a normal distribution
real <lower=0> sigma; //standard deviation parameter of a normal distribution
}
model {
//likelihood of Y given parameters
for (i in 1:N) {
Y[i] ~ normal(mu, sigma);
}
//prior distributions for parameters
mu ~ normal(1.7, 0.3);
sigma ~ cauchy(0, 1);
}
"
# compile model
#library(rstan)
model11 <- stan_model(model_code = stan_code11) #usually, take half a minute to run
# pass data to stan and run model
set.seed(123)
fit11 <- sampling(model11, list(N=N, Y=Y), iter=200, chains=4)
print(fit11)
# mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
# mu 1.62 0.00 0.02 1.58 1.61 1.62 1.63 1.66 455 0.99
# sigma 0.19 0.00 0.01 0.16 0.18 0.18 0.20 0.21 94 1.04
# lp__ 117.68 0.08 0.93 115.24 117.18 117.90 118.45 118.77 149 1.01
##TEST01 was
# mean se_mean sd 2.5% 25% 50% 75% 97.5% n_eff Rhat
# mu 1.62 0.00 0.02 1.58 1.61 1.62 1.63 1.66 473 1.00
# sigma 0.18 0.00 0.01 0.16 0.18 0.18 0.19 0.21 141 1.02
# lp__ 117.84 0.07 0.85 115.77 117.37 118.07 118.51 118.78 169 1.01

Answer 1

Stan does not utilize the same pseudo random-number generator as R. Thus, calling set.seed(123) only makes Y repeatable and does not make the MCMC sampling repeatable. In order to accomplish the later, you need to pass an integer as the seed argument to the stan (or sampling) function in the rstan package like
sampling(model11, list(N = N, Y = Y), seed = 1234).
Even then, I could imagine that permuting the observations could result in different realizations of the draws from the posterior distribution due to floating-point reasons. But none of this really matters (unless you conduct too few iterations or get warning messages) because the posterior distribution is the same even if a finite set of realizations from the posterior distribution are randomly different numbers.