Write output of R loop to output file or indicate progress? - r

Question

I'm running quite a complex loop in R that takes ~ 20-30 days to run. The details of the loop might not be too relevant (it's the Viterbi algorithm to decode the most likely state sequence for post-processing the output of a hidden Markov model). So far, once it has been successively going through all the loops, I was writing the results to a data file.
saveRDS(z_star, file = "z_star.Rda")
However, this is quite frustrating, as I cannot see how far the loops have already made it and I fear that in the end something could happen that ruins all the computation progress (e.g., the resulting data frame could be larger than the RAM of the server I'm using).
My question is if I should rather 1) write to an output file with every loop, or 2) implement some kind of progress tracking. For both options I have no concrete idea on how to implement it and would therefore be very grateful for code.
N_draws = 2000
N = 100000
z_star = matrix(nrow = N_draws, ncol = N)
best_logp = data.frame(matrix(nrow = N, ncol = S))
back_ptr = best_logp
for(d in 1:N_draws){
for (k in 1:K)
best_logp[1, k] = dnorm(y[1],
mean = mu[k],
sd = sigma_q[d],
log = TRUE)
for (t in 2:N) {
for (k in 1:K) {
best_logp[t, k] = -Inf
for (j in 1:K) {
real logp;
logp = best_logp[t-1, j] + log(theta[j, k]) + dnorm(y[t],
mean = mu[k],
sd = sigma_q[d],
log = TRUE)
if (logp > best_logp[t, k]) {
back_ptr[t, k] = j;
best_logp[t, k] = logp;
}
}
}
}
log_p_z_star = max(best_logp[N]);
for (k in 1:K)
if (best_logp[N, k] == log_p_z_star)
z_star[N] = k;
for (t in 1:(N - 1))
z_star[N - t] = back_ptr[N - t + 1, z_star[N - t + 1]];
}

Answer 1

Thanks for your thoughts, I decided to do something simple:
I used the index and the system time to indicate the progress using
for(d in 1:N_draws){
old <- Sys.time()
cat(c("draw", d, "out of", N_draws, "\n"))
and before the last bracket closes
new = Sys.time() - old
cat(c("time for draw", d, ":", new, "seconds", "\n"))
percentage_completed = d/N_draws * 100
cat(c("percentage completed:", percentage_completed, "percent", "\n"))
}
returning
draw 1 out of 2000
time for draw 1 : 0.839755058288574 seconds
percentage completed: 0.05 percent
draw 2 out of 2000
time for draw 2 : 0.735047101974487 seconds
percentage completed: 0.1 percent
draw 3 out of 2000