R Temporarily write csv files during loop in R - r

Question

I have a loop which downloads data from urls. Now I would like that for every x iterations, the information so far gets written away.
As such I have the following code:
baseurl <- "http://zoeken.kvk.nl/Address.ashx?site=handelsregister&partialfields=&q=010"
pages3 <- list()
for(i in 1:99999){
if(i < 10){
mydata <- RJSONIO::fromJSON(paste0(baseurl,"00000",i), flatten=TRUE)
}
if(i < 100 & i >= 10){
mydata <- RJSONIO::fromJSON(paste0(baseurl,"0000",i), flatten=TRUE)
}
if(i < 1000 & i >= 100){
mydata <- RJSONIO::fromJSON(paste0(baseurl,"000",i), flatten=TRUE)
}
if(i < 10000 & i >= 1000){
mydata <- RJSONIO::fromJSON(paste0(baseurl,"00",i), flatten=TRUE)
}
if(i < 100000 & i >= 10000){
mydata <- RJSONIO::fromJSON(paste0(baseurl,"0",i), flatten=TRUE)
}
if(i < 1000000 & i >= 100000){
mydata <- RJSONIO::fromJSON(paste0(baseurl,i), flatten=TRUE)
}
mydata <- RJSONIO::fromJSON(paste0(baseurl,i), flatten=TRUE)
pages3[[i]] <- mydata$resultatenHR
options(timeout = 4000000)
if(i %% 100 == 0){Sys.sleep(5)}
if(i %% 1000 == 0){
final_df<-do.call(rbind,pages3)
final<- Reduce(function(x,y) merge(x, y, all=TRUE), final_df)
mytime <- format(Sys.time(), "%b_%d_%H_%M_%S_%Y")
myfile <- file.path(R(), paste0(mytime, "_", i, ".csv"))
write.csv2(final, file = myfile, sep = "", row.names = FALSE, col.names = FALSE,
quote = FALSE, append = FALSE)
}
}
}
}
However, nothing gets saved in the meantime? Where did I go wrong with the code? Thank you for your insights.

Answer 1

I think that your problem is probably in:
myfile <- file.path(R(), paste0(mytime, "_", i, ".csv"))
As R thinks R() is a function.
Error in R() : could not find function "R"
You can change it to getwd() if you want (don't forget to set working directory first setwd()) or specify a different path .
In addition, here: write.csv2(final, file = myfile, sep = "", row.names = FALSE, col.names = FALSE, quote = FALSE, append = FALSE), your forgot to write paste(), and for convince you can delete the default arguments you used.
write.csv2(final, file = paste(myfile, sep = "" ))
#Edit
This is probably not the most efficient way but it will probably do the trick.
The main issue is that you append the pages3 list object/csv file by the url index. If you'd create a new index for pages3, you can reset it every i urls.
setwd("Your working directory path")
baseurl <- "http://zoeken.kvk.nl/Address.ashx?site=handelsregister&partialfields=&q=010"
pages3 <- list()
#Counter for the url loop
i <- 1
#Counter for the appended csv file/ list object pages3
k <- 1
for(i in 1:99999){
#Read JSON file by i index
mydata <- RJSONIO::fromJSON(paste0(baseurl,i), flatten=TRUE)
#Appending to the Pages3 list object by k index
pages3[[k]] <- mydata$resultatenHR
# Increasing the k counter
k <- k + 1
options(timeout = 4000000)
if(i %% 100 == 0) {Sys.sleep(5)}
if(i %% 1000 == 0) {
final_df <- do.call(rbind, pages3)
final <- Reduce(function(x,y) merge(x, y, all=TRUE), final_df)
mytime <- format(Sys.time(), "%b_%d_%H_%M_%S_%Y")
myfile <- file.path(getwd(), paste0(mytime, "_", i, ".csv"))
write.csv2(final, file = paste(myfile, sep = "" ))
#Resetting the pages3 list object
pages3 <- NULL
#Resting the k index counter
k <- 1
}
}
However, depending on your computer/the size of the files you try to import, maybe it would be more efficient to save and split to different csv files when you finished imported all the urls.