downloading data in a loop, how often to save that information - r

Question

I'm using R, but this question isn't specific to it. Suppose you've written some loop which downloads a url at each iteration. You want to to save this data, so you could do so each iteration or hold the information and save it every nth iteration. Are there any general rules of thumb for doing this? How slow is it to open and close a file for writing all the time? What I have in mind is
for (i in 1:1000) {
data <- url("http://...i")
write.table(data, file="file")
}
versus something like this
data <- list()
length(data) <- 20
j <- 1
for (i in 1:1000) {
data[j] <-url("http://...i")
j <- j+1
if (j = 20) {j <-1}
if (i %% 20 == 0) {
write.table(data, file="file")
}
}

Answer 1

If all your downloaded data are of the same form, you might want to append them to a unique file in which case you can do that at each iteration. Here is a short example:
sites<-c("714A","715A","716A")
for(i in 1:length(sites)){
data<-read.table(file=paste("http://www.ngdc.noaa.gov/mgg/geology/odp/data/115/",sites[i],"paleomag.txt",sep="/"),sep="\t",header=TRUE)
#In this example i downloaded paleomagnetic data from deep sea drilling sites.
ifelse(i==1,h<-TRUE,h<-FALSE) #Here the idea is that we want to output the column names only the first time.
write.table(data,file="paleomag_leg115.txt",sep="\t",append=!h,col.names=h,row.names=FALSE)
}