How to speed up a while-loop in R (perhaps using dopar)? - r

Question

I'm trying to process a huge text file containing dozens millions lines of text. The text file contains the results of a convnet analysis of several millions of images and looks like this:
CUDNN_HALF=1
net.optimized_memory = 0
mini_batch = 1, batch = 8, time_steps = 1, train = 0
nms_kind: greedynms (1), beta = 0.600000
nms_kind: greedynms (1), beta = 0.600000
nms_kind: greedynms (1), beta = 0.600000
seen 64, trained: 447 K-images (6 Kilo-batches_64)
Enter Image Path: data/obj1/H001683-19-1-5-OCT2 [x=13390,y=52118,w=256,h=256].png: Predicted in 19.894000 milli-seconds.
tumor: 99% (left_x: 2 top_y: 160 width: 67 height: 34)
bcell: 98% (left_x: 6 top_y: 54 width: 32 height: 22)
bcell: 80% (left_x: 51 top_y: 0 width: 30 height: 16)
bcell: 98% (left_x: 52 top_y: 198 width: 28 height: 26)
bcell: 98% (left_x: 150 top_y: 216 width: 35 height: 23)
bcell: 56% (left_x: 150 top_y: 78 width: 45 height: 30)
bcell: 91% (left_x: 187 top_y: 132 width: 31 height: 26)
bcell: 96% (left_x: 219 top_y: 185 width: 20 height: 26)
bcell: 37% (left_x: 222 top_y: -0 width: 24 height: 4)
bcell: 98% (left_x: 241 top_y: 208 width: 15 height: 21)
bcell: 64% (left_x: 248 top_y: 35 width: 8 height: 35)
[... a lot of similar lines...]
Enter Image Path: data/obj1/H001683-19-1-5-OCT2 [x=13390,y=52530,w=256,h=256].png: Predicted in 19.195000 milli-seconds.
bcell: 97% (left_x: 45 top_y: 180 width: 29 height: 24)
bcell: 58% (left_x: 59 top_y: 1 width: 35 height: 22)
tumor: 98% (left_x: 105 top_y: 143 width: 99 height: 44)
tumor: 97% (left_x: 113 top_y: 50 width: 57 height: 40)
bcell: 96% (left_x: 191 top_y: 194 width: 29 height: 27)
bcell: 99% (left_x: 201 top_y: 129 width: 34 height: 22)
Enter Image Path:
Each image is mentioned by the image file name after "Enter Image Path" followed by a list of objects which were identified. I do not know a priori, how many objects (here tumor an bcell) are in each image. Sometimes where are no objects at all, sometimes where are hundreds.
I first tried to read the whole file using
test11<-readLines("result.txt")
picsna<-grep(test11,pattern="Enter Image") # line numbers with the image file name
lle<-length(picsna) # length for the subsequent script
and then to go ahead with my script but it proved to take hours to read the file so I came with the Idea to read the file line by line and execute my code "on the fly" using a while-loop:
require(LaF)
n=1
lle<-0 # number of images (to be used in a subsequent code)
picsna<-c() # vector with the line numbers of each image entry
# read the result-file initially (first bunch of lines do not contain image entries
test11<-get_lines(file="result.txt", line_numbers=n)
# as long as the line exists read the next line and do following:
while(is.na(test11)==FALSE){
test11<-get_lines(file="result.txt", line_numbers=n+1)
# I wanted to know how far my reading progressed but had a feeling, print slowed down the loop
#print(n)
# I found here this solution for printing progress periodically
if(n %% 10000==0) {
cat(paste0("iteration: ", n, "\n"))
}
# look for image entry and save the line number (not the iteration number)
if(grepl(test11,pattern="Enter Image")==TRUE){
picsna<-c(picsna,n+1)
lle<-lle+1} # increase the number of images
n<-n+1
}
# the last line of the file is always incomplete but has to be added to the vector to calculate the number of objects (in a following script not shown here) if the previous image had any.
if(is.na(test11)==TRUE){
picsna<-c(picsna,n)
print("The End")
lle<-lle+1
}
I measured the elapsed time for the first and the second script on a small result file containing about 200 lines. The second script was even a little bit slower (0.04 vs 0.01), which confused me.
I thought about rewriting it in a foreach-%dopar%-loop but could not realize how to implement it with the readLines-function or with my while-loop. My problem is, that I do not know apriori how many lines the file contains. I would really appreciate if somebody could help me to parallelize my script!

Answer 1

Thank you #Bas! I tested your suggestion on a Linux machine: for a file with ~239 million lines it took less than 1 min. By adding >lines.txt I could save the results. Interestingly, my first readLines R script needed "only" 29 min, which was surprisingly fast compared with my first experience (so I might have had some problem with my Windows computer at work which was not related to R).