Improve RSelenium loop, reduce time navigating to the page - r

Question

I sometimes have to check that our webpages are loading correctly. This usually involves thousands of links. I wrote a for-loop to do this and would like to know if there is a way to cut down the time. I tried doing parallel processing and couldn't figure it out. I'm new to R and selenium is kind of a mystery to me. It seems that waiting for the $navigate() part is where it hangs. It takes about 3-5 sec for this to be complete before moving to the next one.
I tried having 10 browsers going at once and offset so that some steps could run while the page loads but it didn't seem to work. The last assignment I did took 11 hours for 8,200 links. If I could get this to go faster, I'd like to know how. Any help would be appreciated.
library(RSelenium)
#Reproducible data
URL <- read.table(header=TRUE, stringsAsFactors=FALSE,sep=",",text=
"Link,file,Corr
http://stackoverflow.com/questions,questions.png,0
http://www.google.com/,MATCH.png,0
http://stackoverflow.com/unanswered,unanswered.png,0
http://www.google.com,google.png,0")
#Starts Selenium server
checkForServer()
startServer()
#create browsers
for(i in 1:4)
{
name<- paste("remDr", i, sep="")
assign(name, remoteDriver$new())
get(name)$open()
get(name)$maxWindowSize()
}
#Go to site, & save screenshots
system.time(
for(i in seq(0,4, by=4))
{
#goes to link, takes 2.5 each
remDr1$navigate(URL$Link[i+1])
remDr2$navigate(URL$Link[i+2])
remDr3$navigate(URL$Link[i+3])
remDr4$navigate(URL$Link[i+4])
#waits until page loads or max 30 sec, usually takes no time
remDr1$setTimeout(type = "page load", milliseconds = 30000)
remDr2$setTimeout(type = "page load", milliseconds = 30000)
remDr3$setTimeout(type = "page load", milliseconds = 30000)
remDr4$setTimeout(type = "page load", milliseconds = 30000)
#saves screenshot
remDr1$screenshot(file = URL$file[i+1])
remDr2$screenshot(file = URL$file[i+2])
remDr3$screenshot(file = URL$file[i+3])
remDr4$screenshot(file = URL$file[i+4])
}
)
PS: Perhaps this interests someone: I then compare the screenshots to find the ones that match what I'm looking for
library(raster)
library(doSNOW)
library(doParallel)
#Create cluster for parallel processing
cl <- makeCluster(4)
registerDoSNOW(cl)
#Make a raster of the page I want to find copies of
MATCH <- raster("MATCH.png")
#for loop to compare pics
system.time(
URL$Corr <- unlist(
foreach(h=1:4, .packages = "raster", .verbose = T) %dopar%
{
comp <- raster(URL$file[h])
comp.samp <- round(resample(comp, MATCH, "bilinear"))
round(cor(getValues(MATCH),getValues(comp.samp), use="complete.obs"), 6)
})
)
MATCH <- URL$file[which(URL$Corr==1)]
MATCH #lists pages that match
stopCluster(cl)

Answer 1

Yes in java there a library called selnium grid. You can use selenium grid which can help you distribute the load into multiple browsers in multiple machines. You will need more than 1 machine ( Called nodes where you can run the tests on browsers (upto 5 chrome browser on each node) and fire the tests to a central hub which will control and manage these nodes. This is a good start. You can also find many questions in selenium-grid which can help you.