RSelenium through docker - r

My OS is windows 8.1 and I have the version 3.3.3 of R.
I have installed the RSelenium packages and I try to run it using this:
library("RSelenium")
#start RSelenium server
startServer()
checkForServer()
and I receive this error:
Error: checkForServer is now defunct. Users in future can find the function in
file.path(find.package("RSelenium"), "examples/serverUtils"). The
recommended way to run a selenium server is via Docker. Alternatively
see the RSelenium::rsDriver function.
Is there anything changed in the way RSelenium opens? I search for the error and I found only this but it doesn't help me. What can I do?
Also an alternative I tried is to download the chromedrive from here 'https://sites.google.com/a/chromium.org/chromedriver/downloads'
and using this script:
require(RSelenium)
cprof <- getChromeProfile("C:/Users/Peri/Desktop/chromedriver/chromedriver.exe", "Profile 1")
require(RSelenium)
remDr <- remoteDriver(remoteServerAddr = "localhost"
, port = 4444
, browserName = "chrome", extraCapabilities = cprof
)
remDr$open()
and I receive this error:
Error in checkError(res) :
Couldnt connect to host on http://localhost:4444/wd/hub.
Please ensure a Selenium server is running.
what can I do to run chrome instead of the pre-default browser Firefox?

You need to use the function rsDriver. The Selenium Version wants you to use Docker (which I also would recommend), but if you are not familiar with this you can go this way.
rsdriver will manage the binaries needed for running a Selenium Server. This provides a wrapper around the wdman::selenium function.
Here is what you have to do to start a Chrome Browser:
driver<- rsDriver()
remDr <- driver[["client"]]
And then you can work with it:
remDr$navigate("http://www.google.de")
remDr$navigate("http://www.spiegel.de")
And stop it:
remDr$close()

Related

Scrape website with load more button with R (rvest)

I'm trying to scrape a website with a load more button. I've set up a selenium server by using a windows prompt. The Selenium server is running, but I keep getting the following error when I run the script in R. I've red a lot of blog posts and tried to find the answer, but I lack the technical knowledge to figure this out, so I hope someone is willing to help me.
Error
[1] "Connecting to remote server"
Selenium message:The path to the driver executable must be set by the
webdriver.gecko.driver system property; for more information, see
https://github.com/mozilla/geckodriver. The latest version can be
downloaded from https://github.com/mozilla/geckodriver/releases
Error: Summary: UnknownError Detail: An unknown server-side error
occurred while processing the command. class:
java.lang.IllegalStateException Further Details: run errorDetails
method
Windows prompt
cd c:\selenium
java -Dwebdriver.chrome.driver=c:\geckodriver\chromedriver.exe -
Dwebdriver.gecko.driver.driver=c:\geckodriver\geckodriver.exe -jar selenium-
server-standalone-3.4.0.jar
R SCRIPT
library(rvest)
library(RSelenium)
library(stringr)
library(xm12)
library(wdman)
url <- "https://www.social-enterprise.nl/wie-doen-het/"
remDr <- remoteDriver()
# Open the browser webpage
remDr$open()
#navigate to your page
remDr$navigate(url)
# Locate the load more button
loadmorebutton <- remDr$findElement(using = 'css selector', "#morenews")
for (i in 1:2){
print(i)
loadmorebutton$clickElement()
Sys.sleep(30)
}
page_source<-remDr$getPageSource()
Merken <- read_html(page_source[[1]]) %>%
html_nodes("#membershipCntr span") %>%
html_text()
remDr$close()
You are missing some options in remote web driver instantiate. You can try the following code,
remDr <- remoteDriver(remoteServerAddr = "localhost"
, port = 4444L
, browserName = "firefox"
)

Rselenium issue

I'm trying to scrape a website using Rselenium. However I'm getting an error:
Error: checkForServer is now defunct. Users in future can find the function in
file.path(find.package("RSelenium"), "examples/serverUtils"). The
recommended way to run a selenium server is via Docker. Alternatively
see the RSelenium::rsDriver function.
My chrome is updted to version 58 and moxilla to version 45, rselenium used to work earlier but I'm not sure what happened please help guys.
The following script works for me with the new RSelenium...
rD <- rsDriver(port=4444L,browser="chrome")
remDr <- rD$client
remDr$navigate(url)
Just make sure you have docker account and you have it installed.
try this
library('RSelenium')
rD=rsDriver()
remDr =rD[["client"]]
remDr$navigate("https://www.vinmonopolet.no/vmp/Land/Chile/Gato-Negro-Cabernet-Sauvignon-2017/p/295301")
webElement = remDr$findElement('xpath', '//*[#id="product_2953010"]/span[2]')
webElement$clickElement()

RSelenium Error: Can't Connect to Host; Selenium Server is not running

I am getting the following error: "Error in checkError(res) :
Couldnt connect to host on http://localhost:4444/wd/hub.
Please ensure a Selenium server is running."
I'm using a mac version 10.9.5, and downloaded all of the latest versions of packages and java. My code is:
library(rvest)
library(RSelenium)
library(wdman)
setwd(Path to selenium standalone file)
pJS <- phantomjs(pjs_cmd = "/phantomjs-2.1.1-macosx/bin/phantomjs")
remDr <- remoteDriver(browserName = "phantomjs")
Sys.sleep(5)
remDr$open(silent = FALSE)
And then I get the mentioned error. I've tried using the "java -jar selenium-server-standalone.jar" command in the terminal (after us the cd command to navigate to the correct directory). I've tried changing my port in the remoteDriver() function (to 4444, 5556). I've tried various Sys.sleep() times (up to 20 seconds). When I googled this error, most of the fixes were for FireFox or Windows, and not applicable to using PhantomJS
What else can I try?
The RSelenium::phantom function is deprecated. This had a pjs_cmd argument which I think you refer to above. You can use the rsDriver function from the RSelenium or the phantomjs function from the wdman package:
library(RSelenium)
rD <- rsDriver(browser = "phantomjs")
remDr <- rD[["client"]]
# no need for remDr$open a phantom browser is already initialised
remDr$navigate("http://www.google.com/ncr")
....
....
# clean up
rm(rD)
gc()
Alternatively using the wdman package
library(RSelenium)
library(wdman)
pDrv <- phantomjs(port = 4567L)
remDr <- remoteDriver(browserName = "phantomjs", port = 4567L)
remDr$open()
remDr$navigate("http://www.google.com/ncr")
...
...
# clean up
remDr$close()
pDrv$stop()

How to set up rselenium for R?

"everything was better back then"...
since firefox 49 (?) you can't use the rselenium package not straightforward anymore. I have searched the whole internet to find a SIMPLE How To Manual for setting up rselenium but did not find anything relevant and uptodate.
Can someone provide me and all the others out there who have no clue a simple How To manual? Like:
download XY
open AB
so I can run code like the following
require(RSelenium)
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4444L,
browserName = "firefox")
remDr$open()
Download latest version of RSelenium >= 1.7.1. Run the following:
library(RSelenium)
rD <- rsDriver() # runs a chrome browser, wait for necessary files to download
remDr <- rD$client
# no need for remDr$open() browser should already be open
If you want a firefox browser use rsDriver(browser = "firefox").
This is detailed in http://rpubs.com/johndharrison/RSelenium-Basics appendix. The recommended way to run RSelenium is via Docker containers however. Instructions for use of Docker with RSelenium can be found at http://rpubs.com/johndharrison/RSelenium-Docker
ISSUES:
If you have issues which may occur due to admin rights or other variables such as anti-virus software you can run a Selenium server manually. The easiest way to do this is via the wdman package:
selCommand<-
wdman::selenium(jvmargs = c("-Dwebdriver.chrome.verboseLogging=true"),
retcommand = TRUE)
> cat(selCommand)
C:\PROGRA~3\Oracle\Java\javapath\java.exe -Dwebdriver.chrome.verboseLogging=true -Dwebdriver.chrome.driver="C:\Users\john\AppData\Local\binman\binman_chromedriver\win32\2.27/chromedriver.exe" -Dwebdriver.gecko.driver="C:\Users\john\AppData\Local\binman\binman_geckodriver\win64\0.14.0/geckodriver.exe" -Dphantomjs.binary.path="C:\Users\john\AppData\Local\binman\binman_phantomjs\windows\2.1.1/phantomjs-2.1.1-windows/bin/phantomjs.exe" -jar "C:\Users\john\AppData\Local\binman\binman_seleniumserver\generic\3.0.1/selenium-server-standalone-3.0.1.jar" -port 4567
Using one of the wdman functions with the retcommand option enabled will return the
commandline call that would have been ran.
Now you can run the output of cat(selCommand) in a terminal
C:\Users\john>C:\PROGRA~3\Oracle\Java\javapath\java.exe -Dwebdriver.chrome.verboseLogging=true -Dwebdriver.chrome.driver="C:\Users\john\AppData\Local\binman\binman_chromedriver\win32\2.27/chromedriver.exe" -Dwebdriver.gecko.driver="C:\Users\john\AppData\Local\binman\binman_geckodriver\win64\0.14.0/geckodriver.exe" -Dphantomjs.binary.path="C:\Users\john\AppData\Local\binman\binman_phantomjs\windows\2.1.1/phantomjs-2.1.1-windows/bin/phantomjs.exe" -jar "C:\Users\john\AppData\Local\binman\binman_seleniumserver\generic\3.0.1/selenium-server-standalone-3.0.1.jar" -port 4567
12:15:29.206 INFO - Selenium build info: version: '3.0.1', revision: '1969d75'
12:15:29.206 INFO - Launching a standalone Selenium Server
2017-02-08 12:15:29.223:INFO::main: Logging initialized #146ms
12:15:29.265 INFO - Driver class not found: com.opera.core.systems.OperaDriver
12:15:29.265 INFO - Driver provider com.opera.core.systems.OperaDriver registration is skipped:
Unable to create new instances on this machine.
12:15:29.265 INFO - Driver class not found: com.opera.core.systems.OperaDriver
12:15:29.266 INFO - Driver provider com.opera.core.systems.OperaDriver is not registered
12:15:29.271 INFO - Driver provider org.openqa.selenium.safari.SafariDriver registration is skipped:
registration capabilities Capabilities [{browserName=safari, version=, platform=MAC}] does not match the current platform WIN10
2017-02-08 12:15:29.302:INFO:osjs.Server:main: jetty-9.2.15.v20160210
2017-02-08 12:15:29.317:INFO:osjsh.ContextHandler:main: Started o.s.j.s.ServletContextHandler#c4c815{/,null,AVAILABLE}
2017-02-08 12:15:29.332:INFO:osjs.ServerConnector:main: Started ServerConnector#4af044{HTTP/1.1}{0.0.0.0:4567}
2017-02-08 12:15:29.333:INFO:osjs.Server:main: Started #257ms
12:15:29.334 INFO - Selenium Server is up and running
Now try and run a browser
remDr <- remoteDriver(port = 4567L, browserName = "chrome")
remDr$open()
If you are unable to manually run a Selenium Server then you will need to address your issues (including relevant log files) to the Selenium project or the appropriate driver project (chromedriver/geckodriver/ghostdirver etc.)
Download Docker at https://www.docker.com/products/docker-desktop
Run docker pull selenium/standalone-chrome-debug in terminal (or cmd for windows)
In Docker Desktop's Dashboard, go to the "images" tab on the left. After that, you should see something like this:
Click Run
A popup will appear. There, click on "Optional Settings"
Type 4445 on Ports. Click on the "plus" sign, type 5901 on the other input that will be created on Ports. It should look like the image below. After that, click Run.
Now, if you click on the Containers / Apps tab on the left, there should be something like this:
In Rs console, go:
install.packages("RSelenium")
library(RSelenium)
remDr <- remoteDriver(
remoteServerAdd = "localhost",
port = 4445L,
browser = "chrome"
)
remDr$open()
Every time you want RSelenium to work you will need to run the Docker container as you did in steps 3 and 5 above.
The steps also allow you to use VNC to watch what happens and debug. If you need to learn a bit about it go to https://www.realvnc.com/pt/connect/download/viewer/ More details are out of the scope of this topic.
Well, I think this can take you to a point where you can now follow these instructions of RSelenium's basic usage vignette: https://cran.r-project.org/web/packages/RSelenium/vignettes/basics.html
You should also read about security related to exposed ports and how to handle it.
These videos from R Consortium may help you out from here on:
https://www.youtube.com/watch?v=OxbvFiYxEzI and https://www.youtube.com/watch?v=JcIeWiljQG4
I hope it may help you as you would have helped me some time ago.

RSelenium, Can't start server

I'm trying to use RSelenium for web-scraping purposes behind a login and I can't get the server to run.
Current result:
library(RSelenium)
startServer()
remDr <- remoteDriver(port = 4444,
browserName = "firefox")
remDr$open()
# [1] "Connecting to remote server"
Error: Summary: UnknownError
Detail: An unknown server-side error occurred while processing the command.
class: org.openqa.selenium.firefox.NotConnectedException
I've tried running the server myself by downloading and trying to open it (nothing happens).
This was a tough one and stopped me for a couple of days when I could search on it. In the end I uninstalled Firefox and installed version 37.0 while also disabling the update service. That fixed it for me and RSelenium works fine again.
Run the following code first then it should work:
RSelenium::checkForServer()
This line of code installs the selenium server which you need for running RSelenium commands.
Try below.
rD <- rsDriver(port=4444L,browser="firefox")
mybrowser <- remoteDriver(browserName = "firefox")
mybrowser$open()
RSelenium has problems to establish serwer at the begginig on respective port. Subsequently we are telling which driver should be used.

Resources