I'm trying to use RSelenium for web-scraping purposes behind a login and I can't get the server to run.
Current result:
library(RSelenium)
startServer()
remDr <- remoteDriver(port = 4444,
browserName = "firefox")
remDr$open()
# [1] "Connecting to remote server"
Error: Summary: UnknownError
Detail: An unknown server-side error occurred while processing the command.
class: org.openqa.selenium.firefox.NotConnectedException
I've tried running the server myself by downloading and trying to open it (nothing happens).
This was a tough one and stopped me for a couple of days when I could search on it. In the end I uninstalled Firefox and installed version 37.0 while also disabling the update service. That fixed it for me and RSelenium works fine again.
Run the following code first then it should work:
RSelenium::checkForServer()
This line of code installs the selenium server which you need for running RSelenium commands.
Try below.
rD <- rsDriver(port=4444L,browser="firefox")
mybrowser <- remoteDriver(browserName = "firefox")
mybrowser$open()
RSelenium has problems to establish serwer at the begginig on respective port. Subsequently we are telling which driver should be used.
Related
I'm trying to scrape a website with a load more button. I've set up a selenium server by using a windows prompt. The Selenium server is running, but I keep getting the following error when I run the script in R. I've red a lot of blog posts and tried to find the answer, but I lack the technical knowledge to figure this out, so I hope someone is willing to help me.
Error
[1] "Connecting to remote server"
Selenium message:The path to the driver executable must be set by the
webdriver.gecko.driver system property; for more information, see
https://github.com/mozilla/geckodriver. The latest version can be
downloaded from https://github.com/mozilla/geckodriver/releases
Error: Summary: UnknownError Detail: An unknown server-side error
occurred while processing the command. class:
java.lang.IllegalStateException Further Details: run errorDetails
method
Windows prompt
cd c:\selenium
java -Dwebdriver.chrome.driver=c:\geckodriver\chromedriver.exe -
Dwebdriver.gecko.driver.driver=c:\geckodriver\geckodriver.exe -jar selenium-
server-standalone-3.4.0.jar
R SCRIPT
library(rvest)
library(RSelenium)
library(stringr)
library(xm12)
library(wdman)
url <- "https://www.social-enterprise.nl/wie-doen-het/"
remDr <- remoteDriver()
# Open the browser webpage
remDr$open()
#navigate to your page
remDr$navigate(url)
# Locate the load more button
loadmorebutton <- remDr$findElement(using = 'css selector', "#morenews")
for (i in 1:2){
print(i)
loadmorebutton$clickElement()
Sys.sleep(30)
}
page_source<-remDr$getPageSource()
Merken <- read_html(page_source[[1]]) %>%
html_nodes("#membershipCntr span") %>%
html_text()
remDr$close()
You are missing some options in remote web driver instantiate. You can try the following code,
remDr <- remoteDriver(remoteServerAddr = "localhost"
, port = 4444L
, browserName = "firefox"
)
My OS is windows 8.1 and I have the version 3.3.3 of R.
I have installed the RSelenium packages and I try to run it using this:
library("RSelenium")
#start RSelenium server
startServer()
checkForServer()
and I receive this error:
Error: checkForServer is now defunct. Users in future can find the function in
file.path(find.package("RSelenium"), "examples/serverUtils"). The
recommended way to run a selenium server is via Docker. Alternatively
see the RSelenium::rsDriver function.
Is there anything changed in the way RSelenium opens? I search for the error and I found only this but it doesn't help me. What can I do?
Also an alternative I tried is to download the chromedrive from here 'https://sites.google.com/a/chromium.org/chromedriver/downloads'
and using this script:
require(RSelenium)
cprof <- getChromeProfile("C:/Users/Peri/Desktop/chromedriver/chromedriver.exe", "Profile 1")
require(RSelenium)
remDr <- remoteDriver(remoteServerAddr = "localhost"
, port = 4444
, browserName = "chrome", extraCapabilities = cprof
)
remDr$open()
and I receive this error:
Error in checkError(res) :
Couldnt connect to host on http://localhost:4444/wd/hub.
Please ensure a Selenium server is running.
what can I do to run chrome instead of the pre-default browser Firefox?
You need to use the function rsDriver. The Selenium Version wants you to use Docker (which I also would recommend), but if you are not familiar with this you can go this way.
rsdriver will manage the binaries needed for running a Selenium Server. This provides a wrapper around the wdman::selenium function.
Here is what you have to do to start a Chrome Browser:
driver<- rsDriver()
remDr <- driver[["client"]]
And then you can work with it:
remDr$navigate("http://www.google.de")
remDr$navigate("http://www.spiegel.de")
And stop it:
remDr$close()
"everything was better back then"...
since firefox 49 (?) you can't use the rselenium package not straightforward anymore. I have searched the whole internet to find a SIMPLE How To Manual for setting up rselenium but did not find anything relevant and uptodate.
Can someone provide me and all the others out there who have no clue a simple How To manual? Like:
download XY
open AB
so I can run code like the following
require(RSelenium)
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4444L,
browserName = "firefox")
remDr$open()
Download latest version of RSelenium >= 1.7.1. Run the following:
library(RSelenium)
rD <- rsDriver() # runs a chrome browser, wait for necessary files to download
remDr <- rD$client
# no need for remDr$open() browser should already be open
If you want a firefox browser use rsDriver(browser = "firefox").
This is detailed in http://rpubs.com/johndharrison/RSelenium-Basics appendix. The recommended way to run RSelenium is via Docker containers however. Instructions for use of Docker with RSelenium can be found at http://rpubs.com/johndharrison/RSelenium-Docker
ISSUES:
If you have issues which may occur due to admin rights or other variables such as anti-virus software you can run a Selenium server manually. The easiest way to do this is via the wdman package:
selCommand<-
wdman::selenium(jvmargs = c("-Dwebdriver.chrome.verboseLogging=true"),
retcommand = TRUE)
> cat(selCommand)
C:\PROGRA~3\Oracle\Java\javapath\java.exe -Dwebdriver.chrome.verboseLogging=true -Dwebdriver.chrome.driver="C:\Users\john\AppData\Local\binman\binman_chromedriver\win32\2.27/chromedriver.exe" -Dwebdriver.gecko.driver="C:\Users\john\AppData\Local\binman\binman_geckodriver\win64\0.14.0/geckodriver.exe" -Dphantomjs.binary.path="C:\Users\john\AppData\Local\binman\binman_phantomjs\windows\2.1.1/phantomjs-2.1.1-windows/bin/phantomjs.exe" -jar "C:\Users\john\AppData\Local\binman\binman_seleniumserver\generic\3.0.1/selenium-server-standalone-3.0.1.jar" -port 4567
Using one of the wdman functions with the retcommand option enabled will return the
commandline call that would have been ran.
Now you can run the output of cat(selCommand) in a terminal
C:\Users\john>C:\PROGRA~3\Oracle\Java\javapath\java.exe -Dwebdriver.chrome.verboseLogging=true -Dwebdriver.chrome.driver="C:\Users\john\AppData\Local\binman\binman_chromedriver\win32\2.27/chromedriver.exe" -Dwebdriver.gecko.driver="C:\Users\john\AppData\Local\binman\binman_geckodriver\win64\0.14.0/geckodriver.exe" -Dphantomjs.binary.path="C:\Users\john\AppData\Local\binman\binman_phantomjs\windows\2.1.1/phantomjs-2.1.1-windows/bin/phantomjs.exe" -jar "C:\Users\john\AppData\Local\binman\binman_seleniumserver\generic\3.0.1/selenium-server-standalone-3.0.1.jar" -port 4567
12:15:29.206 INFO - Selenium build info: version: '3.0.1', revision: '1969d75'
12:15:29.206 INFO - Launching a standalone Selenium Server
2017-02-08 12:15:29.223:INFO::main: Logging initialized #146ms
12:15:29.265 INFO - Driver class not found: com.opera.core.systems.OperaDriver
12:15:29.265 INFO - Driver provider com.opera.core.systems.OperaDriver registration is skipped:
Unable to create new instances on this machine.
12:15:29.265 INFO - Driver class not found: com.opera.core.systems.OperaDriver
12:15:29.266 INFO - Driver provider com.opera.core.systems.OperaDriver is not registered
12:15:29.271 INFO - Driver provider org.openqa.selenium.safari.SafariDriver registration is skipped:
registration capabilities Capabilities [{browserName=safari, version=, platform=MAC}] does not match the current platform WIN10
2017-02-08 12:15:29.302:INFO:osjs.Server:main: jetty-9.2.15.v20160210
2017-02-08 12:15:29.317:INFO:osjsh.ContextHandler:main: Started o.s.j.s.ServletContextHandler#c4c815{/,null,AVAILABLE}
2017-02-08 12:15:29.332:INFO:osjs.ServerConnector:main: Started ServerConnector#4af044{HTTP/1.1}{0.0.0.0:4567}
2017-02-08 12:15:29.333:INFO:osjs.Server:main: Started #257ms
12:15:29.334 INFO - Selenium Server is up and running
Now try and run a browser
remDr <- remoteDriver(port = 4567L, browserName = "chrome")
remDr$open()
If you are unable to manually run a Selenium Server then you will need to address your issues (including relevant log files) to the Selenium project or the appropriate driver project (chromedriver/geckodriver/ghostdirver etc.)
Download Docker at https://www.docker.com/products/docker-desktop
Run docker pull selenium/standalone-chrome-debug in terminal (or cmd for windows)
In Docker Desktop's Dashboard, go to the "images" tab on the left. After that, you should see something like this:
Click Run
A popup will appear. There, click on "Optional Settings"
Type 4445 on Ports. Click on the "plus" sign, type 5901 on the other input that will be created on Ports. It should look like the image below. After that, click Run.
Now, if you click on the Containers / Apps tab on the left, there should be something like this:
In Rs console, go:
install.packages("RSelenium")
library(RSelenium)
remDr <- remoteDriver(
remoteServerAdd = "localhost",
port = 4445L,
browser = "chrome"
)
remDr$open()
Every time you want RSelenium to work you will need to run the Docker container as you did in steps 3 and 5 above.
The steps also allow you to use VNC to watch what happens and debug. If you need to learn a bit about it go to https://www.realvnc.com/pt/connect/download/viewer/ More details are out of the scope of this topic.
Well, I think this can take you to a point where you can now follow these instructions of RSelenium's basic usage vignette: https://cran.r-project.org/web/packages/RSelenium/vignettes/basics.html
You should also read about security related to exposed ports and how to handle it.
These videos from R Consortium may help you out from here on:
https://www.youtube.com/watch?v=OxbvFiYxEzI and https://www.youtube.com/watch?v=JcIeWiljQG4
I hope it may help you as you would have helped me some time ago.
I'm trying to install Rselenium and I get this
Connecting to remote server"
Error: Summary: UnknownError
Detail: An unknown server-side error occurred while processing the command.
class: java.net.ConnectException"
Code which I have tried is
install.packages("RSelenium")
library(RSelenium)
startServer()
checkForServer()
mybrowser <- remoteDriver(browserName = "chrome")
mybrowser$open()
mybrowser$navigate("http://www.weather.gov")
Sometime I face a connection problem. Here what I make:
library("RSelenium")
startServer()
checkForServer()
remDr <- remoteDriver()
remDr$open()
remDr$navigate("http://www.weather.gov")
Sys.sleep(5)
I leave remoteDriver() which is the Firefox. However sometimes when I run the code I receive a connection problem. As a workaround I open the firefox on my own and after that I run the code and the code runs successfully.
Note: I would have made a comment, but I don't have 50 rep yet
I've had problems with this in the past (see here). It has to do with opening the server, if you click manually on the server file (something like selenium-server-standalone-2.48.2.jar), then run your code again, it should work.
I try to install and run a simple example for R Selenium package using this:
install.packages("RSelenium")
library("RSelenium")
startServer()
checkForServer()
startServer()
remDr <- remoteDriver(browserName = "Chrome")
remDr$open()
In the last code I receive this:
[1] "Connecting to remote server"
Error: Summary: UnknownError
Detail: An unknown server-side error occurred while processing the command.
class: org.openqa.selenium.WebDriverException
I tried some workarounds from google but nothing worked. What can I do?
From comments:
Click start
Select Control Panel > System
Select Advance system settings
Click Environment Variables...
Under System Variables
Scroll to Path and double click
At the end of Variable value: add ;C:\path\to\directory that holds the chromedriver.exe file. Note the ; that separates the paths
Restart your R session and you should now be able to run:
> require(RSelenium)
RSelenium::startServer()
remDr <- remoteDriver(browserName = "chrome")
remDr$open()
EDIT
For RSelenium to operate with chrome you first need to download chromedriver.exe you can download this from https://sites.google.com/a/chromium.org/chromedriver/downloads. Once downloaded unzip the folder and place chromedriver.exe where you would like to store it.
The directory that you store chromedriver.exe and add to your system PATH can be anywhere you choose. As stated in comments, for example, mine currently resides in C:\Python27\Scripts.