I am trying to scrape a web page with a JavaScript drop down menu in R. I can follow the directions listed here, but nothing happens and no errors are shown. Instead, it gives an empty list:
dropdown <- remDr$findElement(using = "id", "s2id_autogen4_search")
remDr$executeScript("arguments[0].setAttribute('class','select2-input select2-focused');", list(dropdown))
> list()
Also, nothing happens (and no console output) with dropdown$clickElement().
This is somewhat related to this post, but I need to click first to activate the drop down.
There was a mask over it, so I needed to find the mask, click on that, then supply arguments to the dropdown itself:
dropdown <- remDr$findElement(using = "id", "s2id_autogen4_search")
mask <- remDr$findElement(using = "xpath", "//*[#id='select2-drop-mask']")
mask$clickElement()
dropdown$sendKeysToElement(list("l"))
Related
Would appreciate some help as I am stuck here.
I am trying to write an automated script to download data from the Microsoft Power BI site of the WHO, which can be found here.
But when I try to retrieve the data, the right click function doesn't seem to work - or more likely: I am doing something wrong.
I created a container on Docker that I am accessing with Selenium in R. The script below generates a click on the first page (on the "Download data" button at the lower left-hand side corner of the screen. After a long load time the next screen appears. The goal is to RIGHT-CLICK on the download button of "Vaccine uptake by target group" x "Data".
Here's two screenshots of where I need to create the first left-click and the second right-click.
I have tried multiple approaches, including first selecting the iframe, switching the frame view and then selecting an xpath pointing to the clickable area. That seemed to work.
But when I give the command to right click nothing happens as verified in the VNC rendition. The contextual menu doesn't appear.
Anyone knows what went wrong?
Here's the code I entered:
library(RSelenium)
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "firefox")
remDr$open()
remDr$navigate("https://app.powerbi.com/view?r=eyJrIjoiMWNjNzZkNjctZTNiNy00YmMzLTkxZjQtNmJiZDM2MTYxNzEwIiwidCI6ImY2MTBjMGI3LWJkMjQtNGIzOS04MTBiLTNkYzI4MGFmYjU5MCIsImMiOjh9")
Sys.sleep(30) # WHO is taking its time
#This is first button to bring us to the next page
webElem <- remDr$findElement(using = "xpath", value = "/html/body/div[1]/report-embed/div/div/div[1]/div/div/div/exploration-container/div/docking-container/div/div/div/div/exploration-host/div/div/exploration/div/explore-canvas/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container[22]/transform/div/div[3]/div/visual-modern")
webElem$highlightElement()
webElem$clickElement()
Sys.sleep(30) # again need some time to fully load
# This selects the iframe
webElem <- remDr$findElement(using = "xpath", value = "/html/body/div[1]/report-embed/div/div/div[1]/div/div/div/exploration-container/div/docking-container/div/div/div/div/exploration-host/div/div/exploration/div/explore-canvas/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container[19]/transform/div/div[3]/div/visual-modern/div/iframe")
remDr$switchToFrame(webElem)
# This selects the area for the second click
webElem <- remDr$findElement(using = "xpath", value="/html/body/div/div/a[1]")
remDr$mouseMoveToLocation(webElement = webElem)
# And then the right-click but none of these seem to work:
remDr$click(buttonId = 2)
remDr$click('right')
Thanks for any advice.
I am trying to use RSelenium to navigate this page: https://championsleague.len.eu/calendar/
For some reason, I can't seem to be able to find any element on the page. Firstly, the selector gadget does not work on it.
In addition, when I use the developer tools to grab the class or xpath of an object that I want to click (for example, let's say I want on the DAY 10 button of the calendar), the findElements function always returns an empty list.
remDr$navigate("https://championsleague.len.eu/calendar")
#using CSS selector
remDr$findElements("css selector", '#tblBottoniDay > div')
#using xpath
remDr$findElements("xpath", '//*[#id="tblBottoniDay"]/div')
Does anyone have an idea of what I can do to solve this problem?
Thank you very much.
You are missing a delay here.
Before accessing element on the page you need to wait for these elements to be completely loaded.
The simplest way is to add a delay there, like this:
remDr$navigate("https://championsleague.len.eu/calendar")
Sys.sleep(5)
remDr$findElements("css selector", '#tblBottoniDay > div')
The more preferred way is to use Expected Conditions to wait for elements visibility
The item(DAY) you want to click is in iframe first we shall swtich to iframe and use fullxpath to click the item.
#launch browser
library(RSelenium)
driver <- rsDriver(browser = "chrome")
remDr<-driver[["client"]]
url = "https://championsleague.len.eu/calendar"
#navigate
remDr$navigate(url)
#accept cookie
remDr$findElement(using = "xpath",'//*[#id="cmplz-cookiebanner-container"]/div/div[6]/button[1]')$clickElement()
#swtich to iframe
webElem <- remDr$findElements(using = "xpath", value = ' //*[#id="advanced_iframe"]')
remDr$switchToFrame(webElem[[1]])
#now we shall click on DAY12
remDr$findElement(using = "xpath",'/html/body/div[1]/div[1]/div/div/div/div[2]/div/div[2]/div[12]/div/span')$clickElement()
#note that last number in xpath represents the day
The element with text as DAY 10 is within an <frame> so you have to switchToFrame and you can use the following Locator Strategies:
Using xpath:
webElem <- remDr$findElement(using = "id", value = "advanced_iframe")
remDr$switchToFrame(webElem$elementId)
remDr$findElement("xpath", "//span[text()='DAY 10']")
References
Package ‘RSelenium’
I'm trying to write a code for web scraping in R when you have to introduce inputs.
Exactly, I have a platform where I need to complete 2 fields and after that click submit and get the results.
But I don't know how to use my columns in R like inputs in platform.
I searched for an example but I did't find any.
Pls, if anyone can give me a simple e.g.
Thank you
EDIT:
I don't have a code yet. I was looking for an example where you can use input for complete a field on a site and after that to scrape the result.
In the photo are the fields on my URL. So, in R I have a dataframe with 2 columns. One for CNP/CUI and one for VIN/SASIU with 100 rows or more. And I want to use this columns like input and take the output for every row.
EDIT2:
The example provided by #Dominik S.Meier it worked for me when I had a list for inputs. For column inputs I will post another question.
But, till then I want to mention few thing that helped me, maybe it will hep somebody else.
You need to be sure that all the versions matches: R version, browser version, browser driver version, Java version. For me it didn't match chromedriver version, even if I downloaded the right version. The problem was that I had 3 chromeversion and I think it didn't choose the right. I fixed with: rD <- rsDriver(browser = c("chrome"),port = 4444L,chromever = "83.0.4103.39"). More info here:enter link description here
Because one element didn't have id like in e.g. webElem <- remDr$findElement(using = "id", "trimite"), I used css selector. You can find the css selector with right click -> copy -> copy selector (in the html code on the page).
If you don't get the results, maybe you don't use the right selector. I did that and the result was list(). Then I tried more css selector from the "above" in the html code. I don'y know if it is the right solution, but for me it worked.
Hope it will help. Thank you.
Using RSelenium (see here for more infos):
library(RSelenium)
rD <- rsDriver(browser = c("firefox")) #specify browser type you want Selenium to open
remDr <- rD$client
remDr$navigate("https://pro.rarom.ro/istoric_vehicul/dosar_vehicul.aspx") # navigates to webpage
# select first input field
option <- remDr$findElement(using='id', value="inputEmail")
option$highlightElement()
option$clickElement()
option$sendKeysToElement(list("email#email.com"))
# select second input field
option <- remDr$findElement(using='id', value="inputEmail2")
option$highlightElement()
option$clickElement()
option$sendKeysToElement(list("email#email.com"))
# select second input field
option <- remDr$findElement(using='id', value="inputVIN")
option$highlightElement()
option$clickElement()
option$sendKeysToElement(list("123"))
#press key
webElem <- remDr$findElement(using = "id", "trimite")
webElem$highlightElement()
webElem$clickElement()
I would like to use RSelenium to download files (by clicking on the excel image) from this website http://highereducationstatistics.education.gov.au/. However, before downloading the file, a series of drag-and-drop actions (see this image http://highereducationstatistics.education.gov.au/images/preDragDimension.png) have to be performed so the right dataset could be chosen (See this http://highereducationstatistics.education.gov.au/GettingStarted.aspx for instruction).
I am wondering whether RSelenium has this type of drag and drop functions. I have searched this whole day and guess that mouseMoveToLocation combined with other functions like buttondown function might be the answers, but have no idea how to use them. Can anyone help with this?
Thanks very much.
First navigate with RSelenium to the page using:
library(RSelenium)
rD <- rsDriver() # runs a chrome browser, wait for necessary files to download
remDr <- rD$client
remDr$navigate("http://highereducationstatistics.education.gov.au/")
Then locate the element you want to drag to the chart/grid. In this example I will be selecting the Course Level from the left menu.
webElem1 <- remDr$findElement('xpath', "//span[#title = 'Course Level']")
Select the element where you want to drop this menu item. In this case the element has an id = "olapClientGridXrowHeader":
webElem2 <- remDr$findElement(using = 'id', "olapClientGridXrowHeader")
Once both items are selected, drag the first one into the second one like this:
remDr$mouseMoveToLocation(webElement = webElem1)
remDr$buttondown()
remDr$mouseMoveToLocation(webElement = webElem2)
remDr$buttonup()
Notice that these methods work on the remote driver, not the elements.
Like the beginning to any problem before I post it on stack overflow I think I have tried everything. This is a learning experience for me on how to work with javascript and xml so I'm guessing my problem is there.
My question is how to get the results of clicking on the parcel number links that are javascript links? I've tried getting the xpath of the link and using the $click method which following my intuition but this wasn't right or is at least not working for me.
Firefox 26.0
R 3.0.2
require(relenium)
library(XML)
library(stringr)
initializing_parcel_number <- "00000000000"
firefox <- firefoxClass$new()
firefox$get("http://www.muni.org/pw/public.html")
inputElement <- firefox$findElementByXPath("/html/body/form[2]/table/tbody/tr[2]/td/table[1]/tbody/tr[3]/td[4]/input[1]")
inputElement$sendKeys(initializing_parcel_number)
inputElement$sendKeys(key = "ENTER")
##xpath to the first link. Or is it?
first_link <- "/html/body/table/tbody/tr[2]/td/table[5]/tbody/tr[2]/td[1]/a"
##How I'm trying to click the thing.
linkElement <- firefox$findElementByXPath("/html/body/table/tbody/tr[2]/td/table[5]/tbody/tr[2]/td[1]/a")
linkElement$click()
You can do this using RSelenium. See http://johndharrison.github.io/RSelenium/ . DISCLAIMER I am the author of the RSelenium package. A basic vignette on operation can be viewed at RSelenium basics and
RSelenium: Testing Shiny apps
If you are unsure of what element is selected you can use the highlightElement utility method in the webElement class see the commented out code.
The element click event wont work in this case. You need to simulate a click using javascript:
require(RSelenium)
# RSelenium::startServer # if needed
initializing_parcel_number <- "00000000000"
remDr <- remoteDriver()
remDr$open()
remDr$navigate("http://www.muni.org/pw/public.html")
webElem <- remDr$findElement(using = "name", "PAR1")
# webElem$highlightElement() # to visually check what elemnet is selected
webElem$sendKeysToElement(list(initializing_parcel_number, key = "enter"))
# get first link containing javascript:getParcel
webElem <- remDr$findElement(using = "css selector", '[href*="javascript:getParcel"]')
# webElem$highlightElement() # to visually check what elemnet is selected
# send a webElement as an argument.
remDr$executeScript("arguments[0].click();", list(webElem))
#