I am trying to scrape a page, getting the move list of a game of chess, which is located in the menu on the right, under the "moves" tab.
library(RSelenium)
url <- "https://play.xiangqi.com/game/oX00ly"
rD <- RSelenium::rsDriver(browser = "firefox", check = F)
remDr <- rD$client
remDr$navigate(url = url)
when manually clicking the Moves tab in the browser, I can get the desired text via
webElem <- remDr$findElement("css selector", ".Wrapper__MovesTabWrapper-sc-13rqht3-2")
webElem$getElementText()[[1]]
which (correctly) returns
[1] "1\np3+1\nP3+1\n2\ne3+5\nH2+3\n3\nh8+7\nH8+7\n4\nh2+3\nR1+1\n5\nc8=9\nH3+2\n6\nc2+1\nE7+5\n7\nh3+4\nA6+5\n8\nh4+3\nR9=6\n9\nr1=3\nR6+6\n10\nc2+2\nH2+3\n11\nr9=8\nC2=3\n12\nr8+3\nR1=4\n13\nc2-1\nR6=8\n14\nr8+4\nH3+1\n15\ne7+9\nC3+5\n16\ne9-7\nR4+3\n17\nc2=1\nR8=9\n18\nh3-4\nR4=6\n19\nc1=2\nR9-1\n20\nr3=2\nC8+7\n21\ne5-3\nR9=8\n22\nh4-3\nR8+2\n23\nh3-2\nR8+2\n24\ne7+5\nH7+8\n25\nr8-5\nC3+1\n26\nr8+2\nH8+7\n27\np9+1\nH7+5\n28\na6+5\nH5+7\n29\nk5=6\nR6=4\n30\na5+6\nR4+3"
Problem
When trying to click the button through RSelenium, by using
webElem <- remDr$findElement("css selector", "#moves-tab")
webElem <-webElem$clickElement() # or webElem$click()
Nothing seems to happen, and I'm at a loss on how to proceed troubleshooting.
Question
How can I switch to the Moves tab by simulating a click (active event listener)?
Bonus pts: is this possible using the rvest package?
Sometimes being too trigger happy is a problem.
Adding
webElem <- webElem$clickElement()
Sys.sleep(2)
solved the problem.
Related
Would appreciate some help as I am stuck here.
I am trying to write an automated script to download data from the Microsoft Power BI site of the WHO, which can be found here.
But when I try to retrieve the data, the right click function doesn't seem to work - or more likely: I am doing something wrong.
I created a container on Docker that I am accessing with Selenium in R. The script below generates a click on the first page (on the "Download data" button at the lower left-hand side corner of the screen. After a long load time the next screen appears. The goal is to RIGHT-CLICK on the download button of "Vaccine uptake by target group" x "Data".
Here's two screenshots of where I need to create the first left-click and the second right-click.
I have tried multiple approaches, including first selecting the iframe, switching the frame view and then selecting an xpath pointing to the clickable area. That seemed to work.
But when I give the command to right click nothing happens as verified in the VNC rendition. The contextual menu doesn't appear.
Anyone knows what went wrong?
Here's the code I entered:
library(RSelenium)
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "firefox")
remDr$open()
remDr$navigate("https://app.powerbi.com/view?r=eyJrIjoiMWNjNzZkNjctZTNiNy00YmMzLTkxZjQtNmJiZDM2MTYxNzEwIiwidCI6ImY2MTBjMGI3LWJkMjQtNGIzOS04MTBiLTNkYzI4MGFmYjU5MCIsImMiOjh9")
Sys.sleep(30) # WHO is taking its time
#This is first button to bring us to the next page
webElem <- remDr$findElement(using = "xpath", value = "/html/body/div[1]/report-embed/div/div/div[1]/div/div/div/exploration-container/div/docking-container/div/div/div/div/exploration-host/div/div/exploration/div/explore-canvas/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container[22]/transform/div/div[3]/div/visual-modern")
webElem$highlightElement()
webElem$clickElement()
Sys.sleep(30) # again need some time to fully load
# This selects the iframe
webElem <- remDr$findElement(using = "xpath", value = "/html/body/div[1]/report-embed/div/div/div[1]/div/div/div/exploration-container/div/docking-container/div/div/div/div/exploration-host/div/div/exploration/div/explore-canvas/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container[19]/transform/div/div[3]/div/visual-modern/div/iframe")
remDr$switchToFrame(webElem)
# This selects the area for the second click
webElem <- remDr$findElement(using = "xpath", value="/html/body/div/div/a[1]")
remDr$mouseMoveToLocation(webElement = webElem)
# And then the right-click but none of these seem to work:
remDr$click(buttonId = 2)
remDr$click('right')
Thanks for any advice.
I'm trying to scrape reviews from this Google Play site using R (manily "RSelenium" and "rvest":
https://play.google.com/store/apps/details?id=hr.mireo.arthur&hl=en&fbclid=IwAR3c-PkUXOea8KrKLp9Q3JUjCidGmgO2jYX_Qb7O8VuWlHXPIS5nDOfKRKI&showAllReviews=true
I've managed to load page using RSelenium and make a loop which scrolls down the page and clicks on all "Show more" buttons. Here is the code I've used:
#Load packages
library(rvest)
library(dplyr)
library(wdman)
library(RSelenium)
#Open website using RSelenium
url <- 'https://play.google.com/store/apps/details?id=hr.mireo.arthur&hl=en&fbclid=IwAR3c-PkUXOea8KrKLp9Q3JUjCidGmgO2jYX_Qb7O8VuWlHXPIS5nDOfKRKI&showAllReviews=true'
rD <- rsDriver(port = 4567L, browser=c("chrome"), chromever="80.0.3987.106")
remDr <- rD[["client"]]
remDr$open()
remDr$navigate(url)
#Load whole page by scrolling and showing more
xp_show_more <- "//*[#id='fcxH9b']/div[4]/c-wiz/div/div[2]/div/div[1]/div/div/div[1]/div[2]/div[2]/div"
replicate(5,
{
replicate(5,
{
# scroll down
webElem <- remDr$findElement("css", "body")
webElem$sendKeysToElement(list(key = "end"))
# wait
Sys.sleep(1)
})
# find button
morereviews <- remDr$findElement(using = 'xpath', xp_show_more)
# click button
tryCatch(morereviews$clickElement(),error=function(e){print(e)}) # trycatch to prevent any error from stopping the loop
# wait
Sys.sleep(3)
})
This results with loading all 573 comments, but several comments have "Full review" buttons which have to be clicked in order to see the rest of it. I'm trying to make a script which clicks on all "Full review" buttons (I beleave there are just over 30 of them), but I can't manage to do so. My current script clicks on the first "Full review" buttons
#Click on "Full Review"
xp_full_review <- '//*[#id="fcxH9b"]/div[4]/c-wiz/div/div[2]/div/div[1]/div/div/div[1]/div[2]/div/div[2]/div/div[2]/div[2]/span[1]/div/button'
replicate(35,
{
fullreviews <- remDr$findElement(using = 'xpath', xp_full_review)
fullreviews$clickElement()
Sys.sleep(0.5)
})
Can someone spot a mistake and find a way to click on all "Full review" buttons?
Thank you
I am trying to web scrape the data from the Flipkart site. The link for the webpage is as follows:
https://www.flipkart.com/mi-a1-black-64-gb/product-reviews/itmexnsrtzhbbneg?aid=overall&pid=MOBEX9WXUSZVYHET
I need to automate navigation to the NEXT page by clicking on NEXT button the webpage. Below is the code I'm using
nextButton <-remDr$findElement(value ='//div[#class="_2kUstJ"]')$clickElement()
Error
Selenium message:Element is not clickable at point
I even tried scrolling the webpage as suggested by many stackoverflow questions using the below code
remDr$executeScript("arguments[0].scrollIntoView(true);", nextButton)
But this code is also giving error as
Error in checkError(res) : Undefined error in httr call. httr output: No method for S4 class:webElement
Kindly suggest the solution. I'm using firefox browser and selenium to automate using R programming.
If you do not mind using Chrome driver, the following code worked:
eCaps <- list(chromeOptions = list(
args = c('--headless', '--disable-gpu', '--window-size=1880,1000', "--no-sandbox", "--disable-dev-shm-usage")
))
remDr <- rsDriver(port = 4565L,browser = "chrome",extraCapabilities = eCaps)
remCl <- remDr[["client"]]
remCl$navigate("https://www.flipkart.com/mi-a1-black-64-gb/product-reviews/itmexnsrtzhbbneg?aid=overall&pid=MOBEX9WXUSZVYHET")
remCl$findElement(using = "css selector", "._3fVaIS > span:nth-child(1)")$clickElement()
We shall first scroll to the end of the page and then click Next.
#Navigate to webpage
remDr$navigate("https://www.flipkart.com/mi-a1-black-64-gb/product-reviews/itmexnsrtzhbbneg?aid=overall&pid=MOBEX9WXUSZVYHET")
#Scroll to the end
webElem <- remDr$findElement("css", "html")
webElem$sendKeysToElement(list(key="end"))
#click on Next
remDr$findElement(using = "xpath", '//*[#id="container"]/div/div[3]/div/div/div[2]/div[13]/div/div/nav/a[11]/span')$clickElement()
I want to download a file from a website using RSelenium, with Firefox browser.
I do everything correctly (navigate, select the correct element and write what I want);
now I click the "download" button, then a firefox popup opens and ask me if I want to download the file or "open with..." something else.
Unfortunately I cannot write an example due to privacy constraints.
My question is: how can I switch to the popup window / alert and click "OK" when needed?
I tried the following methods with no success:
remDrv$acceptAlert() -> tells me: NoAlertOpenError
remDrv$executeScript("driver.switchTo().alert().accept()")
I also tried the method
remDrv$getWindowHandles()
but even if the popup is open, the command return me one window only (the beginning one, not the popup), therefore I'm not able to use the:
remDrv$switchToWindow()
to switch to the popup window.
Any ideas?
Thanks
What you are seeing is not a popup it is a download dialog. The download dialog is native in all browsers and cannot be controlled with JavaScript. You can configure Firefox to automatically download for certain file types. You havent given us alot of information.
It can be done by setting an appropriate profile. Here is an example that downloads some financial data. We set four options in a bespoke profile. We have to jump through some hoops selecting options before we get a file to download:
require(RSelenium)
fprof <- makeFirefoxProfile(list(browser.download.dir = "C:\\temp"
, browser.download.folderList = 2L
, browser.download.manager.showWhenStarting = FALSE
, browser.helperApps.neverAsk.saveToDisk = "application/zip"))
RSelenium::startServer()
remDr <- remoteDriver(extraCapabilities = fprof)
remDr$open(silent = TRUE)
remDr$navigate("https://www.chicagofed.org/applications/bhc_data/bhcdata_index.cfm")
# click year 2012
webElem <- remDr$findElement("name", "SelectedYear")
webElems <- webElem$findChildElements("css selector", "option")
webElems[[which(sapply(webElems, function(x){x$getElementText()}) == "2012" )]]$clickElement()
# click required quarter
webElem <- remDr$findElement("name", "SelectedQuarter")
Sys.sleep(1)
webElems <- webElem$findChildElements("css selector", "option")
webElems[[which(sapply(webElems, function(x){x$getElementText()}) == "4th Quarter" )]]$clickElement()
# click button
webElem <- remDr$findElement("id", "downloadDataFile")
webElem$clickElement()
Like the beginning to any problem before I post it on stack overflow I think I have tried everything. This is a learning experience for me on how to work with javascript and xml so I'm guessing my problem is there.
My question is how to get the results of clicking on the parcel number links that are javascript links? I've tried getting the xpath of the link and using the $click method which following my intuition but this wasn't right or is at least not working for me.
Firefox 26.0
R 3.0.2
require(relenium)
library(XML)
library(stringr)
initializing_parcel_number <- "00000000000"
firefox <- firefoxClass$new()
firefox$get("http://www.muni.org/pw/public.html")
inputElement <- firefox$findElementByXPath("/html/body/form[2]/table/tbody/tr[2]/td/table[1]/tbody/tr[3]/td[4]/input[1]")
inputElement$sendKeys(initializing_parcel_number)
inputElement$sendKeys(key = "ENTER")
##xpath to the first link. Or is it?
first_link <- "/html/body/table/tbody/tr[2]/td/table[5]/tbody/tr[2]/td[1]/a"
##How I'm trying to click the thing.
linkElement <- firefox$findElementByXPath("/html/body/table/tbody/tr[2]/td/table[5]/tbody/tr[2]/td[1]/a")
linkElement$click()
You can do this using RSelenium. See http://johndharrison.github.io/RSelenium/ . DISCLAIMER I am the author of the RSelenium package. A basic vignette on operation can be viewed at RSelenium basics and
RSelenium: Testing Shiny apps
If you are unsure of what element is selected you can use the highlightElement utility method in the webElement class see the commented out code.
The element click event wont work in this case. You need to simulate a click using javascript:
require(RSelenium)
# RSelenium::startServer # if needed
initializing_parcel_number <- "00000000000"
remDr <- remoteDriver()
remDr$open()
remDr$navigate("http://www.muni.org/pw/public.html")
webElem <- remDr$findElement(using = "name", "PAR1")
# webElem$highlightElement() # to visually check what elemnet is selected
webElem$sendKeysToElement(list(initializing_parcel_number, key = "enter"))
# get first link containing javascript:getParcel
webElem <- remDr$findElement(using = "css selector", '[href*="javascript:getParcel"]')
# webElem$highlightElement() # to visually check what elemnet is selected
# send a webElement as an argument.
remDr$executeScript("arguments[0].click();", list(webElem))
#