I have this website (https://www.sofascore.com/pt/torneio/futebol/brazil/brasileiro-serie-a/325) that I want to get some stats from games by round. There is 38 rounds and the base just shows the first 11. For me to get the rest of the rounds I have to scroll this inner scroll bar but I don't know how to do it.
I use the package RSelenium in R.
Here's the code (so far)...
After this, I don't know what to do...
require(RSelenium)
click <- function(xpath){
webElem <- remDr$findElement(using = "xpath", value = xpath)
webElem$clickElement()
}
driver <- rsDriver(port = 5799L, browser = c('chrome'), chromever = "88.0.4324.96")
url = 'https://www.sofascore.com/pt/torneio/futebol/brazil/brasileiro-serie-a/325'
remDr <- driver[['client']]
remDr$navigate(url) #link
Sys.sleep(1)
# games by round
click('//*[#id="__next"]/main/div/div[2]/div[1]/div[1]/div[6]/div/div[1]/a[2]')
# round options
click('//*[#id="__next"]/main/div/div[2]/div[1]/div[1]/div[6]/div/div[2]/div/div/div/div[1]/div/div[1]/div[2]/div')
Related
I want to take a screenshot of an entire webpage using RSelenium. I have this code working:
library(RSelenium)
driver <- rsDriver(browser = "firefox")
remdriv <- driver$client
remdriv$navigate("https://stackoverflow.com/questions/73115385/counting-all-elements-row-wise-in-list-column")
remdriv$screenshot(file = "post.png")
But when I run this I get a screenshot of exactly what the driver's browser is showing, like this:
What I want is the full-length screenshot of the entire webpage. What can I do to capture that within RSelenium or another R tool?
In the end I want it to look like this:
I think you have to scroll down and take multiple screenshots, then combine these multiple screenshots into a single one. I haven't managed yet to zoom out yet, which would be another option.
The scrolling doesn't work perfectly and you need to loop and finish up the script, but I hope this is useful to start with:
# Load packages
if (!require(pacman)) {install.packages("pacman")}
pacman::p_load(here, RSelenium, stringr)
# Stop currently running server
if(exists("rD")) suppressWarnings(rD$server$stop())
# Load RSelenium
rD <- rsDriver(browser = "chrome", chromever = "106.0.5249.61", port = 4567L)
remDr <- rD[["client"]]
link <- "https://stackoverflow.com/questions/73115385/counting-all-elements-row-wise-in-list-column"
remDr$open()
remDr$navigate(link)
# Get browser height and width
browser_height <- remDr$executeScript("return document.body.offsetHeight;")[[1]]
browser_width <- remDr$executeScript("return document.body.offsetWidth;")[[1]]
remDr$getWindowSize()
remDr$setWindowSize(browser_width, browser_height)
remDr$getWindowSize()
# This is what actually can be displayed
browser_final_height <- remDr$getWindowSize()$height # 1175
# Get inner window height and width
inner_window_height <- remDr$executeScript("return window.innerHeight")[[1]]
inner_window_width <- remDr$executeScript("return window.innerWidth")[[1]]
# Check how many xtimes the inner window fits in what should be the document size
num_screen <- (browser_height / inner_window_height)
# Move to top of window
remDr$executeScript("window.scrollBy(0, -5000);")
# Scroll down (loop from here to end)
remDr$executeScript(str_c("window.scrollBy(0, ", inner_window_height, ");"))
# Take screenshot
remDr$screenshot(file = here("results", "screenshots", "ex2.png"))
# Close server
remDr$close()
rD$server$stop()
Would appreciate some help as I am stuck here.
I am trying to write an automated script to download data from the Microsoft Power BI site of the WHO, which can be found here.
But when I try to retrieve the data, the right click function doesn't seem to work - or more likely: I am doing something wrong.
I created a container on Docker that I am accessing with Selenium in R. The script below generates a click on the first page (on the "Download data" button at the lower left-hand side corner of the screen. After a long load time the next screen appears. The goal is to RIGHT-CLICK on the download button of "Vaccine uptake by target group" x "Data".
Here's two screenshots of where I need to create the first left-click and the second right-click.
I have tried multiple approaches, including first selecting the iframe, switching the frame view and then selecting an xpath pointing to the clickable area. That seemed to work.
But when I give the command to right click nothing happens as verified in the VNC rendition. The contextual menu doesn't appear.
Anyone knows what went wrong?
Here's the code I entered:
library(RSelenium)
remDr <- remoteDriver(remoteServerAddr = "localhost", port = 4445L, browserName = "firefox")
remDr$open()
remDr$navigate("https://app.powerbi.com/view?r=eyJrIjoiMWNjNzZkNjctZTNiNy00YmMzLTkxZjQtNmJiZDM2MTYxNzEwIiwidCI6ImY2MTBjMGI3LWJkMjQtNGIzOS04MTBiLTNkYzI4MGFmYjU5MCIsImMiOjh9")
Sys.sleep(30) # WHO is taking its time
#This is first button to bring us to the next page
webElem <- remDr$findElement(using = "xpath", value = "/html/body/div[1]/report-embed/div/div/div[1]/div/div/div/exploration-container/div/docking-container/div/div/div/div/exploration-host/div/div/exploration/div/explore-canvas/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container[22]/transform/div/div[3]/div/visual-modern")
webElem$highlightElement()
webElem$clickElement()
Sys.sleep(30) # again need some time to fully load
# This selects the iframe
webElem <- remDr$findElement(using = "xpath", value = "/html/body/div[1]/report-embed/div/div/div[1]/div/div/div/exploration-container/div/docking-container/div/div/div/div/exploration-host/div/div/exploration/div/explore-canvas/div/div[2]/div/div[2]/div[2]/visual-container-repeat/visual-container[19]/transform/div/div[3]/div/visual-modern/div/iframe")
remDr$switchToFrame(webElem)
# This selects the area for the second click
webElem <- remDr$findElement(using = "xpath", value="/html/body/div/div/a[1]")
remDr$mouseMoveToLocation(webElement = webElem)
# And then the right-click but none of these seem to work:
remDr$click(buttonId = 2)
remDr$click('right')
Thanks for any advice.
I am trying to scrape a page, getting the move list of a game of chess, which is located in the menu on the right, under the "moves" tab.
library(RSelenium)
url <- "https://play.xiangqi.com/game/oX00ly"
rD <- RSelenium::rsDriver(browser = "firefox", check = F)
remDr <- rD$client
remDr$navigate(url = url)
when manually clicking the Moves tab in the browser, I can get the desired text via
webElem <- remDr$findElement("css selector", ".Wrapper__MovesTabWrapper-sc-13rqht3-2")
webElem$getElementText()[[1]]
which (correctly) returns
[1] "1\np3+1\nP3+1\n2\ne3+5\nH2+3\n3\nh8+7\nH8+7\n4\nh2+3\nR1+1\n5\nc8=9\nH3+2\n6\nc2+1\nE7+5\n7\nh3+4\nA6+5\n8\nh4+3\nR9=6\n9\nr1=3\nR6+6\n10\nc2+2\nH2+3\n11\nr9=8\nC2=3\n12\nr8+3\nR1=4\n13\nc2-1\nR6=8\n14\nr8+4\nH3+1\n15\ne7+9\nC3+5\n16\ne9-7\nR4+3\n17\nc2=1\nR8=9\n18\nh3-4\nR4=6\n19\nc1=2\nR9-1\n20\nr3=2\nC8+7\n21\ne5-3\nR9=8\n22\nh4-3\nR8+2\n23\nh3-2\nR8+2\n24\ne7+5\nH7+8\n25\nr8-5\nC3+1\n26\nr8+2\nH8+7\n27\np9+1\nH7+5\n28\na6+5\nH5+7\n29\nk5=6\nR6=4\n30\na5+6\nR4+3"
Problem
When trying to click the button through RSelenium, by using
webElem <- remDr$findElement("css selector", "#moves-tab")
webElem <-webElem$clickElement() # or webElem$click()
Nothing seems to happen, and I'm at a loss on how to proceed troubleshooting.
Question
How can I switch to the Moves tab by simulating a click (active event listener)?
Bonus pts: is this possible using the rvest package?
Sometimes being too trigger happy is a problem.
Adding
webElem <- webElem$clickElement()
Sys.sleep(2)
solved the problem.
I'm trying to interact with all elements of the list in R.
Precisely, I'm trying to click on all elements in list using RSelenium using clickElement() function.
I'm scraping data from this webpage:
https://play.google.com/store/apps/details?id=hr.mireo.arthur&hl=en&fbclid=IwAR3c-PkUXOea8KrKLp9Q3JUjCidGmgO2jYX_Qb7O8VuWlHXPIS5nDOfKRKI&showAllReviews=true
Here is my code so far:
url <- 'https://play.google.com/store/apps/details?id=hr.mireo.arthur&hl=en&fbclid=IwAR3c-PkUXOea8KrKLp9Q3JUjCidGmgO2jYX_Qb7O8VuWlHXPIS5nDOfKRKI&showAllReviews=true'
#Open webpage using RSelenium
rD <- rsDriver(port = 4445L, browser=c("chrome"), chromever="80.0.3987.106")
remDr <- rD[["client"]]
remDr$navigate(url)
#-----------------------------------------Load whole page by scrolling and showing more
xp_show_more <- "//*[#id='fcxH9b']/div[4]/c-wiz/div/div[2]/div/div[1]/div/div/div[1]/div[2]/div[2]/div"
replicate(5,
{
replicate(5,
{
# scroll down
webElem <- remDr$findElement("css", "body")
webElem$sendKeysToElement(list(key = "end"))
# wait
Sys.sleep(1)
})
# find button
morereviews <- remDr$findElement(using = 'xpath', xp_show_more)
# click button
tryCatch(morereviews$clickElement(),error=function(e){print(e)}) # trycatch to prevent any error from stopping the loop
# wait
Sys.sleep(3)
})
This code will load the whole page and show all comments, but some comments are long and have the "Full review" buttons which have to be clicked in order to show the whole lenght of the comment. I have managed to find all of those buttons (there are 36 of them) using the "findElements" function withfollowing code:
fullreviews <- remDr$findElements(using = 'css selector', value = ".OzU4dc")
This code results in a list of 36 elements and when I want to click on them using this code:
fullreviews$clickElement()
I get the this error:
Error: attempt to apply non-function
I can click on all 36 elements using this 36 lines of code:
fullreviews[[1]]$clickElement()
fullreviews[[2]]$clickElement()
fullreviews[[3]]$clickElement()
...ans so on until 36.
But I would like to make this possible with one function. In case I have to scrape some bigger webpage and have thousands of elements to click.
I have tried this code, but it doesn't work
fullreviews[[1:36]]$clickElement()
I guess some sort of lapply function is needed, but I can't manage to creat one which is working.
Is there a way to do this in a single loop or function?
It does not really align with stackoverflow policy since I am not showing what I have done but I really have no clue how to even start on this question given my lack of technical expertise. Hope someone can post a solution or at least point me to the right direction.
I want to download all the data from this website:
http://aps.dac.gov.in/APY/Public_Report1.aspx
I need to download all the data i.e. all season * all year * all states * all crops. The longer (frustrating!) way to approach is to just click all the boxes and press download.
However, I was wondering if anyone has any programming solution to download this data. I would preferably want to do this in R because that's the language I understand but feel free to tag other programming languages.
Here's a solution using RSelenium to instance a browser and direct it to do your bidding.
library(RSelenium)
driver <- rsDriver()
remDr <- driver[["client"]]
remDr$navigate("http://aps.dac.gov.in/APY/Public_Report1.aspx") #navigate to your page
You basically need to tell the browser to select each button you want to mark, using SelectorGadget to find the unique ID for each, then pass them one-by-one to webElem. Then use the webElem methods to make the page do things.
webElem <- remDr$findElement(using = 'id', value = "TreeViewSeasonn0CheckBox")
webElem$highlightElement() #quick flash as a check we're in the right box
webElem$clickElement() #performs the click
#now do the same for each other box
webElem <- remDr$findElement(using = 'id', value = "TreeView1n0CheckBox")
webElem$highlightElement()
webElem$clickElement()
webElem <- remDr$findElement(using = 'id', value = "TreeView2n0CheckBox")
webElem$highlightElement()
webElem$clickElement()
webElem <- remDr$findElement(using = 'id', value = "TreeViewYearn0CheckBox")
webElem$highlightElement()
webElem$clickElement()
Now choose the report form you want and click the download button. Assuming it's Excel format here.
webElem <- remDr$findElement(using = 'id', value = "DdlFormat")
webElem$sendKeysToElement(list("Excel", key = "enter"))
webElem <- remDr$findElement(using = 'id', value = "Button1")
webElem$clickElement() #does the click
For what it's worth, the site timed out on trying to download all the data for me. Your results may vary.