I use RSelenium to fill in a webform. To select an option from a dropdown I use the following:
xpathoption <- paste0("//select[#id = '",samplepatient[p,'name'],"']/option[",samplepatient[p,'value'],"]")
optionelem <- remDrv$findElement(using = "xpath", xpathoption)
selectelem <- remDrv$findElement(using = "xpath"
, paste0("//select[#id = '",samplepatient[p,'name'],"']"))
optionelem$clickElement()
selectelem$screenshot(display = T)
I use the following to check if the correct option was selected:
remDrv$findElement(using = "xpath", paste0("//select[#id = '",samplepatient[p,'name'],"']"))$getElementAttribute("value")[[1]]
The problem I have is when the clickElement() command is run twice, the result of the last command changes. I also checked the outcome with screenshot(). It also shows that a different option is switched to when using the clickElement() command twice.
Is there a different way to select the option from a dropdown list, that is not creating this behavior?
I use a docker on ubuntu with firefox 3.0.1.
The form is from a calculator I want to use. To open the form itself you need to first check the disclaimer, like so:
remDrv$navigate('http://riskcalculator.facs.org/RiskCalculator/')
remDrv$findElement(using = "xpath", "//input[#id = 'chkDisclaimer']")$clickElement()
Sys.sleep(1)
remDrv$findElement(using = "xpath", "//input[#id = 'btnContinue']")$clickElement()
Sys.sleep(1)
a reproducable example after the disclaimer is:
#select age group
optionelem <- remDrv$findElement(using = "xpath", "//select[#id = 'AgeGroup']/option[3]")
selectelem <- remDrv$findElement(using = "xpath", "//select[#id = 'AgeGroup']")
#first attempt
optionelem$clickElement()
selectelem$getElementAttribute("value")
# result = 3
#second attempt
optionelem$clickElement()
selectelem$getElementAttribute("value")
# result = 1
As mentioned in one of the comments, the issue has not to do with RSelenium but with the docker used. I now use a chrome docker (standalone-chrome) that does not have the same issue with selecting an option in a dropdown.
I don't come across any issues when selecting options using clickElement
for example:
remDrv$navigate('http://riskcalculator.facs.org/RiskCalculator/')
remDrv$findElement("id", "chkDisclaimer")$clickElement()
Sys.sleep(1)
remDrv$findElement("id", "btnContinue")$clickElement()
Sys.sleep(1)
#select age group
ageElems <- remDrv$findElements("css", "#AgeGroup option")
ageElems[[3]]$clickElement()
#select Diabetes
diaElems <- remDrv$findElements("css", "#Diabetes option")
diaElems[[2]]$clickElement()
# Select Gender
genderElems <- remDrv$findElements("css", "#Gender option")
genderElems[[1]]$clickElement()
When running in Docker you can use "debug" image and a VNC viewer to see exactly whats happening in the browser.
Related
I scraped a bunch of url's and was using RSelenium to add the links to the archive.org website. My loop was about half way finished when it broke. So I used the which() function to determine where it left off, then I restarted the loop from that point.
The problem is that the driver kept typing the wrong index instead of the one I specified in the loop. To fix the problem, I subset 'mget_links' to 'mget_links2' so that it only contained the remaining links I needed. I ran the loop again and it somehow still typed an index from 'mget_links'. The driver keeps typing in this link: https://www.bjjcompsystem.com/tournaments/1869/categories/2053146
What is going on here? How do I fix it?
library(rvest)
library(tidyverse)
library(RSelenium); library(netstat)
pageMen = read_html('https://www.bjjcompsystem.com/tournaments/1869/categories')
mget_links <- pageMen %>%
html_nodes('.categories-grid__category a') %>%
html_attr('href') %>%
paste0('https://www.bjjcompsystem.com', .)
# know the length for the loop
length(mget_links)
remote_driver = rsDriver(browser = 'firefox',
verbose = F,
port = free_port())
rd = remote_driver$client
rd$open()
rd$navigate('https://web.archive.org/save')
rd$maxWindowSize()
for (i in 1:length(mget_links[255:259])){
save_page_box = rd$findElement(using = 'xpath', '//*[(#id = "web-save-url-input")]')
save_page_box$clickElement()
save_page_box$sendKeysToElement(list(mget_links[i], key='enter'))
Sys.sleep(26)
return.to.save.page = rd$findElement(using = 'link text', 'Return to Save Page Now')
return.to.save.page$clickElement()
}
# Reran above code with mget_links2 instead of mget_links[255:259]. I also tried #mget_links2[1:156] with no luck.
mget_links2= mget_links[104:259]
Im trying to create a function that makes and action using selenium, extract a text and creates an element. But, I dont know how to automate the part when the desc element is created for every link used in lapply, so I wonder if anyone have an idea of how to do that, and then those elements created, go to a table as final result.
This is my Code, so far:
## web scrapper
library(tidiverse)
library(RSelenium)
library(netstat)
rs_driver_object=rsDriver(browser = "chrome", chromever = "104.0.5112.79", verbose = FALSE, port = free_port())
RemDr=rs_driver_object$client
links=list(c("https://in.linkedin.com/company/almacenes-exito-s-a-",
"https://in.linkedin.com/company/acerias-de-colombia---acesco",
"https://www.linkedin.com/company/amt-consultores-legales",
"https://www.linkedin.com/company/constructora-amarilo-sa"))
allwebs=function(x){
RemDr$open()
RemDr$navigate("https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin")
username <- RemDr$findElement(using = "id", value = "username")
username$sendKeysToElement(list("XXX#XXXX.COM"))
password=RemDr$findElement(using = "id", value = "password")
password$sendKeysToElement(list("XXXXPASS", key="enter"))
RemDr$navigate(x)
seedesc= RemDr$findElement(using = "id", value = "line-clamp-show-more-button")$clickElement()
desc=RemDr$findElement(using = "id", value = "ember52")
desc$getElementText()
}
lapply(links, allwebs)
I have used the rvest package in R to scrape unique URLs before.
However, I am now stuck with a particular website. The URL stays static and I need to select the following dropdowns now and scrape the resulting table that appears.
Will be helpful if someone can guide me on what direction to take with websites like these? Is R even capable of doing this?
Edit: I have researched and it seems RSelenium can handle such tasks. Unfortunately, I have no exposure to it. Can someone recommend an example/blog/material online on using Selenium specifically for clicking and scraping for someone as noob as I am?
I have made a blog post about an RSelenium example:
https://guillaumepressiat.github.io/blog/2021/04/RSelenium-paginated-tables
this website contains a lot of things about selenium, you will have to plug it to RSelenium api package.(verbs are almost the same in all languages, findElement, etc) https://www.guru99.com/selenium-tutorial.html
But as an example based on your question maybe something like this to begin:
# https://stackoverflow.com/q/67021563/10527496
# java -jar selenium-server-standalone-3.9.1.jar
library(RSelenium)
library(tidyverse)
library(rvest)
library(httr)
remDr <- remoteDriver(
remoteServerAddr = "localhost",
port = 4444L, # change port according to terminal
browserName = "firefox"
)
remDr$open()
# remDr$getStatus()
url <- "https://fcainfoweb.nic.in/reports/Report_Menu_Web.aspx"
remDr$navigate(url)
Sys.sleep(5)
# first : radio buttons
u1 <- remDr$findElement(using = "id", value = 'ctl00_MainContent_Rbl_Rpt_type_0')
u2 <- remDr$findElement(using = "id", value = 'ctl00_MainContent_Rbl_Rpt_type_1')
u3 <- remDr$findElement(using = "id", value = 'ctl00_MainContent_Rbl_Rpt_type_2')
u4 <- remDr$findElement(using = "id", value = 'ctl00_MainContent_Rbl_Rpt_type_3')
dynam <- remDr$mouseMoveToLocation(webElement = u1)
u1$click()
Sys.sleep(5)
# second : Select input
s1 <- remDr$findElement(using = "id", value = 'ctl00_MainContent_Ddl_Rpt_Option0')
# get available choices
s_choices <- read_html(s1$getElementAttribute('innerHTML')[[1]]) %>%
html_nodes('option') %>%
html_attrs() %>%
unlist() %>%
.[3:length(.)] %>%
as.vector()
dynam <- remDr$mouseMoveToLocation(webElement = s1)
s1$click()
s1$sendKeysToElement(sendKeys = list(s_choices[1], key = "enter"))
# s_choices[1] is "Daily Prices"
Sys.sleep(5)
# get date choices
s_date_choices <- remDr$findElement(using = "id", value = "ctl00_MainContent_Txt_FrmDate")
dynam <- remDr$mouseMoveToLocation(webElement = s_date_choices)
s_date_choices$click()
s_date_choices$sendKeysToElement(sendKeys = list('01/01/2021', key = "enter"))
Sys.sleep(5)
s_table <- remDr$findElement(using = "id", value = "Panel1")
# get first tables as an example
results_1 <- read_html(s_table$getElementAttribute('innerHTML')[[1]]) %>%
html_table(fill = TRUE) %>%
.[2:length(.)]
we get a list of three tables as a result:
Making a function from this code to loop on a date vector is possible after that I think (you will have to reload a fresh start page on base URL for each date I suppose).
I'm trying to scrape links to all minutes and agenda provided in this website: https://www.charleston-sc.gov/AgendaCenter/
I've managed to scrape section IDs associated with each category (and years for each category) to loop through the contents within each category-year (please see below). But I don't know how to scrape the hrefs that lives inside the contents. Especially because the links to Agenda lives inside the drop down menu under 'download', it seems like I need to go through extra clicks to scrape the hrefs.
How do I scrape the minutes and agenda (inside the download dropdown) for each table I select? Ideally, I would like a table with the date, title of the agenda, links to minutes, and links to agenda.
I'm using RSelenium for this. Please see the code I have so far below, which allows me to click through each category and year, but not else much. Please help!
rm(list = ls())
library(RSelenium)
library(tidyverse)
library(httr)
library(XML)
library(stringr)
library(RCurl)
t <- readLines('https://www.charleston-sc.gov/AgendaCenter/', encoding = 'UTF-8')
co <- str_match(t, 'aria-label="(.*?)"[ ]href="java')[,2]
yr <- str_match(t, 'id="(.*?)" aria-label')[,2]
df <- data.frame(cbind(co, yr)) %>%
mutate_all(as.character) %>%
filter_all(any_vars(!is.na(.))) %>%
mutate(id = ifelse(grepl('^a0', yr), gsub('a0', '', yr), NA)) %>%
tidyr::fill(c(co,id), .direction='down')%>% drop_na(co)
remDr <- remoteDriver(port=4445L, browserName = "chrome")
remDr$open()
remDr$navigate('https://www.charleston-sc.gov/AgendaCenter/')
remDr$screenshot(display = T)
for (j in unique(df$id)){
remDr$findElement(using = 'xpath',
value = paste0('//*[#id="cat',j,'"]/h2'))$clickElement()
for (k in unique(df[which(df$id==j),'yr'])){
remDr$findElement(using = 'xpath',
value = paste0('//*[#id="',k,'"]'))$clickElement()
# NEED TO SCRAPE THE HREF ASSOCIATED WITH MINUTES AND AGENDA DOWNLOAD HERE #
}
}
Maybe you don't really need to click through all the elements? You can use the fact that all downloadable links have ViewFile in their href:
t <- readLines('https://www.charleston-sc.gov/AgendaCenter/', encoding = 'UTF-8')
viewfile <- str_extract_all(t, '.*ViewFile.*', simplify = T)
viewfile <- viewfile[viewfile!='']
library(data.table) # I use data.table because it's more convenient - but can be done without too
dt.viewfile <- data.table(origStr=viewfile)
# list the elements and patterns we will be looking for:
searchfor <- list(
Title='name=[^ ]+ title=\"(.+)\" href',
Date='<strong>(.+)</strong>',
href='href=\"([^\"]+)\"',
label= 'aria-label=\"([^\"]+)\"'
)
for (this.i in names(searchfor)){
this.full <- paste0('.*',searchfor[[this.i]],'.*');
dt.viewfile[grepl(this.full, origStr), (this.i):=gsub(this.full,'\\1',origStr)]
}
# Clean records:
dt.viewfile[, `:=`(Title=na.omit(Title),Date=na.omit(Date),label=na.omit(label)),
by=href]
dt.viewfile[,Date:=gsub('<abbr title=".*">(.*)</abbr>','\\1',Date)]
dt.viewfile <- unique(dt.viewfile[,.(Title,Date,href,label)]); # 690 records
What you have as the result is a table with the links to all downloadable files. You can now download them using any tool you like, for example using download.file() or GET():
dt.viewfile[, full.url:=paste0('https://www.charleston-sc.gov', href)]
dt.viewfile[, filename:=fs::path_sanitize(paste0(Title, ' - ', Date), replacement = '_')]
for (i in seq_len(nrow(dt.viewfile[1:10,]))){ # remove `1:10` limitation to process all records
url <- dt.viewfile[i,full.url]
destfile <- dt.viewfile[i,filename]
cat('\nDownloading',url, ' to ', destfile)
fil <- GET(url, write_disk(destfile))
# our destination file doesn't have extension, we need to get it from the server:
serverFilename <- gsub("inline;filename=(.*)",'\\1',headers(fil)$`content-disposition`)
serverExtension <- tools::file_ext(serverFilename)
# Adding the extension to the file we just saved
file.rename(destfile,paste0(destfile,'.',serverExtension))
}
Now the only problem we have is that the original webpage was only showing records for the last 3 years. But instead of clicking View More through RSelenium, we can simply load the page with earlier dates, something like this:
t <- readLines('https://www.charleston-sc.gov/AgendaCenter/Search/?term=&CIDs=all&startDate=10/14/2014&endDate=10/14/2017', encoding = 'UTF-8')
then repeat the rest of the code as necessary.
I'm using RSelenium to click on a dynamic element after a search on this webpage: http://www.in.gov.br/web/guest/inicio.
Every time I search for a word, I would like to find the words/link 'Ministério Da Educação' (it is the portuguese equivalent for Ministry of Education) on the right side of the results webpage and click on it.
I have used the inspect element feature from Google Chrome, but I am not having any success on finding and clicking that element. I have already tried using xpath, css selector, id ...
I am using the following code:
## search parameters
string_search <- "contrato"
date_search <- format(
as.Date("17/04/2019", "%d/%m/%Y"),
"%d/%m/%Y") #brazilian format
## start Selenium driver
library(RSelenium)
selCommand <- wdman::selenium(
jvmargs = c("-Dwebdriver.firefox.verboseLogging=true"),
retcommand = TRUE)
shell(selCommand, wait = FALSE, minimized = TRUE) # for windows
# system(selCommand) # for Linux
remDr <- remoteDriver(port = 4567L, browserName = "firefox")
remDr$open()
## navigation & search
remDr$navigate("http://www.in.gov.br/web/guest/inicio")
Sys.sleep(5)
# from date
datefromkey<-remDr$findElement(using = 'css', "#calendario_advanced_from")
datefromkey$clickElement()
datefromkey$sendKeysToElement(list(key = "enter"))
datefromkey$clearElement()
datefromkey$sendKeysToElement(list(date_search))
datefromkey$sendKeysToElement(list(key = "enter"))
# to date
datetokey<-remDr$findElement(using = 'css', "#calendario_advanced_to")
datetokey$clickElement()
datetokey$sendKeysToElement(list(key = "enter"))
datetokey$clearElement()
datetokey$sendKeysToElement(list(date_search))
datetokey$sendKeysToElement(list(key = "enter"))
# string to search
wordkey<-remDr$findElement(using = 'css', "#input-advanced_search")
wordkey$sendKeysToElement(list('"', string_search, '"'))
# click search button
press_button <- remDr$findElement(using = 'class', "btn")
press_button$clickElement()
Here is where I struggle:
1) first attempt: using a broader tag
# using a broader tag
categorykey <- remDr$findElement(using = 'id', '_3_facetNavigation')
categorykey$getElementText()
With getElementText() I see that "Ministério da Educação" is there, but I do not know how to click on the link.
2) second attempt: using the xpath
categorykey <- remDr$findElement('xpath', '//li
[#id="yui_patched_v3_11_0_1_1555545676970_404"]/text()')
It returns an error. Selenium can't locate the element.
Found the solution myself after watching this video on YouTube:
How to locate Dynamic Elements in Selenium Webdriver - XPATH Tutorial
The code would be like this:
categorykey <-remDr$findElement('xpath', '//*[contains(#data-value,"ministério da
educação")]')
categorykey$getElementText()
# just to see if it's right
categorykey$clickElement()