How to scrape with rvest? - web-scraping

Question

I need to get three different numbers (in yellow, see picture) from this page:
https://www.scopus.com/authid/detail.uri?authorId=7006040753
I used this code using rvest and inspectorgadget:
site=read_html("https://www.scopus.com/authid/detail.uri?authorId=7006040753")
hindex=site %>% html_node(".row3 .valueColumn span")%>% html_text()
documents=site %>% html_node("#docCntLnk")%>% html_text()
citations=site %>% html_node("#totalCiteCount")%>% html_text()
print(citations)
I can get the h-index and documents but the citations do not work
Can you help me?

Answer 1

Now I've found a solution - I noticed the value took some time to load so I included a little timeout in the PhnatomJS script. Now it works on my machine with the following R code:
setwd("path/to/phantomjs/bin")
system('phantomjs readexample.js') # call PhantomJS script (stored in phantomjs/bin)
totalCiteCount <- "rendered_page.html" %>% # "rendered_page.html" is created by PhantomJS
read_html() %>%
html_nodes("#totalCiteCount") %>%
html_text()
## totalCiteCount
## [1] "52018"
The corresponding PhantomJS script file "readexample.js" looks like the following (kudos to https://www.r-bloggers.com/web-scraping-javascript-rendered-sites/):
var webPage = require('webpage');
var url ='https://www.scopus.com/authid/detail.uri?authorId=7006040753';
var fs = require('fs');
var page = webPage.create();
var system = require('system');
page.settings.userAgent = 'Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)';
page.open(url, function (status) {
setTimeout(function() {
fs.write('rendered_page.html', page.content, 'w');
phantom.exit();
}, 2500);
});
The code throws the following errors in R, but at least the value is scraped correctly.
> system('phantomjs readexample.js') TypeError: undefined is not a constructor (evaluating 'mutation.addedNodes.forEach')
https://www.scopus.com/gzip_N1846232499/bundles/SiteCatalystTop.js:73 :0 in forEach https://www.scopus.com/gzip_N1846232499/bundles/SiteCatalystTop.js:73 ReferenceError: Can't find variable: SDM
https://www.scopus.com/gzip_N1729184664/bundles/AuthorProfileTop.js:73 in sendIndex https://www.scopus.com/gzip_N1729184664/bundles/AuthorProfileTop.js:67 in loadEvents
Using PhantomJS is quite convenient as you don't have to install anything (so also works if you don't have admin rights on your machine). Simply download the .zip file and unpack it to any folder. Afterwards set the working directory in R (setwd()) to the "phantomjs/bin" folder and it works.
You can also change the PhantomJS scripte (iteratively if desired) in R e.g. to pass different URLs to the script in a loop. Example:
for (i in 1:n_urls) {
url_var <- urls[i] # assuming you have created a var "urls" with multiple URLs before
lines <- readLines("readexample.js")
lines[2] <- paste0("var url ='", url_var ,"';") # exchange code line with new URL
writeLines(lines, "readexample.js") # new url is stored in the PhantomJS script
system('phantomjs readexample.js')
# <any code> #
}
Hope this brings you one step further?