How to deal with this website in a webscraping format? - r

Question

I am trying to webscrape this website.
I am applying the same code that I always use to webscrape pages:
url_dv1 <- "https://ec.europa.eu/commission/presscorner/detail/en/qanda_20_171?fbclid=IwAR2GqXLmkKRkWPoy3-QDwH9DzJiexFJ4Sp2ZoWGbfmOR1Yv8POdlLukLRaU"
url_dv1 <- paste(html_text(html_nodes(read_html(url_dv1), "#inline-nav-1 .ecl-paragraph")), collapse = "")
For this website, thought, the code doesn't seem to be working. In fact, I get Error in UseMethod("read_xml") :
no applicable method for 'read_xml' applied to an object of class "c('xml_document', 'xml_node')".
Why is it so? How can I fix it?
Thanks a lot!

Answer 1

The problem is that the web page is dynamically rendered. You can overcome this using phantomjs (can be downloaded here https://phantomjs.org/download.html). You will also need a custom javascript script (see below). The below R code works for me.
library(tidyverse)
library(rvest)
dir_js <- "path/to/a/directory" # JS code needs to be inserted here, the name of the file needs to be javascript.js
url <- "https://ec.europa.eu/commission/presscorner/detail/en/qanda_20_171?fbclid=IwAR2GqXLmkKRkWPoy3-QDwH9DzJiexFJ4Sp2ZoWGbfmOR1Yv8POdlLukLRaU"
system2("path/to/where/you/have/phantomjs.exe", # directory to phantomJS
args = c(file.path(dir_js, "javascript.js"), url))
read_html("myhtml.html") %>%
html_nodes("#inline-nav-1 .ecl-paragraph") %>%
html_text()
# this is the javascript code to be saved in javascript directory as javascript.js
// create a webpage object
var page = require('webpage').create(),
system = require('system')
// the url for each country provided as an argument
country= system.args[1];
// include the File System module for writing to files
var fs = require('fs');
// specify source and path to output file
// we'll just overwirte iteratively to a page in the same directory
var path = 'myhtml.html'