Problems with web-scraping EconPapers repository - web-scraping

Question

I am looking for some advice on how to web-scrape a repository of economics papers. The site is EconPapers, I found some tutorials on how to instruct R (unfortunately I am not able to use Python since it does not work on my laptop) but all the examples I have seen deal with HTML tags that are easily recoverable from the developer tool. In my specific case, when I open the developer tool I see the very different structures with respect to those that I read. I am not very into these things, since am I an economist and I am looking for someone that can explain to me where I have to look for the information I need. Here attached you will find the HTML code, I hope that someone can help me.
The website I am trying to web-scraping is available at the following link: https://econpapers.repec.org/scripts/search.pf?ft=;wp=on;pl=;sort=date;lgc=AND;aus=;ar=on;kw=;jel=;nep=;ni=1%20year;nit=epdate

Answer 1

Scraping the title is pretty simple since it's in an <h1> tag with class colored under the <div> tag with class bodytext (the bodytext classed div is the whole main content). (I scrape a single page https://econpapers.repec.org/paper/haljournl/hal-02490178.htm for example):
library(rvest)
url <- "https://econpapers.repec.org/paper/haljournl/hal-02490178.htm"
pg <- read_html(url)
title <- pg %>% html_node(xpath = "//div[#class='bodytext']/h1[#class='colored']") %>% html_text()
Output:
> title
[1] "I CRACKED UP... BUT I'M NOT PROUD … \": WHEN THE RESPONSIBLE CONSUMER DEVIATES FROM HIS PERSONAL NORMS"
Other information can be tricky to scrape since they aren't assigned any ID, name, class,... You have to be creative to find a way to identify them. I'll give you some examples for you to get the idea:
Authors: Since all authors' names are italic and nothing else in the content is italic, you can use that to identify authors:
authors <- paste(pg %>% html_nodes(xpath = "//div[#class='bodytext']/p/i") %>% html_text(), collapse = ", ")
Output:
> authors
[1] "Sophie Martins, Stéphanie Montmasson, Fabien Rogeon"
Abstract (similar to Date, Note,...): The abstract info followed after the <b> tag contains Abstract text:
abstract <- pg %>% html_node(xpath = "//div[#class='bodytext']/p/b[contains(., 'Abstract')]/following-sibling::text()[1]") %>% html_text(trim = TRUE)
Output:
> abstract
[1] "Souvent considéré comme exemplaire de par son comportement à l'égard de l'environnement, le consommateur responsable est, lui aussi, amené à agir consciemment en désaccord avec ses normes personnelles. Cette présente recherche apporte une compréhension des raisons et circonstances qui incitent les consommateurs responsables à transgresser leurs normes personnelles et propose un éclairage sur la façon dont ces consommateurs gèrent leurs comportements déviants de leurs convictions environnementales grâce aux stratégies de coping. Au terme d'une exploration empirique s'appuyant sur la technique des incidents critiques, cette recherche met à jour l'influence de facteurs émotionnels, sociaux et situationnels dans l'apparition de comportements allant à l'encontre de leurs normes personnelles. Puis sont abordées les différentes stratégies de coping mises en place par le consommateur responsable. Celles-ci sont en ligne avec les stratégies identifiées par la recherche en management et en marketing à la différence que la culpabilité demeure peu évoquée par les répondants. Abstract : Mostly considered as an example because of their pro-environmental behaviour, responsible consumers are also called upon to act consciously in disagreement with their perso nal norms. This research provides an understanding of the reasons and circumstances that lead responsible consumers to transgress their personal norms and sheds light on how these consumers manage their behaviour that deviates from their environmental beliefs through coping strategies. After an empirical exploration based on the critical incident technique, this research reveals the influence of emotional, social and situational factors in the development of behaviours that run counter to their personal norms. The various coping strategies implemented by the responsible consumer are then discussed. These are in line with the strategies management and marketing research has pointed out, with the difference that respondents rarely mention guilt."
Date:
date <- pg %>% html_node(xpath = "//div[#class='bodytext']/p/b[contains(., 'Date')]/following-sibling::text()[1]") %>% html_text(trim = TRUE)
Output:
> date
[1] "2020-05-07"
I think you get the point. To scrape all sites, create a loop with this code (I haven't tried, let me know if you can't do it yourself).