Web scraping dynamic webpage with R - r

Question

my goal is to get data from this site : https://www.insee.fr/fr/recherche?q=Emploi-Population+active+en+2018&taille=20&debut=0, especially id links of different items.
I know that GET function doesn't work because it's dynamic and needed to be process by javascript (same that Web Scraping dynamic webpage Python). So i get info via inspector mode of my browser and found a POST query with the url.Here is a reproductible example :
library(httr)
body <- list(q="Emploi-Population%20active%20en%202018",
start="0",
sortFields=data.frame(field="score",order="desc"),
filters=data.frame(NULL),
rows="50",
facetsQuery=data.frame(NULL))
TMP <- httr::POST(url = "http://www.insee.fr/fr/solr/consultation?q=Emploi-Population%20active%20en%202018",
body = body,
config = config(http_version=1.1),
encode = "json",verbose())
Note that a i have to put http instead of https because i get nothing otherwise (My proxy is correctly configured and rstudio can connect to the internet).
All i get is a nice 500 error. Any Idea of what i miss ?

Answer 1

You can change the q param and remove it from your url. I would use https and remove your config line to avoid the curl fetch error. However, the below, adapted to return 100 results, still works.
library(httr)
body <- list(
q = "Emploi-Population active en 2018",
start = "0",
sortFields = data.frame(field = "score", order = "desc"),
rows = "100"
)
TMP <- httr::POST(
url = "http://www.insee.fr/fr/solr/consultation",
body = body,
config = config(http_version = 1.1),
encode = "json", verbose()
)
data <- fromJSON(content(TMP, type = "text"))
print(data$documents$titre)

Answer 2

I found that passing the json as a string worked fine:
library(httr)
json <- paste0('{"q":"Emploi-Population active en 2018 ",',
'"start":"0","sortFields":[{"field":"score","order":"desc"}],',
'"filters":[],"rows":"20","facetsQuery":[]}')
url <- paste0('https://www.insee.fr/fr/solr/consultation?q=Emploi-Population',
'%20active%20en%202018%20')
res <- POST(url, body = json, content_type_json())
output <- content(res)
Now output is a massive list, but here for example are the document titles:
sapply(output$documents, function(x) x$titre)
#> [1] "Emploi-Population active en 2018"
#> [2] "Emploi – Population active"
#> [3] "Dossier complet"
#> [4] "Base du dossier complet"
#> [5] "Emploi-Population active en 2017"
#> [6] "Comparateur de territoire"
#> [7] "Emploi – Population active"
#> [8] "L'essentiel sur... les entreprises"
#> [9] "Emploi - population active en 2014"
#> [10] "Population active"
#> [11] "Emploi salarié et non salarié par activité"
#> [12] "Évolution de l'emploi"
#> [13] "Logements, individus, activité, mobilités scolaires et professionnelles, migrations résidentielles en 2018"
#> [14] "Emploi selon le sexe et l’âge"
#> [15] "Statut d’emploi et type de contrat selon le sexe et l’âge"
#> [16] "Sous-emploi selon le sexe et l’âge"
#> [17] "Emploi salarié par secteur"
#> [18] "Fiche - Chômage"
#> [19] "Emploi-Activité en 2018"
#> [20] "Activité professionnelle des individus : lieu de travail localisé à la zone d'emploi en 2017"
Created on 2022-05-31 by the reprex package (v2.0.1)