R - Get html-addresses from data frame to rvest - r

Question

I am new to R, and I have come upon a problem I can't solve. I would like to scrape Swedish election data at electoral district level. They are structured as can be found here http://www.val.se/val/val2014/slutresultat/K/valdistrikt/25/82/0134/personroster.html
I get the data I want by using this code:
library(rvest)
district.data <- read_html("http://www.val.se/val/val2014/slutresultat/K/kommun/25/82/0134/personroster.html")
prost <- district.data %>%
html_nodes("table") %>%
.[[2]] %>%
html_table()
But that is just for one district out of 6,227 districts. The districts are identified by the html address. In the website mentioned above it is identified by "25/82/0134". I can find the identities of all districts here http://www.val.se/val/val2014/statistik/2014_riksdagsval_per_valdistrikt.skv
And I read this semi-colon separated file into R by using this code:
valres <- read_csv2("http://www.val.se/val/val2014/statistik/2014_riksdagsval_per_valdistrikt.skv" )
(as a side note, how can I change the encoding so that the Swedish letters (e.g. å, ä, ö) are imported correctly? I manage to do that with read.csv and specifying encoding='utf-8' but not with read_csv)
In this data frame, the columns LAN, KOM and VALDIST give the identities of the districts (note that VALDIST sometimes just have 2 characters). Hence the addresses have the following structure http://www.val.se/val/val2014/slutresultat/K/kommun/LAN/KOM/VALDIST/personroster.html
So, I would like to use the combination in each row to get the identity of district, scrape the information into R, add a column with the district identity (i.e. LAN, KOM and VALDIST combined into one string), and do so over all 6,227 districts and append the information from each of those districts into one single data frame. I assume I need to use some kind of loop or some of those apply functions, to iterate over the data frame, but I have not figured out how.
UPDATE:
After the help I received (thank you!) in the answer below, the code now is as follows. My remaining problem is that I want to add the district identity (i.e. paste0(LAN, KOM, VALDIST)) for each website that is scraped to a column in the final data frame. Can someone help me with this final step?
# Read the indentities of the districts (w Swedish letters)
districts_url <- "http://www.val.se/val/val2014/statistik/2014_riksdagsval_per_valdistrikt.skv"
valres <- read_csv2(districts_url, locale=locale("sv",encoding="ISO-8859-1", asciify=FALSE))
# Add a variabel to separate the two types of electoral districts
valres$typ <- "valdistrikt"
valres$typ [nchar(small_valres$VALDIST) == 2] <- "onsdagsdistrikt"
# Create a vector w all the web addresses to the district data
base_url <- "http://www.val.se/val/val2014/slutresultat/K/%s/%s/%s/%s/personroster.html"
urls <- with(small_valres, sprintf(base_url, typ, LAN, KOM, VALDIST))
# Scrape the data
pb <- progress_estimated(length(urls))
map_df(urls, function(x) {
pb$tick()$print()
# Maybe add Sys.sleep(1)
read_html(x) %>%
html_nodes("table") %>%
.[[2]] %>%
html_table()
}) -> df
Any help would be greatly appreciated!
All the best,
Richard

Answer 1

You can use sprintf() to do positional substitution and then use purrr::map_df() to iterate over a vector of URLs and generate a data frame:
library(rvest)
library(readr)
library(purrr)
library(dplyr)
districts_url <- "http://www.val.se/val/val2014/statistik/2014_riksdagsval_per_valdistrikt.skv"
valres <- read_csv2(districts_url, locale=locale("sv",encoding="UTF-8", asciify=FALSE))
base_url <- "http://www.val.se/val/val2014/slutresultat/K/valdistrikt/%s/%s/%s/personroster.html"
urls <- with(valres, sprintf(base_url, LAN, KOM, VALDIST))
pb <- progress_estimated(length(urls))
map_df(urls, function(x) {
pb$tick()$print()
read_html(x) %>%
html_nodes("table") %>%
.[[2]] %>%
html_table()
}) -> df
HOWEVER, you should add a randomized delay to avoid being blocked as a bot and should look at wrapping read_html() with purrr::safely() since not all those LAN/KOM/VALDIST combinations are valid URLs (at least in my testing).
That code also provides a progress bar since it's going to take a while (prbly an hour on a moderately decent connection).