rvest | Webscraping data into a long format - r

Question

While web-scraping i came across the following problem for which i think there might be a better solution:
Having this data:
dat <- data.frame(query = c("Washington, USA", "Frankfurt, Germany"))
query
1 Washington, USA
2 Frankfurt, Germany
I would like to query e.g. the Google Maps Api and return the formatted address(es). There might be multiple formatted. The result should be the following:
query formatted_address
1 Washington, USA Washington, DC, USA
2 Washington, USA Washington, UT, USA
3 Washington, USA Washington, VA 22747, USA
4 Washington, USA Washington, IA 52353, USA
5 Washington, USA Washington, GA 30673, USA
6 Washington, USA Washington, PA 15301, USA
7 Frankfurt, Germany Frankfurt, Germany
What i do by now is this:
require(RCurl)
require(rvest)
require(magrittr)
build_url <- function(x, base_url = "https://maps.googleapis.com/maps/api/geocode/xml?address="){
paste0(base_url, RCurl::curlEscape(x))
}
l <- lapply(dat$query, function(q){
formatted_address <- q %>% build_url %>% read_xml %>% xml_nodes("formatted_address") %>% xml_text
data.frame(query = q, formatted_address)
})
do.call(rbind, l) # This can be done via data.table::rbindlist as well
Is there a better solution? Maybe more data.table or dplyr style?

Answer 1

I've written the package googleway to access google maps API with a valid API key (so if your data is greater than 2,500 items you can pay for an API key).
To get the address details use google_geocode()
library(googleway)
key <- "your_api_key"
dat <- data.frame(query = c("Washington, USA", "Frankfurt, Germany"))
## To get all the data:
res <- apply(dat, 1, function(x){
google_geocode(address = x["query"],
key = key) ## use simplify = F to return JSON
})
## to access the 'formatted address' part, see
res[[1]]$results$formatted_address
# [1] "Washington, DC, USA" "Washington, UT, USA" "Washington, VA 22747, USA" "Washington, IA 52353, USA"
# [5] "Washington, GA 30673, USA" "Washington, PA 15301, USA"
## so to get everything as a list
lapply(res, function(x){
x$results$formatted_address
})
# [[1]]
# [1] "Washington, DC, USA" "Washington, UT, USA" "Washington, VA 22747, USA" "Washington, IA 52353, USA"
# [5] "Washington, GA 30673, USA" "Washington, PA 15301, USA"
#
# [[2]]
# [1] "Frankfurt, Germany"
## and to put back onto your original data.frame:
lst <- lapply(1:length(res), function(x){
data.frame(query = dat[x, "query"],
formatted_address = res[[x]]$results$formatted_address)
})
data.table::rbindlist(lst)
# query formatted_address
# 1: Washington, USA Washington, DC, USA
# 2: Washington, USA Washington, UT, USA
# 3: Washington, USA Washington, VA 22747, USA
# 4: Washington, USA Washington, IA 52353, USA
# 5: Washington, USA Washington, GA 30673, USA
# 6: Washington, USA Washington, PA 15301, USA
# 7: Frankfurt, Germany Frankfurt, Germany