I try to do geocoding of French addresses. I'd like to use the following website : http://adresse.data.gouv.fr/
There is an example on this website on how is working the API but I think it's some Linux code and I'd like to translate in R code. The aim is to give a csv file with addresses and the result should be geo coordinates.
Linux code (example give on the website)
http --timeout 600 -f POST http://api-adresse.data.gouv.fr/search/csv/ data#path/to/file.csv
I tried to "translate" this in R with the following code
library(httr)
library(RCurl)
queryResults=POST("http://api-adresse.data.gouv.fr/search/csv/",body=list(data=fileUpload("file.csv")))
result_geocodage=content(queryResults)
But unfortunately I have a bad request error.
Does somebody knows what I'm missing in the translation to R?
Thanks!
Here's an example. First, some example data plus the request:
library(httr)
df <- data.frame(c("13 Boulevard Chanzy", "Gloucester St"),
c("93100 Montreuil", "Jersey"))
write.csv2(df, tf <- tempfile(fileext = ".csv"))
res <- POST("http://api-adresse.data.gouv.fr/search/csv/",
timeout(600),
body = list(data = upload_file(tf)))
Then, the result:
content(res, sep = ";", row.names = 1)
# c..13.Boulevard.Chanzy....Gloucester.St.. c..93100.Montreuil....Jersey.. latitude longitude
# 1 13 Boulevard Chanzy 93100 Montreuil 48.85825 2.434462
# 2 Gloucester St Jersey 49.46712 1.145554
# result_label result_score result_type result_id result_housenumber
# 1 13 Boulevard Chanzy 93100 Montreuil 0.88 housenumber ADRNIVX_0000000268334929 13
# 2 2 Résidence le Jersey 76160 Saint-Martin-du-Vivier 0.24 housenumber ADRNIVX_0000000311480901 2
# result_name result_street result_postcode result_city result_context result_citycode
# 1 Boulevard Chanzy NA 93100 Montreuil 93, Seine-Saint-Denis, Île-de-France 93048
# 2 Résidence le Jersey NA 76160 Saint-Martin-du-Vivier 76, Seine-Maritime, Haute-Normandie 76617
Or, just the coordinates:
subset(content(res, sep = ";", row.names = 1, check.names = FALSE), select = c("latitude", "longitude"))
# latitude longitude
# 1 48.85825 2.434462
# 2 49.46712 1.145554
Related
I am trying to use R to retrieve data from the YouTube API v3 and there are few/no tutorials out there that show the basic process. I have figured out this much so far:
# Youtube API query
base_url <- "https://youtube.googleapis.com/youtube/v3/"
my_yt_search <- function(search_term, max_results = 20) {
my_api_url <- str_c(base_url, "search?part=snippet&", "maxResults=", max_results, "&", "q=", search_term, "&key=",
my_api_key, sep = "")
result <- GET(my_api_url)
return(result)
}
my_yt_search(search_term = "salmon")
But I am just getting some general meta-data and not the search results. Help?
PS. I know there is a package 'tuber' out there but I found it very unstable and I just need to perform simple searches so I prefer to code the requests myself.
Sadly there is no way to directly get the durations, you'll need to call the videos endpoint (with the part set to part=contentDetails) after doing the search if you want to get those infos, however you can pass as much as 50 ids in a single call thus we can save some time by pasting all the ids together.
library(httr)
library(jsonlite)
library(tidyverse)
my_yt_duration <- function(...){
my_api_url <- paste0(base_url, "videos?part=contentDetails", paste0("&id=", ..., collapse=""), "&key=",
my_api_key )
GET(my_api_url) -> resp
fromJSON(content(resp, "text"))$items %>% as_tibble %>% select(id, contentDetails) -> tb
tb$contentDetails$duration %>% tibble(id=tb$id, duration=.)
}
### getting the video IDs
my_yt_search(search_term = "salmon")->res
## Converting from JSON then selecting all the video ids
# fromJSON(content(res,as="text") )$items$id$videoId
my_yt_duration(fromJSON(content(res,as="text") )$items$id$videoId) -> tib.id.duration
# A tibble: 20 x 2
id duration
<chr> <chr>
1 -x2E7T3-r7k PT4M14S
2 b0ahREpQqsM PT3M35S
3 ROz8898B3dU PT14M17S
4 jD9VJ92xyzA PT5M42S
5 ACfeJuZuyxY PT3M1S
6 bSOd8r4wjec PT6M29S
7 522BBAsijU0 PT10M51S
8 1P55j9ub4es PT14M59S
9 da8JtU1YAyc PT3M4S
10 4MpYuaJsvRw PT8M27S
11 _NbbtnXkL-k PT2M53S
12 3q1JN_3s3gw PT6M17S
13 7A-4-S_k_rk PT9M37S
14 txKUTx5fNbg PT10M2S
15 TSSPDwAQLXs PT3M11S
16 NOHEZSVzpT8 PT7M51S
17 4rTMdQzsm6U PT17M24S
18 V9eeg8d9XEg PT10M35S
19 K4TWAvZPURg PT3M3S
20 rR9wq5uN_q8 PT4M53S
I previously figured out how to convert lattitude/longitude to county FIPS codes using the FCC API (Apply an API Function over 2 columns of Dataframe, Output a Third Column) thanks to #caldwellst and #rohit. Unfortunately, the FCC modified the API and I can't figure out how to fix the code to work again.
Here is a link to the new API: https://geo.fcc.gov/api/census/
Here is my dataframe:
> head(df_coords)
# A tibble: 6 x 3
lon lat censusYear
<dbl> <dbl> <dbl>
1 -112. 33.4 2010
2 -73.2 44.5 2010
3 -88.2 41.9 2010
4 -88.2 41.9 2010
5 -88.4 41.9 2010
6 -77.1 39.0 2010
Here is the function I previously borrowed / adapted as well as the command to run it:
geo2fips <- function(latitude, longitude) {
url <- "https://geo.fcc.gov/api/census/block/find?format=json&latitude=%f&longitude=%f"
url <- sprintf(url, latitude, longitude)
json <- RCurl::getURL(url)
json <- RJSONIO::fromJSON(json)
as.character(json$County['FIPS'])
}
df_fips$county_fips <- mapply(geo2fips, df_fips$lat, df_fips$lon)
And here is the error message I get when I run it:
Error in function (type, msg, asError = TRUE) :
Unknown SSL protocol error in connection to geo.fcc.gov:443
Can anyone help me figure this out? I figured it may be related to a requirement for census year, so I tried to modify the code as follows but it returned the same error message:
geo2fips <- function(latitude, longitude, censusYear) {
+ url <- "https://geo.fcc.gov/api/census/block/find?format=json&latitude=%f&longitude=%f&censusYear=%f"
+ url <- sprintf(url, latitude, longitude, censusYear)
+ json <- RCurl::getURL(url)
+ json <- RJSONIO::fromJSON(json)
+ as.character(json$County['FIPS'])
+ }
> df_coords$county_fips <- mapply(geo2fips, df_coords$lat, df_coords$lon, df_coords$censusYear)
Error in function (type, msg, asError = TRUE) :
Unknown SSL protocol error in connection to geo.fcc.gov:443
>
Huge thank you to anyone who can help. -Mike
There's been a slight change to the URL and parameters - you can use:
geo2fips <- function(latitude, longitude) {
url <- "https://geo.fcc.gov/api/census/area?lat=%f&lon=%f&format=json"
res <- jsonlite::fromJSON(sprintf(url, latitude, longitude))[["results"]][["county_fips"]]
unique(res)
}
You can also simplify things a little if you use the jsonlite package instead of RSJONIO as the former accepts connections directly.
I am not able to access a function in a package of R. The function name is get_league_listing and the package name is Rdota2. It displays the following error:
Error in (function (..., deparse.level = 1, make.row.names = TRUE, stringsAsFactors = default.stringsAsFactors()) :
numbers of columns of arguments do not match
This was the command used:
get_league_listing(dota_id = 570, language = "en", key = NULL)
Please help me out.
This used to return a data.frame, but Valve decided that one of the leagues should not have a description field, which makes the rbind part of the code (as #r2evans correctly mentions in the comments) below fail (code from get_league_listing), due to the different number of columns:
args <- list(key = key, language = 'en')
dota_result <- get_response(570, 'GetLeagueListing', 'IDOTA2Match', 1, args)
#the below fails
dota_result$content <-
do.call(rbind.data.frame, c(dota_result$content[[1]][[1]], stringsAsFactors = FALSE))
Error in (function (..., deparse.level = 1, make.row.names = TRUE, stringsAsFactors = default.stringsAsFactors()) :
numbers of columns of arguments do not match
This is the culprit btw (there should be a description element here in order for rbind to work):
> dota_result$content[[1]][[1]][[1651]]
$name
[1] "The International 2017 Open Qualifiers"
$leagueid
[1] 5498
$tournament_url
[1] "http://www.dota2.com"
$itemdef
[1] 17377
The solution, until I update the package, is to use the response and then probably data.table to convert to a data.frame / data.table:
library(data.table)
#if your key is stored using the method described at the vignette
#leave it as NULL, otherwise add your key.
args <- list(key = key, language = 'en')
dota_result <- get_response(570, 'GetLeagueListing', 'IDOTA2Match', 1, args)
rbindlist(dota_result$content[[1]][[1]], fill = TRUE, use.names = TRUE)
which will return:
name leagueid
1: Dota 2 Just For Fun 1212
2: joinDOTA League Season 3 1640
3: Killing Spree: North America 25
4: Wild Cards West 2
5: Wild Cards East 3
---
1683: GESC: Thailand 9663
1684: Dota 2 Asia Championships 2018 9643
1685: ESL One Genting 2018 8093
1686: Corsair DreamLeague season 9 9683
1687: StarLadder ImbaTV Invitational Season 5 9908
description
1: 64 of the best Brazilian amateur teams compete to become the winner of the first Dota 2 Just For Fun tournament.
2: The global Dota 2 league for everyone. Featured are all of the matches from division 1 and 2. There are three leagues: Europe, America, and Asia. The top 10 from each league meet in the premier division. $1.00 from each ticket purchased will go directly into the prizepool.
I have seen many posts on here about using read.xls with a url and they all worked on my Mac, but now when I am trying to use the code on my Windows computer, it is not working. I used the below code on my Mac:
tmp <- tempfile()
download.file("https://www.spdrs.com/site-content/xls/SPY_All_Holdings.xls?fund=SPY&docname=All+Holdings&onyx_code1=1286&onyx_code2=1700", destfile = tmp, method = "curl")
SPY <- read.xls(tmp, skip=3)
unlink(tmp)
Using "curl" no longer woks ("had status 127" is the warning message) and when I try "internal" or "wininet", it says " formal argument "method" matched by multiple actual arguments". When I try read.xls, it says the file is "missing" and "invalid". I have downloaded Perl, Java, gdata, Rcurl and the "downloader" package (because I heard that works better with https) and could use that instead....Is there something else I would have to do on a Windows computer to make this code work?
Thanks!
> library(RCurl)
> library(XLConnect)
> URL <- "https://www.spdrs.com/site-content/xls/SPY_All_Holdings.xls?fund=SPY&docname=All+Holdings&onyx_code1=1286&onyx_code2=1700"
> f = CFILE("SPY_All_Holdings.xls", mode="wb")
> curlPerform(url = URL, writedata = f#ref, ssl.verifypeer = FALSE)
# OK
# 0
> close(f)
# An object of class "CFILE"
# Slot "ref":
# <pointer: (nil)>
> out <- readWorksheetFromFile(file = "SPY_All_Holdings.xls",sheet="SPY_All_Holdings")
> head(out)
# Fund.Name. SPDR..S.P.500..ETF Col3 Col4 Col5
# 1 Ticker Symbol: SPY <NA> <NA> <NA>
# 2 Holdings: As of 06/06/2016 <NA> <NA> <NA>
# 3 Name Identifier Weight Sector Shares Held
# 4 Apple Inc. AAPL 2.945380 Information Technology 54545070.000
# 5 Microsoft Corporation MSFT 2.220684 Information Technology 77807630.000
# 6 Exxon Mobil Corporation XOM 1.998224 Energy 40852760.000
I would like to create an interactive map showing the public transport lines of a city. I am trying to do this using Leaflet in R (but I'm open to alternatives, suggestions?)
Data: The data of the transport system is in GTFS format, organized in text files (.txt), which I read into R as a data frame.*
The Problem: I cannot find how to indicate the id of each Poly line (variable shape_id) so the plot would actually follow the route of each transit line. Instead, it is connecting the dots in a random sequence.
Here is what I've tried, so far without success:
# Download GTFS data of the Victoria Regional Transit System
tf <- tempfile()
td <- tempdir()
ftp.path <- "http://www.gtfs-data-exchange.com/agency/bc-transit-victoria-regional-transit-system/latest.zip"
download.file(ftp.path, tf)
# Read text file to a data frame
zipfile <- unzip( tf , exdir = td )
shape <- read.csv(zipfile[9])
# Create base map
basemap <- leaflet() %>% addTiles()
# Add transit layer
basemap %>% addPolylines(lng=shape$shape_pt_lon, lat=shape$shape_pt_lat,
fill = FALSE,
layerId =shape$shape_id)
I would be glad to have your comments on this.
*I know it is possible to import this data into a GIS software (e.g. QGIS) to create a shapefile and then read the shapefile into R with readOGR. Robin Lovelace has shown how to do this. BUT, I am looking for a pure R solution. ;)
ps. Kyle Walker has written a great intro to interactive maps in R using Leaflet. Unfortunately, he doesn't cover poly lines in his tutorial.
Your problem is not one of method but of data: note that you download 8 MB and that the line file you try to load into Leaflet via shiny is 5 MB. As a general principle, you should always try new methods with tiny datasets first, before scaling them up. This is what I do below to diagnose the problem and solve it.
Stage 1: Explore and subset the data
pkgs <- c("leaflet", "shiny" # packages we'll use
, "maps" # to test antiquated 'maps' data type
, "maptools" # to convert 'maps' data type to Spatial* data
)
lapply(pkgs, "library", character.only = TRUE)
class(shape)
## [1] "data.frame"
head(shape)
## shape_id shape_pt_lon shape_pt_lat shape_pt_sequence
## 1 1-39-220 -123.4194 48.49065 0
## 2 1-39-220 -123.4195 48.49083 1
## 3 1-39-220 -123.4195 48.49088 2
## 4 1-39-220 -123.4196 48.49123 3
## 5 1-39-220 -123.4197 48.49160 4
## 6 1-39-220 -123.4196 48.49209 5
object.size(shape) / 1000000 # 5 MB!!!
## 5.538232 bytes
summary(shape$shape_id)
shape$shape_id <- as.character(shape$shape_id)
ids <- unique(shape$shape_id)
shape_orig <- shape
shape <- shape[shape$shape_id == ids[1],] # subset the data
Stage 2: Convert to a Spatial* object
Is this like the data.frame objects from maps?
state.map <- map("state", plot = FALSE, fill = TRUE)
str(state.map)
## List of 4
## $ x : num [1:15599] -87.5 -87.5 -87.5 -87.5 -87.6 ...
## $ y : num [1:15599] 30.4 30.4 30.4 30.3 30.3 ...
## $ range: num [1:4] -124.7 -67 25.1 49.4
## $ names: chr [1:63] "alabama" "arizona" "arkansas" "california" ...
## - attr(*, "class")= chr "map"
Yes, it's similar, so we can use map2Spatial* to convert it:
shape_map <- list(x = shape$shape_pt_lon, y = shape$shape_pt_lat)
shape_lines <- map2SpatialLines(shape_map, IDs = ids[1])
plot(shape_lines) # success - this plots a single line!
Stage 3: Join all the lines together
A for loop will do this nicely. Note we only use the first 10 lines. Use 2:length(ids) for all lines:
for(i in 2:10){
shape <- shape_orig[shape_orig$shape_id == ids[i],]
shape_map <- list(x = shape$shape_pt_lon, y = shape$shape_pt_lat)
shape_temp <- map2SpatialLines(shape_map, IDs = ids[i])
shape_lines <- spRbind(shape_lines, shape_temp)
}
Stage 4: Plot
Using the SpatialLines object makes the code a little shorter - this will plot the first 10 lines in this case:
leaflet() %>%
addTiles() %>%
addPolylines(data = shape_lines)
Conclusion
You needed to play around with the data and manipulate it before converting it into a Spatial* data type for plotting, with the correct IDs. maptools::map2Spatial*, unique() and a clever for loop can solve the problem.