I am trying to scrape some web data using the API that I can see is being called by viewing the Safari Network Tab.
Either the API doesn't seem to get the form parameters correctly if passed as json or I get an error from R if I try to pass them as URLEncoded. I can't see what am I doing wrong? I suspect part of the problem is that my form is a list containing a list.
Request Data as shown in Safari Network Tab
MIME Type: application/x-www-form-urlencoded; charset=UTF-8
method: POST
section[]: 1
section[]: 4
period[]: 20170501
HTTR Post to mimic the above
form <- list(
section = list(1,4),
period = 20170501
)
resp<-POST(URL, body=form, encode="json", verbose())
Then the code runs without error and the API does return results but seems to have ignored the specific parameters.
The output from verbose suggests the parameters are being included:
{"section":[1,4],"period":20170501}
Adjustment for Form Type
I can see the above is not using the correct form type, so I change encode to "form" to so that the form is sent as x-www-form-urlencoded. However, I then get the following error.
Error in vapply(elements, encode, character(1)) :
values must be length 1,
but FUN(X[[1]]) result is length 2
Fixed! I had to use Query instead of Body and add the [] after each item.
query <- list(
"section[]" = 1,
"section[]" = 4,
"period[]" = 20170501
)
resp<-POST(URL, query=query, verbose())
Related
I can retrieve EPPO DB info from GET requests.
I am looking for help to retrieve the info from POST requests.
Example code and other info in the linked Rmarkdown HTMP output
As suggested, I have gone trough the https://httr.r-lib.org/ site.
Interesting. I followed the links to https://httr.r-lib.org/articles/api-packages.html and then to https://cdn.zapier.com/storage/learn_ebooks/e06a35cfcf092ec6dd22670383d9fd12.pdf.
I suppose that the arguments for the POST() function should be (more or less) as follows, but yet the response is always 404
url = "https://data.eppo.int/api/rest/1.0/"
config = list(authtoken=my_authtoken)
body = list(intext = "Fraxinus anomala|Tobacco ringspot virus|invalide name|Sequoiadendron giganteum")
encode = "json"
#handle = ???
Created on 2021-04-26 by the reprex package (v0.3.0)
How do I find the missing pieces?
It is a little bit tricky:
You need to use correct url with name of the service from https://data.eppo.int/documentation/rest, e.g. to use Search preferred names from EPPOCodes list:
url = "https://data.eppo.int/api/rest/1.0/tools/codes2prefnames"
Authorization should be passed to body:
body = list(authtoken = "yourtoken", intext = "BEMITA")
So, if you want to check names for two eppocodes: XYLEFA and BEMITA the code should look like:
httr::POST(
url = "https://data.eppo.int/api/rest/1.0/tools/codes2prefnames",
body = list(authtoken = "yourtoken", intext = "BEMITA|XYLEFA")
)
Nonetheless, I would also recommend you to just use the pestr package. However, to find eppocodes it uses SQLite database under the hood instead of EPPO REST API. Since the db is not big itself (circa 40MB) this shouldn't be an issue.
I found the easy way following a suggestion in the DataCamp course:
"To find an R client for an API service search 'CRAN '"
I found the 'pestr' package that gives great access to EPPO database.
I still do not know how to use the POST() function myself. Any hint on that side is warmly welcome.
Here is a solution to loop over names to get EPPO-codes. Whit minor adjustments this also works for distribution and other information in the EPPO DB
# vector of species names
host_name <- c("Fraxinus anomala", "Tobacco ringspot virus", "invalide name",
"Sequoiadendron giganteum")
EPPO_key <- "enter personal key"
# EPPO url
path_eppo_code <- "https://data.eppo.int/api/rest/1.0/tools/names2codes"
# epty list
my_list <- list()
for (i in host_name) {
# POST request on eppo database
response <- httr::POST(path_eppo_code, body=list(authtoken=EPPO_key,
intext=i))
# get EPPO code
pest_eppo_code <- strsplit(httr::content(response)[[1]], ";")[[1]][2]
# add to list
my_list[[i]] <- pest_eppo_code
}
# list to data frame
data <- plyr::ldply(my_list)
Im attempting to setup some R code to create a new work item task in Azure Devops. Im okay with a mostly empty work item to start with if thats okay to do (my example code is only trying to create a work item with a title).
I receive a 203 response but the work item doesn't appear in Devops.
Ive been following this documentation from Microsoft, I suspect that I might be formatting the body incorrectly.
https://learn.microsoft.com/en-us/rest/api/azure/devops/wit/work%20items/create?view=azure-devops-rest-5.1
Ive tried updating different fields and formatting the body differently with no success. I have attempted to create either a bug or feature work item but both return the same 203 response.
To validate that my token is working I can GET work item data by ID but the POST continues to return a 203.
require(httr)
require(jsonlite)
url <- 'https://dev.azure.com/{organization}/{project}/_apis/wit/workitems/$bug?api-version=5.1'
headers = c(
'Authorization' = sprintf('basic %s',token),
'Content-Type' = 'application/json-patch+json',
'Host' = 'dev.azure.com'
)
data <- toJSON(list('body'= list("op"= "add",
"path"= "/fields/System.AreaPath",
"value"= "Sample task")), auto_unbox = TRUE, pretty = TRUE)
res <- httr::POST(url,
httr::add_headers(.headers=headers),
httr::verbose(),
body = data)
Im expecting a 200 response (similar to the example in the link above) and a work item task in Azure DevOps Services when I navigate to the website.
Im not the best with R so please be detailed. Thank you in advanced!
The POST continues to return a 203.
The HTTP response code 203 means Non-Authoritative Information, it should caused by your token format is converted incorrectly.
If you wish to provide the personal access token through an HTTP
header, you must first convert it to a Base64 string.
Refer to this doc described, if you want to use VSTS rest api, you must convert your token to a Base64 string. But in your script, you did not have this script to achieve this convert.
So, please try with the follow script to convert the token to make the key conformant with the requirements(load the base64enc package first):
require(base64enc)
key <- token
keys <- charToRaw(paste0(key,":token"))
auth <- paste0("Basic ",base64encode(keys))
Hope this help you get 200 response code
I know this question is fairly old, but I cannot seem to find a good solution posted yet. So, I will add my solution in case others find themselves in this situation. Note, this did take some reading through other SO posts and trial-and-error.
Mengdi is correct that you do need to convert your token to a Base64 string.
Additionally, Daniel from this SO question pointed out that:
In my experience with doing this via other similar mechanisms, you have to include a leading colon on the PAT, before base64 encoding.
Mengdi came up big in another SO solution
Please try with adding [{ }] outside your request body.
From there, I just made slight modifications to your headers and data objects. Removed 'body' from your json, and made use of paste to add square brackets as well. I found that the Rcurl package made base64 encoding a breeze. Then I was able to successfully create a blank ticket (just titled) using the API! Hope this helps someone!
library(httr)
library(jsonlite)
library(RCurl)
#user and PAT for api
userid <- ''
token= 'whateveryourtokenis'
url <- 'https://dev.azure.com/{organization}/{project}/_apis/wit/workitems/$bug?api-version=5.1'
#create a combined user/pat
#user id can actually be a blank string
#RCurl's base64 seemed to work well
c_id <- RCurl::base64(txt = paste0(userid,
":",
devops_pat
),
mode = "character"
)
#headers for api call
headers <- c(
"Authorization" = paste("Basic",
c_id,
sep = " "
),
'Content-Type' = 'application/json-patch+json',
'Host' = 'dev.azure.com'
)
#body
data <- paste0("[",
toJSON(list( "op"= "add",
"path"= "/fields/System.Title",
"value"= "API test - please ignore"),
auto_unbox = TRUE,
pretty = TRUE
),
"]"
)
#make the call
res <- httr::POST(url,
httr::add_headers(.headers=headers),
httr::verbose(),
body = data
)
#check status
status <- res$status_code
#check content of response
check <- content(res)
I am compiling data about a set of artists on Spotify - data for each song on each album of each artist. I use a for loop to automate this API request on about 80 different artists in the data frame albums, then assign a bit of info on each album in albums to its list object from the API.
The problem: my API call doesn't always return a list object. Sometimes it returns an object where class() = raw.
#REQUEST DATA
#------------
library(plyr)
library(httr)
library(lubridate)
collablist <- as.list(NULL)
for(i in 1:nrow(albums)){
tracks_in_one_album <- as.list(NULL)
URI = paste0('https://api.spotify.com/v1/albums/', albums$album_uri[i], '/tracks')
response = GET(url = URI, add_headers(Authorization = HeaderValue))
tracks_in_one_album = content(response)
tracks_in_one_album[["album"]] = albums$album_name[i]
tracks_in_one_album[["album_artist"]] = albums$artists[i]
collablist[[i]] <- tracks_in_one_album
print(albums$artist_name[i])
}
The loop runs for somewhere between 50 and 300 albums before I inevitably get the following message:
Error in tracks_in_one_album[["album"]] <- albums$album_name[i] :
incompatible types (from character to raw) in subassignment type fix
When I assign attempting to assign the character object albums$album_name[i] to the API requested object tracks_in_one_album when it's a list, I have no issue. But occasionally the object is of a raw class. Changing it to a list by encapsulating the content() call with as.list prevents the error from occurring, but it doesn't really fix the issue because for the requests where the data come in as raw instead of as a list by default, they're sort of mangled (just a vector with .
The craziest part? This doesn't happen consistently. It could happen for the 4th album of Cat Stevens one time; if I rerun, that Cat Stevens album will be fine and get pulled into R as a list but perhaps the second album for Migos will come in raw instead.
My Question - why are the data not always coming in as a list when I make a request? How is it possible that this could be happening in such a non-reproducible way?
I am looking to scrape the web pages of several golfers on wikipedia. Some of the player names are identical to other people and, in that case, the URL has to have the text '(golfer)' appended to reach the right page
I am looking to try every player with this addition and if the page does not exist then revert to the plain name
There may be a better approach, but I had hoped that I could obtain the content size of the response. If it did not reach a certain level e.g. 2kb then that was not a valid page
library(httr)
base_url <- "https://en.wikipedia.org/w/api.php"
query_params <- list(action = "parse",
page = "Patrick Reed_(golfer)",
format = "xml")
resp <- GET(url = base_url, query = query_params)
resp
Response [https://en.wikipedia.org/w/api.php?
action=parse&page=Patrick%20Reed_%28golfer%29&format=xml]
Date: 2018-04-08 08:35
Status: 200
Content-Type: text/xml; charset=utf-8
Size: 402 B
So a suitably low size gets reported but I am not sure how to get to it when I expand the list, resp. According to httr quickstart there is something in the headers referencing content-length but I am unable to discover it
TIA
I tried to get data from a netatmo station I have access to via API. The following code I used in R.
myapp <- oauth_app("my_netatmo",key="my_netatmo_client_id",secret="my_netatmo_client_secret")
ep <- oauth_endpoint(authorize = "http://api.netatmo.net/oauth2/authorize",access = "http://api.netatmo.net/oauth2/token")
sig_tok <- oauth2.0_token(ep,myapp, scope="read_station")
#after that I get redirected to my browser to log in and after that sig_tok contains an access token
sig <- config(token = sig_tok)
html_get<-GET("http://api.netatmo.net/api/devicelist",sig)
html_get contains this:
html_get
Response [http://api.netatmo.net/api/devicelist]
Status: 400
Content-type: application/json; charset=utf-8
{"error":{"code":1,"message":"Access token is missing"}}
What am I doing wrong since "sig" seems to contain a token:
sig$token$credentials$access_token
[1] "5**********************f|3**********************a"
There are two tokens or am I wrong (because of "|" in between)?
The '|' inside the access_token is part of it, it is only one access token.
From the documentation: http://dev.netatmo.com/doc/methods/devicelist,
the parameter name is: "access_token". I don't know the R language, but it seems you are sending "token" as a parameter, and not "access_token". It may explain the issue.