R - curl (not httr) POST request w/ JSON body - r

Let me start by saying that I understand how to do a POST request using "httr" and "crul" packages. I am working on developing an asynchronous method to sending multiple POST request with unique JSON body requests using the basic "curl" package. I have legitimate reasons for trying this with this package, but more importantly I'm just determined to get it to work. This may not be possible, or I may even be trying to wrong functions in "curl"...but wanted to see if anyone had any ideas.
I am trying to send a post request using curl_fetch_multi() as a POST request with a JSON in the body like this...
{
"configuration": {
"Id": 4507
},
"age": 0,
"zip": 32411,
"Date": "2020-12-23"
}
I have succeeded in at least getting getting error messages back form the API indicating an invalid body input using something along the lines of starting with an object containing each body i need to submit
library(curl)
library(jsonlite)
library(magrittr)
pool <- new_pool()
# results only available through call back function
cb <- function(req){cat("done:", req$url, ": HTTP:", req$status, "\n", "content:", rawToChar(req$content), "\n")}
# Create request for each body
for(i in 1:nrow(df)){
curl_fetch_multi(
"http://api.com/values?api_key=1234",
done = cb,
pool = pool,
handle = new_handle() %>%
handle_setopt(post = TRUE) %>%
handle_setheaders("Content-Type"="application/vnd.v1+json") %>%
handle_setform(body = df$body[[i]]) ###df$body[[i]] is a JSON string
)
}
# This actually performs requests
out <- multi_run(pool = pool)
done: http://api.com/values?api_key=1234 : HTTP: 400
content: {"errors":[{"code":"Service.input.invalid","message":"Invalid input"}]}
done: http://api.com/values?api_key=1234 : HTTP: 400
content: {"errors":[{"code":"Service.input.invalid","message":"Invalid input"}]}
....
I'm 90% positive it has to do with how it's attempting to call the JSON in handle_setform() setting of the handle. This is about where I am over my head and documentation is scarce.
Also, I am pretty sure the JSON is structured properly, as I can use them in other packages with no problem.
Any assistance would be greatly appreciated.

Found the solution!!
Needed to use following settings with handle_setopts()
for(i in 1:nrow(df)){
curl_fetch_multi(
"http://api.com/values?api_key=1234",
done = cb,
pool = pool,
handle = new_handle() %>%
handle_setheaders("Content-Type"="application/v1+json") %>%
handle_setopt(customrequest = "POST") %>%
handle_setopt(postfields = df$body[[i]]) #df$body is list of JSON
)
}
out <- multi_run(pool = pool)

Related

Why is the Service Unavailable Error using Lapply?

I am using the spotifyr library where I want to find audio features for multiple tracks. For example I can do this in order to find the audio features of a specific song using it's id.
analysis2 <- get_track_audio_features("2xLMifQCjDGFmkHkpNLD9h",
authorization = get_spotify_access_token())
Yesterday, I wrote this function below that takes all the tracks in a dataframe and finds the audio features for all of them and stores them in a list and it was working fine.
get_analysis <- function(track_id)
{
analysis <- get_track_audio_features(track_id,
authorization = get_spotify_access_token())
}
tracks_list <- lapply(all_tracks$track.id, get_analysis)
Now I am getting an error saying Request failed [503] and Error in get_track_audio_features(track_id, authorization = get_spotify_access_token()) : Service Unavailable (HTTP 503).
I am still able to find the audio features of a specific song so I am not sure which service is unavailable.
I suspect you are reaching a song in your data for which the response is denied from spotify. You could try adding an error-catching mechanism to see which one it is:
get_analysis <- function(track_id){
tryCatch(
expr = {
get_track_audio_features(track_id, authorization = get_spotify_access_token())
},
error = function(e){
print(track_id)
}) -> analysis
return(analysis)
}
tracks_list <- lapply(all_tracks$track.id, get_analysis)
I looked at the source code for the package and didn't see any sneaky rate-limiting issues and the Web API page shows error 503 as a generic error that needs waiting to be resolved (https://developer.spotify.com/documentation/web-api/). Thus you could also try just adding a 10 minute wait (I couldn't find how long exactly it is on Spotify's website):
get_analysis <- function(track_id){
tryCatch(
expr = {
get_track_audio_features(track_id, authorization = get_spotify_access_token()) -> output
return(output)
},
error = function(e){
print(track_id)
return(e)
}) -> output
}
wait.function <- funciton(){
Sys.sleep(600)
}
get_analysis_master <- function(all_tracks){
k <- 1
tracks_list <- list()
for(track.id in all_tracks$track.id){
get_analysis(track.id) -> output
if(!inherits(output, "error")){
tracks_list[[k]] <- output
k <- k + 1
} else {
wait.function()
}
return(tracks_list)
}
get_analysis_master(all_tracks) -> tracks_list

R - Translate GET requests in Postman with body raw to GET requests in R

I have a question about to add body to GET request.
In Postman, I easily add body raw json to tab Body
In R, I use httr::GETand I can not find any option to add them. I try to use option query, but it return error 500 "Missing parameter username"
This is my code (sorry about fake data):
library(httr)
api <- ".........................../users/authorize"
query <- list("username": "xxxxx", "password": "xxxxxxxxxxxx")
resp <- GET(api, query = query)
stop_for_status(resp)
# Error: Internal Server Error (HTTP 500).
content(resp, type = "text", encoding = "utf-8")
# [1] "{\"status\":\"0\",\"error_code\":\"S002\",\"errors\":\"Missing param [username].\"}"
Can anybody help me to this case? Thank you so much.
Try this:
resp <- httr::POST(body = query, encode = "json")

Updating Google Tag Manager container via API using R: invalidArgument error

I am trying to write an R script to programmatically update a Google Tag Manager container via API and I have hit a bit of a wall getting it to work, as it keeps returning an invalid argument error. The problem is that I can't quite figure out what the problem is.
The documentation for the API call is here:
https://developers.google.com/tag-manager/api/v2/reference/accounts/containers/update
Here's the code:
library(httr)
url_base <- 'https://www.googleapis.com/tagmanager/v2'
url_path <- paste('accounts',account_id,'containers',container_id,sep='/')
api_url <- paste(url_base,url_path,sep='/')
#since the instructions indicate that the request body parameters are all optional, let's just send a new name
call <- PUT(api_url,
add_headers(Authorization = paste("Bearer", gtm_token$credentials$access_token)),
encode = 'json',
body = list(name = 'new name'))
call_content <- content(call,'parsed')
This is a pretty standard API call to the GTM API, and in fact I have written a bunch of functions for other GTM API methods that work in the same way, so I am a bit perplexed as to why this one keeps failing:
$error
$error$errors
$error$errors[[1]]
$error$errors[[1]]$domain
[1] "global"
$error$errors[[1]]$reason
[1] "invalidArgument"
$error$errors[[1]]$message
[1] "Bad Request"
$error$code
[1] 400
$error$message
[1] "Bad Request"
It seems like the issue is in the message body, but it's not clear if the issue is down to the API expecting different information / more parameters, when the documentation suggests that all of the parameters are optional.
OK, so the documentation is lacking here. This works if you include a name at least. Here's a working function:
gtm_containers_update <- function(account_id,container_id,container_name,usage_context,domain_name,notes,token) {
require(httr)
token$refresh()
#create the post url
api_url <- paste('https://www.googleapis.com/tagmanager/v2','accounts',account_id,'containers',container_id,sep='/')
#create the list with required components
call_body <- list(name = container_name,
usageContext = list(usage_context),
notes = notes,
domainName = domain_name)
call <- POST(url,
add_headers(Authorization = paste("Bearer", token$credentials$access_token)),
encode = 'json',
body = call_body)
print(paste('Status code:',call$status_code))
}

Failing HTTP request to Google Sheets API using googleAuthR, httr, and jsonlite

I'm attempting to request data from a google spreadsheet using the googleAuthR. I need to use this library instead of Jenny Bryan's googlesheets because the request is part of a shiny app with multiple user authentication. When the request range does not contain spaces (e.g. "Sheet1!A:B" the request succeeds. However, when the tab name contains spaces (e.g. "'Sheet 1'!A:B" or "\'Sheet 1\'!A:B", the request fails and throws this error:
Request Status Code: 400
Error : lexical error: invalid char in json text.
<!DOCTYPE html> <html lang=en>
(right here) ------^
Mark Edmondson's googleAuthR uses jsonlite for parsing JSON. I assume this error is coming from jsonlite, but I'm at a loss for how to fix it. Here is a minimal example to recreate the issue:
library(googleAuthR)
# scopes
options("googleAuthR.scopes.selected" = "https://www.googleapis.com/auth/spreadsheets.readonly")
# client id and secret
options("googleAuthR.client_id" = "XXXX")
options("googleAuthR.client_secret" = "XXXX")
# request
get_data <- function(spreadsheetId, range) {
l <- googleAuthR::gar_api_generator(
baseURI = "https://sheets.googleapis.com/v4/",
http_header = 'GET',
path_args = list(spreadsheets = spreadsheetId,
values = range),
pars_args = list(majorDimension = 'ROWS',
valueRenderOption = 'UNFORMATTED_VALUE'),
data_parse_function = function(x) x)
req <- l()
req
}
# authenticate
gar_auth(new_user = TRUE)
# input
spreadsheet_id <- "XXXX"
range <- "'Sheet 1'!A:B"
# get data
df <- get_data(spreadsheet_id, range)
How should I format range variable for the request to work? Thanks in advance for the help.
Use URLencode() to percent-encode spaces.
Details:
Using options(googleAuthR.verbose = 1) shows that the GET request was of the form:
GET /v4/spreadsheets/.../values/'Sheet 1'!A:B?majorDimension=ROWS&valueRenderOption=UNFORMATTED_VALUE HTTP/1.1
I had assumed the space would be encoded, but I guess not. In this github issue from August 2016, Mark states URLencode() was going to be the default for later versions of googleAuthR. Not sure if that will still happen, but it's an easy fix in the meantime.

Asynchronous POST Requests - R, using RCurl?

I am trying to make async requests to a REST API from R. The below curl command illustrates the parameters that I need to the pass to the api. I'm giving you guys the linux curl command as I'm hoping that will make it clear:
curl -v -X POST https://app.example.com/api/ \
-H 'Authorization: somepwd' \
-H "Content-Type: application/json" \
-d {key1: value1, key2: value2}
Right now, I'm accomplishing the same thing in R by executing the following:
library(httr)
library(jsonlite)
content(POST('https://app.example.com/api/'
,add_headers(Authorization = 'somepwd')
,body = toJSON(rDataFrame)
,content_type_json()
)
)
The goal is to submit the above POST request from R but to vary the json string that is sent in the body, and do that asynchronously.
I have been searching for packages that will help me make asynchronous requests rather than making requests serially. The closest thing I could find is the getURIAsynchronous() function from the RCurl package (https://cran.r-project.org/web/packages/RCurl/RCurl.pdf) but do not understand how to submit a PUT request with headers and a body using their function. I would really like to make the above POST request from R but asynchronously where the URI is the same, but the data sent is different for each request.
I found this http://www.omegahat.org/RCurl/concurrent.html
getURIs =
function(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)
{
content = list()
curls = list()
for(i in uris) {
curl = getCurlHandle()
content[[i]] = basicTextGatherer()
opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)
curlSetOpt(.opts = opts, curl = curl)
multiHandle = push(multiHandle, curl)
}
if(.perform) {
complete(multiHandle)
lapply(content, function(x) x$value())
} else {
return(list(multiHandle = multiHandle, content = content))
}
}
My idea is that I could replace for (i in uris) with for(i in jsons) where I am looping over the different data that I want to send to the same URL, however I am having trouble understanding the following concepts from the RCurl Package:
How do I pass a header as part of a PUT request. How do I pass data in the body of the request? This was pretty straight forward using the httr package as I have illustrated above.
I tried passing in the header in the curl options and alternatively the header. The thing is I don't understand where to pass the the component parts of the post request: authentication, header, and body within the getURIAsynchronous() function, or any of the resources I have described above.
Does anyone know how to accomplish this? An example would be incredibly helpful.
The curl package has been recently updated to handle async requests (see here)
Using the curl, magrittr and jsonlite packages you can create asynchronous post requests by:
Creating a generic handle with your header and body content using the handle_setform function
Writing a call back function to retrieve your results
Initializing a pool and adding your concurrent requests to it
Running your pool via multi_run
Example code is below:
library(curl)
library(jsonlite)
library(magrittr)
#create a handle object
h <- new_handle() %>%
handle_setheaders(Authorization = "somepwd",
"Content-Type" = "application/json") %>%
handle_setform(body = toJSON(iris))
pool <- new_pool()
# results only available through call back function
cb <- function(req){cat("done:", req$url, ": HTTP:", req$status, "\n", "content:", rawToChar(req$content), "\n")}
# example vector of uris to loop through
uris <- c("https://app.example.com/api/endpoint1"
,"https://app.example.com/api/endpoint2"
,"https://app.example.com/api/endpoint3")
# all scheduled requests are performed concurrently
sapply(uris, curl_fetch_multi, done=cb, pool=pool)
# This actually performs requests
out <- multi_run(pool = pool)

Resources