Asynchronous POST Requests - R, using RCurl? - r

Question

I am trying to make async requests to a REST API from R. The below curl command illustrates the parameters that I need to the pass to the api. I'm giving you guys the linux curl command as I'm hoping that will make it clear:
curl -v -X POST https://app.example.com/api/ \
-H 'Authorization: somepwd' \
-H "Content-Type: application/json" \
-d {key1: value1, key2: value2}
Right now, I'm accomplishing the same thing in R by executing the following:
library(httr)
library(jsonlite)
content(POST('https://app.example.com/api/'
,add_headers(Authorization = 'somepwd')
,body = toJSON(rDataFrame)
,content_type_json()
)
)
The goal is to submit the above POST request from R but to vary the json string that is sent in the body, and do that asynchronously.
I have been searching for packages that will help me make asynchronous requests rather than making requests serially. The closest thing I could find is the getURIAsynchronous() function from the RCurl package (https://cran.r-project.org/web/packages/RCurl/RCurl.pdf) but do not understand how to submit a PUT request with headers and a body using their function. I would really like to make the above POST request from R but asynchronously where the URI is the same, but the data sent is different for each request.
I found this http://www.omegahat.org/RCurl/concurrent.html
getURIs =
function(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)
{
content = list()
curls = list()
for(i in uris) {
curl = getCurlHandle()
content[[i]] = basicTextGatherer()
opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)
curlSetOpt(.opts = opts, curl = curl)
multiHandle = push(multiHandle, curl)
}
if(.perform) {
complete(multiHandle)
lapply(content, function(x) x$value())
} else {
return(list(multiHandle = multiHandle, content = content))
}
}
My idea is that I could replace for (i in uris) with for(i in jsons) where I am looping over the different data that I want to send to the same URL, however I am having trouble understanding the following concepts from the RCurl Package:
How do I pass a header as part of a PUT request. How do I pass data in the body of the request? This was pretty straight forward using the httr package as I have illustrated above.
I tried passing in the header in the curl options and alternatively the header. The thing is I don't understand where to pass the the component parts of the post request: authentication, header, and body within the getURIAsynchronous() function, or any of the resources I have described above.
Does anyone know how to accomplish this? An example would be incredibly helpful.

Answer 1

The curl package has been recently updated to handle async requests (see here)
Using the curl, magrittr and jsonlite packages you can create asynchronous post requests by:
Creating a generic handle with your header and body content using the handle_setform function
Writing a call back function to retrieve your results
Initializing a pool and adding your concurrent requests to it
Running your pool via multi_run
Example code is below:
library(curl)
library(jsonlite)
library(magrittr)
#create a handle object
h <- new_handle() %>%
handle_setheaders(Authorization = "somepwd",
"Content-Type" = "application/json") %>%
handle_setform(body = toJSON(iris))
pool <- new_pool()
# results only available through call back function
cb <- function(req){cat("done:", req$url, ": HTTP:", req$status, "\n", "content:", rawToChar(req$content), "\n")}
# example vector of uris to loop through
uris <- c("https://app.example.com/api/endpoint1"
,"https://app.example.com/api/endpoint2"
,"https://app.example.com/api/endpoint3")
# all scheduled requests are performed concurrently
sapply(uris, curl_fetch_multi, done=cb, pool=pool)
# This actually performs requests
out <- multi_run(pool = pool)