I would like to expose an endpoint that accepts multipart/form-data, parses the multipart content and returns a csv-file. (the multipart input contains a csv dataset and processing instructions)
I’ve done this with plumber using Rook::Multipart$parse() as suggested here. Because plumber doesn’t support parallel requests, I’d like to re-implement this with RestRserve. The following won’t work – plumber’s inputs are of class environment (which Rook::Multipart$parse() assumes) whereas RestRserve’s inputs are of class Request R6.
application = Application$new(content_type = "text/plain")
application$add_post("/echo", function(req, res) {
multipart <- Rook::Multipart$parse(req$body)
dta <- read_csv(multipart$dta$tempfile, trim_ws=FALSE)
res$set_body(dta)
})
Any ideas on how to get multipart/form-data input to work with RestRserve?
RestRserve parses multipart body when process the incoming request. As result you have a raw request$body and metatdata in the request$files. Request object also provides a get_file method to extract body content. Let me show example for the app and request:
# load packages
library(readr)
library(callr)
library(httr)
# run RestRserve in the background
ps <- r_bg(function() {
library(RestRserve)
library(readr)
app = Application$new(content_type = "text/plain")
app$add_post(
path = "/echo",
FUN = function(request, response) {
# for debug
str(request$body)
str(request$files)
# extract multipart body field
cnt <- request$get_file("csv") # 'csv' from the upload form field
# parse CSV
dt <- read_csv(cnt)
# for debug
str(dt)
# do something with dt
identity(dt)
# write result to temp file
tmp <- tempfile()
write_csv(dt, tmp)
# set output body
response$set_body(c(tmpfile = tmp))
# or simply response$set_body(format_csv(dt))
}
)
backend = BackendRserve$new()
backend$start(app, http_port = 65080)
})
# wait for up
Sys.sleep(2L)
# check is alive
ps$is_alive()
# prepare CSV to upload
tmp <- tempfile()
write_csv(head(iris, 5), tmp)
# POST request with file
rs <- POST(
url = "http:/127.0.0.1:65080/echo",
body = list(csv = upload_file(tmp)),
encode = "multipart"
)
# get response content
cat(content(rs))
# read log from the RestRserve
cat(ps$read_output())
# kill background prcoess
ps$kill()
See ?Request for more details about fields and methods in this class.
Related
I'm trying to create authenticate into the Letterboxd API using R and the httr package. The Letterboxd docs give instructions, but I am not sure how to put everything together into a URL.
I know the url is:
url <- "https://api.letterboxd.com/api/v0/auth/token"
And then they want my username and password, presumably as JSON, but what I'll write as a named list since I'm doing this in R:
login_info <- list(
grant_type = "password",
username = "myemail#gmail.com",
password = "extremelysecurepassword"
)
I've tried various calls, using GET(), oauth2.0_token(), oauth_endpoint() functions from the httr package.
I feel like I have all the necessary information and am circling around a solution, but I can't quite nail it.
The docs contain this information:
When generating or refreshing an access token, make a form request to the /auth/token endpoint with Content-Type: application/x-www-form-urlencoded and Accept: application/json headers
(Full text is linked to above)
And I'm not sure how to add that information; in working with APIs through R, I'm used to just sending URLs with UTM parameters, but the inputs they want don't work here using ? and &.
I'm aware of this related post, but it looks like it relies on having a secret token already. And I don't seem to be able to generate a secret token inside of the GUI of Letterboxd.com, which is again what I'm used to doing with authentication. I think I need to feed it those sources of information above in login_info to the url, but I don't quite know how to connect the dots.
How do I authenticate to the Letterboxd API using R?
This runs for me but I get a 401 Unauthorized since you correctly did not supply valid credentials. It looks like there is a python library for this API https://github.com/swizzlevixen/letterboxd if you need hints how to make subsequent requests.
sign_request() is mimicking python library's api.py#L295-L304
sign_request <- function(apisecret, url, method, body = "") {
signing_bytes <- as.raw(c(charToRaw(method), 0, charToRaw(url), 0, charToRaw(body)))
# https://stackoverflow.com/a/31209556/8996878
# https://stackoverflow.com/q/54606193/8996878
digest::hmac(key = apisecret, object = signing_bytes, algo = "sha256", serialize = FALSE)
}
url <- "https://api.letterboxd.com/api/v0/auth/token"
login_info <- list(
grant_type = "password",
username = "myemail#gmail.com",
password = "extremelysecurepassword"
)
apikey <- "mytopsecretapikey"
apisecret <- "YOUR_API_SECRET"
method <- "POST"
params <- list(
apikey = apikey,
nonce = uuid::UUIDgenerate(),
timestamp = round(as.numeric(Sys.time()))
)
# now we need to sign the request
body <- paste(names(login_info), login_info, sep = "=", collapse = "&")
body <- URLencode(body)
body <- gsub("#","%40", body) # something URLencode doesn't do but post does
destination <- httr::parse_url(url)
destination$query <- params
post_url_with_params <- httr::build_url(destination)
signature <- sign_request(apikey, post_url_with_params, method, body)
token_request <- httr::POST(url, httr::add_headers(
"Accept" = "application/json",
"Authorization" = paste0("Signature ", signature)
),
query = params,
body = login_info, encode = "form", httr::verbose()
)
token_body <- httr::content(token_request, type = "application/json")
# look for the value of"access_token"
My plumber function runs properly in local ( creating two R sessions and sending post request from one to other works ). However when doing the same for digitalocean server it is returning empty sting.
Plumber API
# plumber.R
#* #post /
#* #get /
function(req){
tryCatch({
list( rawbody = req$postBody )
}, error = function(e) {
return( jsonlite::toJSON( list('error' = e), force = T ) )
})
}
POST request
rm( list = ls() )
library(httr)
library(jsonlite)
options(stringsAsFactors = FALSE)
# url for local testing
url <- "http://127.0.0.1:6531" #... have changed URL to server url when testing for server
body = list( 'msg' = 'hi' )
# set API path
path <- 'data'
# send POST Request to API
raw.result <- POST(url = url, path = path, body = body, encode = 'json')
# check status code
raw.result$status_code
# check response
fromJSON(rawToChar(raw.result$content))
Please note GET request works fine but sending data with JSON doesn't work.
Output for server url
fromJSON(rawToChar(raw.result$content))
$rawbody
[1] ""
Output from local url
fromJSON(rawToChar(raw.result$content))
$rawbody
[1] "hi"
What am I doing wrong ?
I'm using R to interact with the Gemini exchange API (https://docs.gemini.com/rest-api/) for private endpoints. I've been able to reduce my problem to endpoints which require more than 2 parameters in the payload. In particular I'm attempting to query the /v1/mytrades endpoint (https://docs.gemini.com/rest-api/#get-past-trades) which I believe requires the 'request', 'nonce' and 'symbol' parameters at a minimum. The error code I receive is HTTP 400 which Gemini describes as:
Auction not open or paused, ineligible timing, market not open, or the request was malformed; in the case of a private API request, missing or malformed Gemini private API authentication headers
I have no issues with other endpoints which require only the 'request' and 'nonce' parameters, so I'm struggling to understand which step is a problem since those queries require similar steps to create a base64 encoding of the payload, a signature of that encoding using the API secret and headers with that data plus the API key.
Below is my example code where MY_API_SECRET and MY_API_KEY are placeholders for the actual secret and key strings
# Set variable for the gemini api URL
geminiHost <- "https://api.gemini.com"
# Set variable for the gemini endpoint
geminiEndpoint <- "/v1/mytrades"
# Create the symbol parameter
symbol <- 'btcusd'
# Create nonce parameter
currentTimeNonce <- round(as.numeric(Sys.time()) * 1000, 0)
# Create JSON payload
payload <-
toJSON(data.frame(
request = geminiEndpoint,
nonce = currentTimeNonce,
symbol = symbol
)) %>% gsub("\\[|\\]", "", .)
# Convert payload to base64
payloadBase64Enc <- base64_enc(payload)
# Create signature
signatureString <- sha384(payloadBase64Enc, key = 'MY_API_SECRET')
# Construct the complete URL
completeURL <- paste0(geminiHost, geminiEndpoint)
# Create header
hdr = c(
"Content-Type" = "text/plain",
"Content-Length" = "0",
"Cache-Control" = "no-cache",
"X-GEMINI-APIKEY" = "MY_API_KEY",
"X-GEMINI-PAYLOAD" = payloadBase64Enc,
"X-GEMINI-SIGNATURE" = signatureString
)
# Request API using the complete URL and the required headers
mytradesAPIResult <- fromJSON(httpPOST(completeURL,
httpheader = hdr,
verbose = TRUE))
For comparison, the following code which requests the /v1/orders endpoint (https://docs.gemini.com/rest-api/#get-active-orders) does indeed come back with a response:
# Set variable for the gemini api URL
geminiHost <- "https://api.gemini.com"
# Set variable for the gemini endpoint
geminiEndpoint <- "/v1/orders"
# Create nonce parameter
currentTimeNonce <- round(as.numeric(Sys.time()) * 1000, 0)
# Create JSON payload
payload <-
toJSON(data.frame(request = geminiEndpoint, nonce = currentTimeNonce)) %>%
gsub("\\[|\\]", "", .)
# Convert payload to base64
payloadBase64Enc <- base64_enc(payload)
# Create signature
signatureString <- sha384(payloadBase64Enc, key = 'MY_API_SECRET')
# Construct the complete URL
completeURL <- paste0(geminiHost, geminiEndpoint)
# Create header
hdr = c(
"Content-Type" = "text/plain",
"Content-Length" = "0",
"Cache-Control" = "no-cache",
"X-GEMINI-APIKEY" = "MY_API_KEY",
"X-GEMINI-PAYLOAD" = payloadBase64Enc,
"X-GEMINI-SIGNATURE" = signatureString
)
# Request API using the complete URL and the required headers
mytradesAPIResult <- fromJSON(httpPOST(completeURL,
httpheader = hdr,
verbose = TRUE))
So in the latter code the only difference is the geminiEndpoint and payload construction which as mentioned above only has 2 required parameters (request and nonce). To expand further, I'm successfully hitting other endpoints such as /v1/tradevolume, /v1/heartbeat, and /v1/balances that also require those 2 parameters only while /v1/order/status is another example of an endpoint requiring at least 3 parameters that doesn't work for me.
I appreciate any help to understand where I'm going wrong with this. Thank you in advance!
As mentioned in the comments, I started to work on an equivalent bash script based on r2evans' reply when I found my issue. It was the base64 encoding step in R that resulted in some unusual output. As seen in my original scripts, I was using the "base64_enc()" function which is part of the jsonlite package. As a simple check I was trying to confirm that the encoding from R was equal to an equivalent encoding using base64 in shell so I started by trying to decode the R result.
In R, the encoding of the payload for the 3 parameter example was coming out with a backslash character '\' which is not valid Base64 and points to my misunderstanding of what the base64_enc function is doing. I replaced this function with base64_encode from the openssl package and now my 3 parameter queries are coming back with results.
I am having trouble with DELETE requests in R. I have been successful in making GET and POST requests using the below code. Any help / pointers will be appreciated.
It will require an api.key, secret & passphrase from GDAX to work.
Here is my function:
library(RCurl)
library(jsonlite)
library(httr)
library(digest)
cancel_order <- function(api.key,
secret,
passphrase) {
api.url <- "https://api.gdax.com"
#get url extension----
req.url <- "/orders/"
#define method----
method = "DELETE"
url <- paste0(api.url, req.url)
timestamp <-
format(as.numeric(Sys.time()), digits = 13) # create nonce
key <- base64Decode(secret, mode = "raw") # encode api secret
#create final end point----
what <- paste0(timestamp, method, req.url)
#create encoded signature----
sign <-
base64Encode(hmac(key, what, algo = "sha256", raw = TRUE)) # hash
#define headers----
httpheader <- list(
'CB-ACCESS-KEY' = api.key,
'CB-ACCESS-SIGN' = sign,
'CB-ACCESS-TIMESTAMP' = timestamp,
'CB-ACCESS-PASSPHRASE' = passphrase,
'Content-Type' = 'application/json'
)
##------------------------------------------------
response <- getURL(
url = url,
curl = getCurlHandle(useragent = "R"),
httpheader = httpheader
)
print(rawToChar(response)) #rawToChar only on macOS and not on Win
}
The error I get is "{\"message\":\"invalid signature\"}" even though the same command will code and signature will work with GET & POST.
Ref: GDAX API DOCs
just a guess as I am not familiar with the API, but perhaps you are missing the 'order-id' ...
look at: https://docs.gdax.com/?javascript#cancel-an-order
Ok. I took #mrflick's advise and pointed my connection to requestbin based on his feedback on a different but related question.
After careful inspection, I realized that the my request for some reason was treated as a POST request and not a DELETE request. So I decided to replace the getURL function with another higher level function from RCurl for it to work.
response <- httpDELETE(
url = url,
curl = getCurlHandle(useragent = "R"),
httpheader = httpheader
)
Everything else remains the same. Apparently there never was an issue with the signature.
I have added this function now to my unofficial wrapper rgdax
EDIT::
The unofficial wrapper is now official and on CRAN.
I have a shiny app that takes a JSON input file, runs it through a classifier and returns a classified JSON object.
I want the app to be able to communicate with an API. I want the API to post a file to the Shiny App which will do its work and return a classified object.
Basically I want the Shiny app to sit in the background until a file is posted and then do its work.
I know that I can use GET from the httr package to get a file from a url. I can put this in the shiny.server file which is fine if I know the file name for the get command
However the filenames coming from the API will be different. So is there any way that I can make this dynamic according to the Post request that comes from the API.
If you do not have to use Shiny, you can use openCPU. OpenCPU provides each of your R packages as REST service automatically. I work with OpenCPU and it works fine! It is the easiest way to use R from another program.
By now library(plumber) needs to be mentioned as an alternative in this context, however the following example is showing how to handle POST requests directly in shiny.
It is based on Joe Cheng's gist here, which suggests to add an attribute "http_methods_supported" to the UI and use httpResponse to answer the requests.
The below code starts a shiny app in a background R process (This is done only to have a single single file MRE - of course, you can put the app in a separate file and remove the r_bg-line). After the app is launched the parent process sends the iris data.frame to the UI.
In the UI function the req$PATH_INFO is checked (see uiPattern = ".*"), then the numerical columns are multiplied by 10 (query_params$factor) and send back as a json string.
library(shiny)
library(jsonlite)
library(callr)
library(datasets)
ui <- function(req) {
# The `req` object is a Rook environment
# See https://github.com/jeffreyhorner/Rook#the-environment
if (identical(req$REQUEST_METHOD, "GET")) {
fluidPage(
h1("Accepting POST requests from Shiny")
)
} else if (identical(req$REQUEST_METHOD, "POST")) {
# Handle the POST
query_params <- parseQueryString(req$QUERY_STRING)
body_bytes <- req$rook.input$read(-1)
if(req$PATH_INFO == "/iris"){
postedIris <- jsonlite::fromJSON(rawToChar(body_bytes))
modifiedIris <- postedIris[sapply(iris, class) == "numeric"]*as.numeric(query_params$factor)
httpResponse(
status = 200L,
content_type = "application/json",
content = jsonlite::toJSON(modifiedIris, dataframe = "columns")
)
} else {
httpResponse(
status = 200L,
content_type = "application/json",
content = '{"status": "ok"}'
)
}
}
}
attr(ui, "http_methods_supported") <- c("GET", "POST")
server <- function(input, output, session) {}
app <- shinyApp(ui, server, uiPattern = ".*")
# shiny::runApp(app, port = 80, launch.browser = FALSE, host = "0.0.0.0")
shiny_process <- r_bg(function(x){ shiny::runApp(x, port = 80, launch.browser = FALSE, host = "0.0.0.0") }, args = list(x = app))
library(httr)
r <- POST(url = "http://127.0.0.1/iris?factor=10", body = iris, encode = "json", verbose())
recievedIris <- as.data.frame(fromJSON(rawToChar(r$content)))
print(recievedIris)
shiny_process$kill()
Please also check this related PR which is providing further examples (also showing how to use session$registerDataObj) and is aiming at a better description of the httpResponse function.