I have created a dummy model using the below code :
#get the data
data(Boston, package="MASS")
# train a model for median house price as a function of the other variables
bos_rf <- lm(medv ~ crim + indus + dis , data=Boston)
# save the model
saveRDS(bos_rf, "bos_rf.rds")
Now i want to expose this model as an API using plumber. For this my code is
# load as bos_rf.R
bos_rf <- readRDS("bos_rf.rds")
#* #param input_json JSON file
#* #post /score
function(input_json)
{
temp <- toJSON(input_json, auto_unbox = T)
data <- fromJSON(temp) %>% as.data.frame
data = data %>% mutate_all(as.numeric)
predict(bos_rf, data)
}
Above my param is a JSON , i am keen to actually keep it as a data frame . I am converting the JSON to data frame in the function
Then i start the API using
# try API 1
#
dummy_model_api <- plumber::plumb("2_R_code_to_API.R")
dummy_model_api$run(host = '127.0.0.1', port = 8000)
API runs fine when i paste the JSON in the swagger portal , but when i run curl using below commands
$ curl "http://127.0.0.1:8000/score" -d "#test.JSON"
$ curl --data #test.json http://localhost:8000/score
None work. how do i directly pass the test JSON to the API to get a prediction. Note that if i check the function with R i get the prediction. Kindly advise how can one pass a JSON or DF directly to curl API request and get a response vs manually copying json / or defining API inputs with each variable one by one. Such a method is not feasible with 100's of variables.
How can a sample of this JSON also reflect in the swagger body already. i.e. above when the swagger opens , a sample JSON is already there in body with some values and ready to execute.
plumber will execute fromJSON on the request body if it dectects it starts with {.
What you would normally do is send a JSON string like
{
"input_json" : _insert toJSON results here_
}
So that plumber can turn that into a named list and map input_json to your function parameter.
Related
Let me start by saying that I understand how to do a POST request using "httr" and "crul" packages. I am working on developing an asynchronous method to sending multiple POST request with unique JSON body requests using the basic "curl" package. I have legitimate reasons for trying this with this package, but more importantly I'm just determined to get it to work. This may not be possible, or I may even be trying to wrong functions in "curl"...but wanted to see if anyone had any ideas.
I am trying to send a post request using curl_fetch_multi() as a POST request with a JSON in the body like this...
{
"configuration": {
"Id": 4507
},
"age": 0,
"zip": 32411,
"Date": "2020-12-23"
}
I have succeeded in at least getting getting error messages back form the API indicating an invalid body input using something along the lines of starting with an object containing each body i need to submit
library(curl)
library(jsonlite)
library(magrittr)
pool <- new_pool()
# results only available through call back function
cb <- function(req){cat("done:", req$url, ": HTTP:", req$status, "\n", "content:", rawToChar(req$content), "\n")}
# Create request for each body
for(i in 1:nrow(df)){
curl_fetch_multi(
"http://api.com/values?api_key=1234",
done = cb,
pool = pool,
handle = new_handle() %>%
handle_setopt(post = TRUE) %>%
handle_setheaders("Content-Type"="application/vnd.v1+json") %>%
handle_setform(body = df$body[[i]]) ###df$body[[i]] is a JSON string
)
}
# This actually performs requests
out <- multi_run(pool = pool)
done: http://api.com/values?api_key=1234 : HTTP: 400
content: {"errors":[{"code":"Service.input.invalid","message":"Invalid input"}]}
done: http://api.com/values?api_key=1234 : HTTP: 400
content: {"errors":[{"code":"Service.input.invalid","message":"Invalid input"}]}
....
I'm 90% positive it has to do with how it's attempting to call the JSON in handle_setform() setting of the handle. This is about where I am over my head and documentation is scarce.
Also, I am pretty sure the JSON is structured properly, as I can use them in other packages with no problem.
Any assistance would be greatly appreciated.
Found the solution!!
Needed to use following settings with handle_setopts()
for(i in 1:nrow(df)){
curl_fetch_multi(
"http://api.com/values?api_key=1234",
done = cb,
pool = pool,
handle = new_handle() %>%
handle_setheaders("Content-Type"="application/v1+json") %>%
handle_setopt(customrequest = "POST") %>%
handle_setopt(postfields = df$body[[i]]) #df$body is list of JSON
)
}
out <- multi_run(pool = pool)
I would like to expose an endpoint that accepts multipart/form-data, parses the multipart content and returns a csv-file. (the multipart input contains a csv dataset and processing instructions)
I’ve done this with plumber using Rook::Multipart$parse() as suggested here. Because plumber doesn’t support parallel requests, I’d like to re-implement this with RestRserve. The following won’t work – plumber’s inputs are of class environment (which Rook::Multipart$parse() assumes) whereas RestRserve’s inputs are of class Request R6.
application = Application$new(content_type = "text/plain")
application$add_post("/echo", function(req, res) {
multipart <- Rook::Multipart$parse(req$body)
dta <- read_csv(multipart$dta$tempfile, trim_ws=FALSE)
res$set_body(dta)
})
Any ideas on how to get multipart/form-data input to work with RestRserve?
RestRserve parses multipart body when process the incoming request. As result you have a raw request$body and metatdata in the request$files. Request object also provides a get_file method to extract body content. Let me show example for the app and request:
# load packages
library(readr)
library(callr)
library(httr)
# run RestRserve in the background
ps <- r_bg(function() {
library(RestRserve)
library(readr)
app = Application$new(content_type = "text/plain")
app$add_post(
path = "/echo",
FUN = function(request, response) {
# for debug
str(request$body)
str(request$files)
# extract multipart body field
cnt <- request$get_file("csv") # 'csv' from the upload form field
# parse CSV
dt <- read_csv(cnt)
# for debug
str(dt)
# do something with dt
identity(dt)
# write result to temp file
tmp <- tempfile()
write_csv(dt, tmp)
# set output body
response$set_body(c(tmpfile = tmp))
# or simply response$set_body(format_csv(dt))
}
)
backend = BackendRserve$new()
backend$start(app, http_port = 65080)
})
# wait for up
Sys.sleep(2L)
# check is alive
ps$is_alive()
# prepare CSV to upload
tmp <- tempfile()
write_csv(head(iris, 5), tmp)
# POST request with file
rs <- POST(
url = "http:/127.0.0.1:65080/echo",
body = list(csv = upload_file(tmp)),
encode = "multipart"
)
# get response content
cat(content(rs))
# read log from the RestRserve
cat(ps$read_output())
# kill background prcoess
ps$kill()
See ?Request for more details about fields and methods in this class.
I'm using R to interact with the Gemini exchange API (https://docs.gemini.com/rest-api/) for private endpoints. I've been able to reduce my problem to endpoints which require more than 2 parameters in the payload. In particular I'm attempting to query the /v1/mytrades endpoint (https://docs.gemini.com/rest-api/#get-past-trades) which I believe requires the 'request', 'nonce' and 'symbol' parameters at a minimum. The error code I receive is HTTP 400 which Gemini describes as:
Auction not open or paused, ineligible timing, market not open, or the request was malformed; in the case of a private API request, missing or malformed Gemini private API authentication headers
I have no issues with other endpoints which require only the 'request' and 'nonce' parameters, so I'm struggling to understand which step is a problem since those queries require similar steps to create a base64 encoding of the payload, a signature of that encoding using the API secret and headers with that data plus the API key.
Below is my example code where MY_API_SECRET and MY_API_KEY are placeholders for the actual secret and key strings
# Set variable for the gemini api URL
geminiHost <- "https://api.gemini.com"
# Set variable for the gemini endpoint
geminiEndpoint <- "/v1/mytrades"
# Create the symbol parameter
symbol <- 'btcusd'
# Create nonce parameter
currentTimeNonce <- round(as.numeric(Sys.time()) * 1000, 0)
# Create JSON payload
payload <-
toJSON(data.frame(
request = geminiEndpoint,
nonce = currentTimeNonce,
symbol = symbol
)) %>% gsub("\\[|\\]", "", .)
# Convert payload to base64
payloadBase64Enc <- base64_enc(payload)
# Create signature
signatureString <- sha384(payloadBase64Enc, key = 'MY_API_SECRET')
# Construct the complete URL
completeURL <- paste0(geminiHost, geminiEndpoint)
# Create header
hdr = c(
"Content-Type" = "text/plain",
"Content-Length" = "0",
"Cache-Control" = "no-cache",
"X-GEMINI-APIKEY" = "MY_API_KEY",
"X-GEMINI-PAYLOAD" = payloadBase64Enc,
"X-GEMINI-SIGNATURE" = signatureString
)
# Request API using the complete URL and the required headers
mytradesAPIResult <- fromJSON(httpPOST(completeURL,
httpheader = hdr,
verbose = TRUE))
For comparison, the following code which requests the /v1/orders endpoint (https://docs.gemini.com/rest-api/#get-active-orders) does indeed come back with a response:
# Set variable for the gemini api URL
geminiHost <- "https://api.gemini.com"
# Set variable for the gemini endpoint
geminiEndpoint <- "/v1/orders"
# Create nonce parameter
currentTimeNonce <- round(as.numeric(Sys.time()) * 1000, 0)
# Create JSON payload
payload <-
toJSON(data.frame(request = geminiEndpoint, nonce = currentTimeNonce)) %>%
gsub("\\[|\\]", "", .)
# Convert payload to base64
payloadBase64Enc <- base64_enc(payload)
# Create signature
signatureString <- sha384(payloadBase64Enc, key = 'MY_API_SECRET')
# Construct the complete URL
completeURL <- paste0(geminiHost, geminiEndpoint)
# Create header
hdr = c(
"Content-Type" = "text/plain",
"Content-Length" = "0",
"Cache-Control" = "no-cache",
"X-GEMINI-APIKEY" = "MY_API_KEY",
"X-GEMINI-PAYLOAD" = payloadBase64Enc,
"X-GEMINI-SIGNATURE" = signatureString
)
# Request API using the complete URL and the required headers
mytradesAPIResult <- fromJSON(httpPOST(completeURL,
httpheader = hdr,
verbose = TRUE))
So in the latter code the only difference is the geminiEndpoint and payload construction which as mentioned above only has 2 required parameters (request and nonce). To expand further, I'm successfully hitting other endpoints such as /v1/tradevolume, /v1/heartbeat, and /v1/balances that also require those 2 parameters only while /v1/order/status is another example of an endpoint requiring at least 3 parameters that doesn't work for me.
I appreciate any help to understand where I'm going wrong with this. Thank you in advance!
As mentioned in the comments, I started to work on an equivalent bash script based on r2evans' reply when I found my issue. It was the base64 encoding step in R that resulted in some unusual output. As seen in my original scripts, I was using the "base64_enc()" function which is part of the jsonlite package. As a simple check I was trying to confirm that the encoding from R was equal to an equivalent encoding using base64 in shell so I started by trying to decode the R result.
In R, the encoding of the payload for the 3 parameter example was coming out with a backslash character '\' which is not valid Base64 and points to my misunderstanding of what the base64_enc function is doing. I replaced this function with base64_encode from the openssl package and now my 3 parameter queries are coming back with results.
I'm attempting to request data from a google spreadsheet using the googleAuthR. I need to use this library instead of Jenny Bryan's googlesheets because the request is part of a shiny app with multiple user authentication. When the request range does not contain spaces (e.g. "Sheet1!A:B" the request succeeds. However, when the tab name contains spaces (e.g. "'Sheet 1'!A:B" or "\'Sheet 1\'!A:B", the request fails and throws this error:
Request Status Code: 400
Error : lexical error: invalid char in json text.
<!DOCTYPE html> <html lang=en>
(right here) ------^
Mark Edmondson's googleAuthR uses jsonlite for parsing JSON. I assume this error is coming from jsonlite, but I'm at a loss for how to fix it. Here is a minimal example to recreate the issue:
library(googleAuthR)
# scopes
options("googleAuthR.scopes.selected" = "https://www.googleapis.com/auth/spreadsheets.readonly")
# client id and secret
options("googleAuthR.client_id" = "XXXX")
options("googleAuthR.client_secret" = "XXXX")
# request
get_data <- function(spreadsheetId, range) {
l <- googleAuthR::gar_api_generator(
baseURI = "https://sheets.googleapis.com/v4/",
http_header = 'GET',
path_args = list(spreadsheets = spreadsheetId,
values = range),
pars_args = list(majorDimension = 'ROWS',
valueRenderOption = 'UNFORMATTED_VALUE'),
data_parse_function = function(x) x)
req <- l()
req
}
# authenticate
gar_auth(new_user = TRUE)
# input
spreadsheet_id <- "XXXX"
range <- "'Sheet 1'!A:B"
# get data
df <- get_data(spreadsheet_id, range)
How should I format range variable for the request to work? Thanks in advance for the help.
Use URLencode() to percent-encode spaces.
Details:
Using options(googleAuthR.verbose = 1) shows that the GET request was of the form:
GET /v4/spreadsheets/.../values/'Sheet 1'!A:B?majorDimension=ROWS&valueRenderOption=UNFORMATTED_VALUE HTTP/1.1
I had assumed the space would be encoded, but I guess not. In this github issue from August 2016, Mark states URLencode() was going to be the default for later versions of googleAuthR. Not sure if that will still happen, but it's an easy fix in the meantime.
I am trying to make async requests to a REST API from R. The below curl command illustrates the parameters that I need to the pass to the api. I'm giving you guys the linux curl command as I'm hoping that will make it clear:
curl -v -X POST https://app.example.com/api/ \
-H 'Authorization: somepwd' \
-H "Content-Type: application/json" \
-d {key1: value1, key2: value2}
Right now, I'm accomplishing the same thing in R by executing the following:
library(httr)
library(jsonlite)
content(POST('https://app.example.com/api/'
,add_headers(Authorization = 'somepwd')
,body = toJSON(rDataFrame)
,content_type_json()
)
)
The goal is to submit the above POST request from R but to vary the json string that is sent in the body, and do that asynchronously.
I have been searching for packages that will help me make asynchronous requests rather than making requests serially. The closest thing I could find is the getURIAsynchronous() function from the RCurl package (https://cran.r-project.org/web/packages/RCurl/RCurl.pdf) but do not understand how to submit a PUT request with headers and a body using their function. I would really like to make the above POST request from R but asynchronously where the URI is the same, but the data sent is different for each request.
I found this http://www.omegahat.org/RCurl/concurrent.html
getURIs =
function(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)
{
content = list()
curls = list()
for(i in uris) {
curl = getCurlHandle()
content[[i]] = basicTextGatherer()
opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)
curlSetOpt(.opts = opts, curl = curl)
multiHandle = push(multiHandle, curl)
}
if(.perform) {
complete(multiHandle)
lapply(content, function(x) x$value())
} else {
return(list(multiHandle = multiHandle, content = content))
}
}
My idea is that I could replace for (i in uris) with for(i in jsons) where I am looping over the different data that I want to send to the same URL, however I am having trouble understanding the following concepts from the RCurl Package:
How do I pass a header as part of a PUT request. How do I pass data in the body of the request? This was pretty straight forward using the httr package as I have illustrated above.
I tried passing in the header in the curl options and alternatively the header. The thing is I don't understand where to pass the the component parts of the post request: authentication, header, and body within the getURIAsynchronous() function, or any of the resources I have described above.
Does anyone know how to accomplish this? An example would be incredibly helpful.
The curl package has been recently updated to handle async requests (see here)
Using the curl, magrittr and jsonlite packages you can create asynchronous post requests by:
Creating a generic handle with your header and body content using the handle_setform function
Writing a call back function to retrieve your results
Initializing a pool and adding your concurrent requests to it
Running your pool via multi_run
Example code is below:
library(curl)
library(jsonlite)
library(magrittr)
#create a handle object
h <- new_handle() %>%
handle_setheaders(Authorization = "somepwd",
"Content-Type" = "application/json") %>%
handle_setform(body = toJSON(iris))
pool <- new_pool()
# results only available through call back function
cb <- function(req){cat("done:", req$url, ": HTTP:", req$status, "\n", "content:", rawToChar(req$content), "\n")}
# example vector of uris to loop through
uris <- c("https://app.example.com/api/endpoint1"
,"https://app.example.com/api/endpoint2"
,"https://app.example.com/api/endpoint3")
# all scheduled requests are performed concurrently
sapply(uris, curl_fetch_multi, done=cb, pool=pool)
# This actually performs requests
out <- multi_run(pool = pool)