R - mocking API requests with `gh` package - r

I am trying to mock the output of a gh API request:
httptest2::with_mock_dir("gh", {
test_that("api works", {
gh::gh("GET /repos/r-lib/gh")
})
})
I am trying to set up testing for custom functions that routinely make API calls to GitHub and I am using gh to make these requests. I am following this tutorial as guidance: https://books.ropensci.org/http-testing/
However, no directory is created when this function is run. Is there anyway to capture the output of gh::gh and store it as a mock API return so that I can run my tests without needing GitHub authentication or even an internet connection?

httptest2 is specifically designed to test httr2 requests:
This package helps with writing tests for packages that use httr2
Unfortunately, gh uses httr:
Imports:
cli (>= 3.0.1),
gitcreds,
httr (>= 1.2),
ini,
jsonlite
This means that you can't directly use httptest2 with gh.
However, using gh source code, you can extract the parameters of the GET request sent to httr by gh:
gh_get <- function(endpoint, ..., per_page = NULL, .token = NULL, .destfile = NULL,
.overwrite = FALSE, .api_url = NULL, .method = "GET",
.limit = NULL, .accept = "application/vnd.github.v3+json",
.send_headers = NULL, .progress = TRUE, .params = list()) {
params <- c(list(...), .params)
params <- gh:::drop_named_nulls(params)
if (is.null(per_page)) {
if (!is.null(.limit)) {
per_page <- max(min(.limit, 100), 1)
}
}
if (!is.null(per_page)) {
params <- c(params, list(per_page = per_page))
}
req <- gh:::gh_build_request(
endpoint = endpoint, params = params,
token = .token, destfile = .destfile,
overwrite = .overwrite, accept = .accept,
send_headers = .send_headers,
api_url = .api_url, method = .method
)
req
}
req <- gh_get("GET /repos/r-lib/gh")
req
#$method
#[1] "GET"
#$url
#[1] "https://api.github.com/repos/r-lib/gh"
#$headers
# User-Agent Accept
# "https://github.com/r-lib/gh" "application/vnd.github.v3+json"
#$query
#NULL
#$body
#NULL
#$dest
#<request>
#Output: write_memory
This allows with the example you provided to use httr2 to send the same request :
library(httr2)
resp_httr2 <- request(base_url=req$url) %>%
req_perform() %>%
resp_body_json()
If you are mainly interested in json content, the results are the same, only the attributes differ :
resp_gh <- gh::gh("GET /repos/r-lib/gh")
all.equal(resp_gh,resp_httr2,check.attributes=FALSE)
#[1] TRUE
If you want to use httptest2, switching to httr2 would work:
with_mock_dir("gh", {
test_that("api works", {
resp <- request(base_url=req$url) %>%
req_perform() %>%
resp_body_json()
expect_equal(resp$full_name,"r-lib/gh")})
})
#Test passed 🎉
#[1] TRUE
Offline testing now works because gh\api.github.com directory was created by httptest2.

Maybe you can take inspiration from tests/testthat/test-mock-repos.R
res <- gh(
TMPL("/repos/{owner}/{repo}"),
owner = "gh-testing",
repo = test_repo,
.token = tt()
)
expect_equal(res$name, test_repo)
expect_equal(res$description, "Test repo for gh")
expect_equal(res$homepage, "https://github.com/r-lib/gh")
expect_false(res$private)
expect_false(res$has_issues)
expect_false(res$has_wiki)
A GET method would not create any directory.

Related

How to implement async calls in R HTTPUV startServer?

The R httpuv startServer function should support async processing in the call portion of the app parameter but I'm not able to get it to work. Does anyone know how to do this? The example below won't work but it shows the idea of what I'm trying to do, run each request (or for a specific page) async so a page can load while another request is processing.
startServer(
host,
port,
app = list(
call = function(req) {
req <- list(
"REQUEST_METHOD" = req$REQUEST_METHOD,
"SCRIPT_NAME" = req$SCRIPT_NAME,
"PATH_INFO" = req$PATH_INFO,
"QUERY_STRING" = req$QUERY_STRING,
"SERVER_NAME" = req$SERVER_NAME,
"SERVER_PORT" = req$SERVER_PORT,
"HEADERS" = req$HEADERS,
"rook.input" = req[["rook.input"]]$read_lines()
)
future_promise({
if(req$PATH_INFO %in% valid_dynamic_paths){
x <- eval(dynamic[[req$PATH_INFO]][req$REQUEST_METHOD])
list(
status = x[["status"]],
headers = x[["headers"]],
body = x[["body"]]
)
}else{
list(
status = 404,
headers = list(
'Content-Type' = 'text/html'
),
body = "404. Page not found."
)
}
})
},
staticPaths = static
)
)
I was able to get something similar to work. The code below shows the gist of it:
# fork a process for each new request
future::plan(future::multicore)
httpuv::runServer("0.0.0.0", 8080, list(
call = function(req) {
# `as.promise` is necessary, because `httpuv` is using `is.promise`
# under the hood to act differently. Unfortunately `is.promise` returns
# `FALSE` for a `future`.
promises::as.promise(
future::future({
Sys.sleep(5)
# Respond with HTTP 200 OK
list(
status = 200,
body = "Slept for 5 seconds",
headers = list(
# Content-Type is important, otherwise you will run
# into a "not compatible with STRSXP" error.
"content-type" = "text/plain"
)
)
})
)
}
))
Calling the server with to requests at (nearly) the same time, will show that you are waiting only for 5 seconds for both requests, and not 5 for one and 10 for the other.
time curl -s localhost:8080 > /dev/null &
time curl -s localhost:8080 > /dev/null
# After 5 seconds you should see output similar to the following:
# real 0m5.089s
# user 0m0.011s
# sys 0m0.010s
# real 0m5.112s
# user 0m0.020s
# sys 0m0.024s

How to provide a Map() via httr

I have an api which calls for one of it's body to be a map, e.g.
{
"functionName": "",
"parameters": "Map[string,?]"
}
How do I do this with the R library httr? My instiction was to provide a nested list along the lines of
httr::POST(..., body = list("functionName" = "copy", "parameters" = list(a = 1, b = 2)))
This delivers an error, with message "Error in curl::handle_setform(handle, .list = req$fields) :
Unsupported value type for form field 'parameters'."
I have no real idea how to construct an object otherwise that will be accepted.
Apologies for the lack of a reprex for this question, the API is private, but hopefully my problem is simple enough that someone can find the correct syntax.
I discovered the solution to this, which is to use jsonlist::toJSON() to construct the body. For example
body = jsonlite::toJSON(
list(
"functionName" = "Rename",
"parameters" = (list(object = "pathtofile",
newName = "myfile",
newExtension = ".csv")
)),
auto_unbox = TRUE
)
This constructs a json object in the expected format, which I don't think httr can do on it's own.

Using httr to place orders through BitMex API

I'm trying to use the httr R package to place orders on BitMex through their API.
I found some guidance over here, and after specifying both my API key and secret in respectively the objects K and S, I've tried the following
verb <- 'POST'
expires <- floor(as.numeric(Sys.time() + 10000))
path <- '/api/v1/order'
data <- '{"symbol":"XBTUSD","price":4500,"orderQty":10}'
body <- paste0(verb, path, expires, data)
signature <- hmac(S, body, algo = 'sha256')
body_l <- list(verb = verb, expires = expires, path = path, data = data)
And then both:
msg <- POST('https://www.bitmex.com/api/v1/order', encode = 'json', body = body_l, add_headers('api-key' = K, 'api-signature' = signature, 'api-expires' = expires))
and:
msg <- POST('https://www.bitmex.com/api/v1/order', body = body, add_headers('api-key' = K, 'api-signature' = signature, 'api-expires' = expires))
Give me the same error message when checked:
rawToChar(msg$content)
[1] "{\"error\":{\"message\":\"Signature not valid.\",\"name\":\"HTTPError\"}}"
I've tried to set it up according to how BitMex explains to use their API, but I appear to be missing something. They list out a couple of issues that might underly my invalid signature issue, but they don't seem to help me out. When following their example I get the exact same hashes, so that seems to be in order.
bit late to the party here but hopefully this helps!
Your POST call just needs some minor changes:
add content_type_json()
include .headers = c('the headers') in add_headers(). See example below:
library(httr)
library(digest)
S <- "your api secret"
K <- "your api key"
verb <- 'POST'
expires <- floor(as.numeric(Sys.time() + 10))
path <- '/api/v1/order'
data <- '{"symbol":"XBTUSD","price":4500,"orderQty":10}'
body <- paste0(verb, path, expires, data)
signature <- hmac(S, body, algo = 'sha256')
msg <- POST('https://www.bitmex.com/api/v1/order',
encode = 'json',
body = data,
content_type_json(),
add_headers(.headers = c('api-key' = K,
'api-signature' = signature,
'api-expires' = expires)))
content(msg, "text")
I have a package on CRAN - bitmexr - that provides a wrapper around the majority of BitMEX's API endpoints that you might be interested in. Still quite a "young" package so I would welcome any feedback!

How to improve formatting of slack messages using slackr?

I'm using slackr to send alert messages to a Slack channel. It works great except the message format is not great and I want to improve it.
install_github("hrbrmstr/slackr")
library(slackr)
slackr_setup(channel="#alerts", username="Mark Davis",
incoming_webhook_url = "https://hooks.slack.com/services/T31P8UDAB/BCH4HKQSC/*********",
api_token = "*********", echo = F)
alert="On Monday, 2018-09-03 # 2pm Pacific..."
slackr(alert)
Here is an example of how a message from slackr looks in Slack:
Here is an example of how I'd like it to look:
slackr doesn't seem to have many options in the way of formatting. I was thinking of building an image and inserting that, but I'm having trouble building an image out of a text file using R.
Perhaps there is another api I could call that could take my text and format it for slack?
I'm open to any suggestions.
Addendum:
Slackr has an option to upload files, so my latest attempt is to create an image from the text message and upload that object.
I am able to create a png file from the text message using the magick library. I created an image with a colored background, and I simply add the message text to the image:
library(magick)
alert_picture <- image_read('alert_480x150_dark_red.png')
alert_picture=image_annotate(alert_picture, DreamCloud_Alert, size = 20, gravity = "southwest",
color = "white", location = "+10+10")
image_write(alert_picture, path = "alert_picture.png", format = "png")
The image looks pretty good (although there doesn't seem to be an easy way to bold or underline specific words in the message), but the obstacle now is that I can't get the upload command to work.
slackr_upload(filename = "alert_picture.png")
I don't get any error messages but nothing is uploaded to slack.
I got around this issue by using the httr package to execute the post image function to slack.
Thanks to Adil B. for providing the solution:
Post Image to Slack Using HTTR package in R
I am not sure this is what you meant, but I solved allowing formatting like in a regular slack message by altering the slackr_bot() function and just removing the 2 sets of 3 back-ticks at the end of the code where it says text. Then just call it slackr_bot1() or something, and then you can post formatted messages. This is the function after the back-ticks removal:
slackr_bot1 <- function(...,
channel=Sys.getenv("SLACK_CHANNEL"),
username=Sys.getenv("SLACK_USERNAME"),
icon_emoji=Sys.getenv("SLACK_ICON_EMOJI"),
incoming_webhook_url=Sys.getenv("SLACK_INCOMING_URL_PREFIX")) {
if (incoming_webhook_url == "") {
stop("No incoming webhook URL specified. Did you forget to call slackr_setup()?", call. = FALSE)
}
if (icon_emoji != "") { icon_emoji <- sprintf(', "icon_emoji": "%s"', icon_emoji) }
resp_ret <- ""
if (!missing(...)) {
# mimics capture.output
# get the arglist
args <- substitute(list(...))[-1L]
# setup in-memory sink
rval <- NULL
fil <- textConnection("rval", "w", local = TRUE)
sink(fil)
on.exit({
sink()
close(fil)
})
# where we'll need to eval expressions
pf <- parent.frame()
# how we'll eval expressions
evalVis <- function(expr) withVisible(eval(expr, pf))
# for each expression
for (i in seq_along(args)) {
expr <- args[[i]]
# do something, note all the newlines...Slack ``` needs them
tmp <- switch(mode(expr),
# if it's actually an expresison, iterate over it
expression = {
cat(sprintf("> %s\n", deparse(expr)))
lapply(expr, evalVis)
},
# if it's a call or a name, eval, printing run output as if in console
call = ,
name = {
cat(sprintf("> %s\n", deparse(expr)))
list(evalVis(expr))
},
# if pretty much anything else (i.e. a bare value) just output it
integer = ,
double = ,
complex = ,
raw = ,
logical = ,
numeric = cat(sprintf("%s\n\n", as.character(expr))),
character = cat(sprintf("%s\n\n", expr)),
stop("mode of argument not handled at present by slackr"))
for (item in tmp) if (item$visible) { print(item$value, quote = FALSE); cat("\n") }
}
on.exit()
sink()
close(fil)
# combined all of them (rval is a character vector)
output <- paste0(rval, collapse="\n")
loc <- Sys.getlocale('LC_CTYPE')
Sys.setlocale('LC_CTYPE','C')
on.exit(Sys.setlocale("LC_CTYPE", loc))
resp <- POST(url = incoming_webhook_url, encode = "form",
add_headers(`Content-Type` = "application/x-www-form-urlencoded",
Accept = "*/*"), body = URLencode(sprintf("payload={\"channel\": \"%s\", \"username\": \"%s\", \"text\": \"%s\"%s}",
channel, username, output, icon_emoji)))
warn_for_status(resp)
}
return(invisible())
}
slackr_bot1("*test* on time")

Retain information about requested URL when using curl::curl_fetch_multi

I'm using the following code to perform multiple simultaneous requests.
urls <- c("https://httpbin.org/status/301", "https://httpbin.org/status/302", "https://httpbin.org/status/200")
result <- list()
p <- curl::new_pool(total_con = 10, host_con = 5, multiplex = T)
cb <- function(res) {
result <<- append(result, list(res))
cat("requested URL: ", url, "last URL: ", res$url, "\n\n")
}
for (url in urls) {
curl::curl_fetch_multi(url, done = cb, handle = curl::new_handle(failonerror = F, nobody = F, followlocation = T, ssl_verifypeer = 0), pool = p)
}
curl::multi_run(pool = p)
As you can see, I would like to print to the console the requested URL and the URL, that finally answered with 200 ok.
The following is printed to the console:
requested URL: https://httpbin.org/status/200 last URL: https://httpbin.org/status/200
requested URL: https://httpbin.org/status/200 last URL: https://httpbin.org/get
requested URL: https://httpbin.org/status/200 last URL: https://httpbin.org/get
The requested URL in the console output is always https://httpbin.org/status/200, because it's the last URL that used in the for-loop. So, that is the wrong way to do it.
How can I retain information about the initial requested URL when using curl_fetch_multi to use it after multi_run returned? That means it would be ideal if the requested URL would be added to the res-list to query it with something like cat("requested URL: ", res$requested_url, "last URL: ", res$url, "\n\n").
I had a similar issue where I wanted to do asynchronous POST requests using curl_fetch_multi and check which requests succeeded and which failed. However, due to the structure of the POST statement (all fields are in the request body) there is no identifying information whatsoever in the response object. My solution was to generate custom callback functions which carried an identifier.
urls <- c("https://httpbin.org/status/301", "https://httpbin.org/status/302", "https://httpbin.org/status/200")
result <- list()
# create an identifier for each url
url.ids = paste0("request_", seq_along(urls))
# custom callback function generator: generate a unique function for each url
cb = function(id){
function(res){
result[[id]] <<- res
}
}
# create the list of callback functions
cbfuns = lapply(url.ids, cb)
p <- curl::new_pool(total_con = 10, host_con = 5, multiplex = T)
for (i in seq_along(urls)) {
curl::curl_fetch_multi(urls[i], done = cbfuns[[i]], handle = curl::new_handle(failonerror = F, nobody = F, followlocation = T, ssl_verifypeer = 0), pool = p)
}
curl::multi_run(pool = p)
In this example, the custom callback functions are simply used to name the elements of result:
names(result)
## [1] "request_3" "request_1" "request_2"
which can then be used to tie each response back to the original request.

Resources