I'm trying to create authenticate into the Letterboxd API using R and the httr package. The Letterboxd docs give instructions, but I am not sure how to put everything together into a URL.
I know the url is:
url <- "https://api.letterboxd.com/api/v0/auth/token"
And then they want my username and password, presumably as JSON, but what I'll write as a named list since I'm doing this in R:
login_info <- list(
grant_type = "password",
username = "myemail#gmail.com",
password = "extremelysecurepassword"
)
I've tried various calls, using GET(), oauth2.0_token(), oauth_endpoint() functions from the httr package.
I feel like I have all the necessary information and am circling around a solution, but I can't quite nail it.
The docs contain this information:
When generating or refreshing an access token, make a form request to the /auth/token endpoint with Content-Type: application/x-www-form-urlencoded and Accept: application/json headers
(Full text is linked to above)
And I'm not sure how to add that information; in working with APIs through R, I'm used to just sending URLs with UTM parameters, but the inputs they want don't work here using ? and &.
I'm aware of this related post, but it looks like it relies on having a secret token already. And I don't seem to be able to generate a secret token inside of the GUI of Letterboxd.com, which is again what I'm used to doing with authentication. I think I need to feed it those sources of information above in login_info to the url, but I don't quite know how to connect the dots.
How do I authenticate to the Letterboxd API using R?
This runs for me but I get a 401 Unauthorized since you correctly did not supply valid credentials. It looks like there is a python library for this API https://github.com/swizzlevixen/letterboxd if you need hints how to make subsequent requests.
sign_request() is mimicking python library's api.py#L295-L304
sign_request <- function(apisecret, url, method, body = "") {
signing_bytes <- as.raw(c(charToRaw(method), 0, charToRaw(url), 0, charToRaw(body)))
# https://stackoverflow.com/a/31209556/8996878
# https://stackoverflow.com/q/54606193/8996878
digest::hmac(key = apisecret, object = signing_bytes, algo = "sha256", serialize = FALSE)
}
url <- "https://api.letterboxd.com/api/v0/auth/token"
login_info <- list(
grant_type = "password",
username = "myemail#gmail.com",
password = "extremelysecurepassword"
)
apikey <- "mytopsecretapikey"
apisecret <- "YOUR_API_SECRET"
method <- "POST"
params <- list(
apikey = apikey,
nonce = uuid::UUIDgenerate(),
timestamp = round(as.numeric(Sys.time()))
)
# now we need to sign the request
body <- paste(names(login_info), login_info, sep = "=", collapse = "&")
body <- URLencode(body)
body <- gsub("#","%40", body) # something URLencode doesn't do but post does
destination <- httr::parse_url(url)
destination$query <- params
post_url_with_params <- httr::build_url(destination)
signature <- sign_request(apikey, post_url_with_params, method, body)
token_request <- httr::POST(url, httr::add_headers(
"Accept" = "application/json",
"Authorization" = paste0("Signature ", signature)
),
query = params,
body = login_info, encode = "form", httr::verbose()
)
token_body <- httr::content(token_request, type = "application/json")
# look for the value of"access_token"
Related
Using a browser, I can navigate to https://myaccount.draftkings.com/login and login using a username and password. Then if I navigate to https://api.draftkings.com/scores/v1/leaderboards/9000?format=json&embed=leaderboard in another tab, I can see the API response.
I want to do this programmatically.
There is a python package that I previously used successfully for this purpose. But for some reason, I can't get that package to work now.
I believe the python package hits an API endpoint with a post request to authenticate and saves the cookies to a file that are later used to make requests.
I tried sending a post request to the login endpoint the python package uses with httr2:
req <- httr2::request("https://api.draftkings.com/users/v3/providers/draftkings/logins?format=json") %>%
httr2::req_method("POST") %>%
httr2::req_headers(
"Contest-Type" = "application/json",
"Accept" = "*/*",
"Accept-Encoding" = "gzip, deflate, br"
) %>%
httr2::req_body_json(
list(
"login" = "email",
"password" = "password",
"host" = "api.draftkings.com",
"challengeResponse" = list("solution" = "", "type" = "Recaptcha")
)
)
req %>%
httr2::req_perform()
but I get a 403 error.
I also tried logging in using rvest:
library(rvest)
url <- "myaccount.draftkings.com/login"
session <- session(url)
form <- session %>%
html_form() %>%
magrittr::extract2(1)
form$action <- url
filled_form <- form %>%
html_form_set(!!! list(EmailOrUsername = "user",
Password = 'password'))
html_form_submit(filled_form, submit = 3)
session_jump_to(session, "https://www.draftkings.com/lobby")
but that didn't seem to do what I want either. Note that the form object returned an empty action element. I'm not sure what to replace the action element with, but if I leave it null I get an error.
I also tried saving the cookies from a browser session after logging into draftkings.com and passing all the cookies to a GET request with httr:
cook <- jsonlite::read_json("cookies.json")
clean_cook <- unlist(lapply(cook, function(x) {stats::setNames(x$value, x$name)}))
resp <- httr::GET(
"https://api.draftkings.com/scores/v1/leaderboards/9000?format=json&embed=leaderboard",
httr::set_cookies(.cookies = clean_cook)
)
httr::content(resp)
but this returns a 400 error code and message "Invalid userKey.". This is the same error message I get if I clear my cache in the browser and then visit https://api.draftkings.com/scores/v1/leaderboards/9000?format=json&embed=leaderboard. I don't think the issue is related to URL encoding of the cache values. I tried restarting my RStudio session.
Update
I figured out how to successfully perform the GET request using the cookies saved from my browser and httr2:
cook <- jsonlite::read_json("cookies.json")
clean_cook <- paste0(unlist(lapply(cook, function(x) {paste0(x$name, "=", x$value)})), collapse = ";")
req <- httr2::request(
"https://api.draftkings.com/scores/v1/leaderboards/9000?format=json&embed=leaderboard"
)
req <- httr2::req_headers(req, "cookie" = clean_cook)
resp <- httr2::req_perform(req)
str(httr2::resp_body_json(resp))
For some reason, I was unable to get httr::set_cookies() to work. Passing cookies in the header (using httr::add_headers) was also not reliably successful using httr.
I need to authenticate AWS API in R. I tried using aws.signature package to do the same and I am getting 403 response with error Missing Authentication Token . It seems that I am missing some necessary parameters. Looking for assistance to debug the below code or ways to authenticate AWS API in R.
# To create aws signature for authentication for the rest API call
library(aws.signature)
library(httr)
# validate arguments and setup request URL
current <- Sys.time()
d_timestamp <- format(current, "%Y%m%dT%H%M%SZ", tz = "UTC")
hdrs <- list(`Content-Type` = "application/x-www-form-urlencoded",
Host = "jteti5wnje.execute-api.eu-central-1.amazonaws.com",
`x-amz-date` = d_timestamp)
params <- signature_v4_auth(
datetime = d_timestamp,
region = "eu-central-1",
service = "execute-api",
verb = "GET",
action = "iMetaAPI",
query_args = list(),
canonical_headers = hdrs,
request_body = "json",
key = "***************",
secret = "*****************",
session_token = NULL,
query = FALSE,
algorithm = "AWS4-HMAC-SHA256",
verbose = TRUE)
a <- GET("https://jteti5wnje.execute-api.eu-central-1.amazonaws.com/iMetaAPI",
query = params)
rawToChar(a$content)
A few things:
request_body needs to be the actual body. You're doing a GET() so it should just be NULL or "".
The response from signature_v4_auth() is a list but you don't need all of its elements - probably just the hds$Authorization <- params$SignatureHeader element.
The actual errors you're getting is because you haven't passed the headers. You need to pass the headers to GET() with something like: do.call(add_headers, hdrs) so you can do:
a <- GET("https://jteti5wnje.execute-api.eu-central-1.amazonaws.com/iMetaAPI", do.call(add_headers, headers))
That will probably work, or you'll at least get a more informative error.
I'm using R to interact with the Gemini exchange API (https://docs.gemini.com/rest-api/) for private endpoints. I've been able to reduce my problem to endpoints which require more than 2 parameters in the payload. In particular I'm attempting to query the /v1/mytrades endpoint (https://docs.gemini.com/rest-api/#get-past-trades) which I believe requires the 'request', 'nonce' and 'symbol' parameters at a minimum. The error code I receive is HTTP 400 which Gemini describes as:
Auction not open or paused, ineligible timing, market not open, or the request was malformed; in the case of a private API request, missing or malformed Gemini private API authentication headers
I have no issues with other endpoints which require only the 'request' and 'nonce' parameters, so I'm struggling to understand which step is a problem since those queries require similar steps to create a base64 encoding of the payload, a signature of that encoding using the API secret and headers with that data plus the API key.
Below is my example code where MY_API_SECRET and MY_API_KEY are placeholders for the actual secret and key strings
# Set variable for the gemini api URL
geminiHost <- "https://api.gemini.com"
# Set variable for the gemini endpoint
geminiEndpoint <- "/v1/mytrades"
# Create the symbol parameter
symbol <- 'btcusd'
# Create nonce parameter
currentTimeNonce <- round(as.numeric(Sys.time()) * 1000, 0)
# Create JSON payload
payload <-
toJSON(data.frame(
request = geminiEndpoint,
nonce = currentTimeNonce,
symbol = symbol
)) %>% gsub("\\[|\\]", "", .)
# Convert payload to base64
payloadBase64Enc <- base64_enc(payload)
# Create signature
signatureString <- sha384(payloadBase64Enc, key = 'MY_API_SECRET')
# Construct the complete URL
completeURL <- paste0(geminiHost, geminiEndpoint)
# Create header
hdr = c(
"Content-Type" = "text/plain",
"Content-Length" = "0",
"Cache-Control" = "no-cache",
"X-GEMINI-APIKEY" = "MY_API_KEY",
"X-GEMINI-PAYLOAD" = payloadBase64Enc,
"X-GEMINI-SIGNATURE" = signatureString
)
# Request API using the complete URL and the required headers
mytradesAPIResult <- fromJSON(httpPOST(completeURL,
httpheader = hdr,
verbose = TRUE))
For comparison, the following code which requests the /v1/orders endpoint (https://docs.gemini.com/rest-api/#get-active-orders) does indeed come back with a response:
# Set variable for the gemini api URL
geminiHost <- "https://api.gemini.com"
# Set variable for the gemini endpoint
geminiEndpoint <- "/v1/orders"
# Create nonce parameter
currentTimeNonce <- round(as.numeric(Sys.time()) * 1000, 0)
# Create JSON payload
payload <-
toJSON(data.frame(request = geminiEndpoint, nonce = currentTimeNonce)) %>%
gsub("\\[|\\]", "", .)
# Convert payload to base64
payloadBase64Enc <- base64_enc(payload)
# Create signature
signatureString <- sha384(payloadBase64Enc, key = 'MY_API_SECRET')
# Construct the complete URL
completeURL <- paste0(geminiHost, geminiEndpoint)
# Create header
hdr = c(
"Content-Type" = "text/plain",
"Content-Length" = "0",
"Cache-Control" = "no-cache",
"X-GEMINI-APIKEY" = "MY_API_KEY",
"X-GEMINI-PAYLOAD" = payloadBase64Enc,
"X-GEMINI-SIGNATURE" = signatureString
)
# Request API using the complete URL and the required headers
mytradesAPIResult <- fromJSON(httpPOST(completeURL,
httpheader = hdr,
verbose = TRUE))
So in the latter code the only difference is the geminiEndpoint and payload construction which as mentioned above only has 2 required parameters (request and nonce). To expand further, I'm successfully hitting other endpoints such as /v1/tradevolume, /v1/heartbeat, and /v1/balances that also require those 2 parameters only while /v1/order/status is another example of an endpoint requiring at least 3 parameters that doesn't work for me.
I appreciate any help to understand where I'm going wrong with this. Thank you in advance!
As mentioned in the comments, I started to work on an equivalent bash script based on r2evans' reply when I found my issue. It was the base64 encoding step in R that resulted in some unusual output. As seen in my original scripts, I was using the "base64_enc()" function which is part of the jsonlite package. As a simple check I was trying to confirm that the encoding from R was equal to an equivalent encoding using base64 in shell so I started by trying to decode the R result.
In R, the encoding of the payload for the 3 parameter example was coming out with a backslash character '\' which is not valid Base64 and points to my misunderstanding of what the base64_enc function is doing. I replaced this function with base64_encode from the openssl package and now my 3 parameter queries are coming back with results.
I am having trouble with DELETE requests in R. I have been successful in making GET and POST requests using the below code. Any help / pointers will be appreciated.
It will require an api.key, secret & passphrase from GDAX to work.
Here is my function:
library(RCurl)
library(jsonlite)
library(httr)
library(digest)
cancel_order <- function(api.key,
secret,
passphrase) {
api.url <- "https://api.gdax.com"
#get url extension----
req.url <- "/orders/"
#define method----
method = "DELETE"
url <- paste0(api.url, req.url)
timestamp <-
format(as.numeric(Sys.time()), digits = 13) # create nonce
key <- base64Decode(secret, mode = "raw") # encode api secret
#create final end point----
what <- paste0(timestamp, method, req.url)
#create encoded signature----
sign <-
base64Encode(hmac(key, what, algo = "sha256", raw = TRUE)) # hash
#define headers----
httpheader <- list(
'CB-ACCESS-KEY' = api.key,
'CB-ACCESS-SIGN' = sign,
'CB-ACCESS-TIMESTAMP' = timestamp,
'CB-ACCESS-PASSPHRASE' = passphrase,
'Content-Type' = 'application/json'
)
##------------------------------------------------
response <- getURL(
url = url,
curl = getCurlHandle(useragent = "R"),
httpheader = httpheader
)
print(rawToChar(response)) #rawToChar only on macOS and not on Win
}
The error I get is "{\"message\":\"invalid signature\"}" even though the same command will code and signature will work with GET & POST.
Ref: GDAX API DOCs
just a guess as I am not familiar with the API, but perhaps you are missing the 'order-id' ...
look at: https://docs.gdax.com/?javascript#cancel-an-order
Ok. I took #mrflick's advise and pointed my connection to requestbin based on his feedback on a different but related question.
After careful inspection, I realized that the my request for some reason was treated as a POST request and not a DELETE request. So I decided to replace the getURL function with another higher level function from RCurl for it to work.
response <- httpDELETE(
url = url,
curl = getCurlHandle(useragent = "R"),
httpheader = httpheader
)
Everything else remains the same. Apparently there never was an issue with the signature.
I have added this function now to my unofficial wrapper rgdax
EDIT::
The unofficial wrapper is now official and on CRAN.
I got some problems accessing tracking API via 0auth;
I have sign key, secret and Authorization_key.
Sign key and secret should form a sig file as I understand and authorization_key that should be added as header.
The code:
auth_info <-
read.csv2('./Data/auth_info.csv',
encoding = 'UTF-8',
stringsAsFactors = F)
express <-
oauth_app("express", key = 'Ur1475', secret = auth_info$sign_key)
sig <- sign_oauth1.0(express)
order <-
GET(
"http://www.express.ru/api/v2/getOrder?orderNumber=WEBN3141018",
authenticate("Ur1475", auth_info$sign_key),
add_headers(Authorization = auth_info$auth_key),
sig
)
order_info <- content(order)
order_info
Returns me auth error;
Official api documentation tells that should be formed md5(client key . URL . GET params . POST params . sign key).
Also I've looked at 0auth function code it does not use http.
Could it cause the problem?
Could you please comment on how to get data from API via R?
API doc (in russian)): https://www.express.ru/docs/APIExpressRu.pdf
The major of API's (Google, Twitter, Facebook) need a token that you must to obtain from an oauth authentication. Depending from the API's you must to use the functions GETor POST. And you must to provide your key encoded in base64. So what I think you need is:
First: Authenticate.
if(!require("jsonlite")){library("jsonlite")}
if(!require("httr")){library("httr")}
#Create your own appication key
consumer_key <- "your consumer key" #needed
consumer_secret <- "your consumer secret" #some api's don't need.
#Use basic auth
secret <- jsonlite::base64_enc(paste(consumer_key, consumer_secret, sep = ":"))
req <- httr::POST("https://url.of.your.api/oauth/token", #"oauth" and "token" are not parameters, they are fixed.
httr::add_headers(
"Authorization" = paste("Basic", gsub("\n", "", secret)),
"Content-Type" = "application/x-www-form-urlencoded;charset=UTF-8"
),
body = "grant_type=client_credentials"
)
#Extract the access token
token <- paste("Bearer", httr::content(req)$access_token)
Next: Make a request.
url<-"http://url.of.your.api/search?parameter1=value¶meter2=value"
req <- httr::POST(url, httr::add_headers("Authorization" = token))
json <- httr::content(req)
json
As I said, some API's need GETand others POST, try both.