API access, Hashing query, possibly related to character encoding - r

Its going to be impossible to provide a reproducible example for this as it has to do with accessing a particular API - apologies in advance. I am trying to access the edinburgh fringe festivals API through R. The documentation says the following:
Authentication
An access key is required for all requests to the API and can be
obtained by registration. This key, combined with the signature
explained below, is required to get any access to the API. If you do
not provide an access key, or provide an invalid key or signature, the
server will return an HTTP 403 error.
Your secret token is used to calculate the correct signature for each
API request and must never be disclosed. Once issued, it is never
transmitted back or forward. The API server holds a copy of your
secret token, which it uses to check that you have signed each request
correctly.
You calculate the signature using the HMAC-SHA1 algorithm:
Build the full API request URL, including your access key but excluding the server domain - eg /events?festival=book&key=12345. See
the note below on URL encoding.
Calculate the hmac hash of the url using the sha1 algorithm and your secret token as the key.
Append the hex-encoded hash to your url as the signature parameter
URL encoding in queries
You should calculate the signature after URL-encoding any parameters -
for example, to search for the title "Mrs Brown" you would first build
the URL /events?title=Mrs%20Brown&key=12345 and then sign this string
and append the signature.
Signature encoding
Some languages - notably C# - default to encoding hashes in UTF-16.
Ensure your signature is encoded in plain ASCII hex or it will not be
valid.
What I have tried so far is below:
library(digest)
library(jsonlite)
source("authentication.R") # credentials stored here
create the query string
query <- paste0('/events?festival=demofringe&size=20&from=1&key=', API_KEY)
create the hashed query
sig <- hmac(SECRET_SIGNING_KEY, query, algo="sha1")
create the final url
url <- paste0('https://api.edinburghfestivalcity.com', query, '&signature=', sig)
submit to the API
results <- fromJSON(url)
and I get the error:
Error in open.connection(con, "rb") : HTTP error 417.
I'm not sure that the signature is ASCII encoded as per the documentation. Does anyone know how to debug this situation? I have tried iconv() to try and convert the encoding and when I call Encoding() on the character object it returns "unknown". I have also tried saving both files in RStudio with "save with encoding" set to ASCII and I have tried sourcing the authentications with encoding = "ASCII".
Incidentally when I paste the final url into a browser, I get the following error:
Invalid Accept header value. ['application/json', 'application/json;ver=2.0'] are supported

That's a server error. It should understand that
Accept: application/json, text/*, */*
Matches application/json. Note that instead of modifying jsonlite, it is beter to manually retrieve the response form the server, and then feed it to jsonlite.
library(httr)
library(jsonlite)
req <- GET(your_url, accept("application/json")
json <- content(req, as = 'text')
data <- fromJSON(json)

In the end, I solved this problem by editing the jsonlite::fromJSON() function. namely the line
curl::handle_setheaders(h, Accept = "application/json, text/*, */*")
I changed to
curl::handle_setheaders(h, Accept = "application/json")
I then also had to point to the internal functions with jsonlite:::. Now it works fine.

Related

How to decode SAML?

I would decode this message sent to our national tax office. I have no clue how to start it. Seems it is a SAML encoding, I do not know much about SAML.
But in the content either xml or json is.
I tried this website to decode, but it did not do it.
https://www.samltool.com/decode.php
How to decode?
The HTTP-Redirect binding uses the DEFLATE encoding. The SAML message has been sent using the HTTP-Post binding. This means that the form data will include either a SAMLRequest or SAMLResponse form variable. This form variable must first be URL-decoded and then base-64 decoded to get the resultant XML.

Problems to access an API in R

I am trying to download some data from an API.
Following the site documentation:
library(jsonlite)
document <- fromJSON("https://api.gurufocus.com/public/user/{your personal token?}/stock/WMT/keyratios");
epv <- document$Valuation$`EPV`
cat(epv)
I inserted my token, but unfortunately, I get this error:
Error in open.connection(con, "rb") : HTTP error 404.
Any idea to solve this problem?
Best regards
Up front: remove the {, ?} from the url.
Replace more of the string. If you notice in their examples:
Request
Method URL
GET https://api.gurufocus.com/public/user/{your personal token?}/stock/{symbol?}/financials
Example https://api.gurufocus.com/public/user/{your personal token?}/stock/WMT/financials
The {symbol?} is replaced with WMT. I'm inferring you need to do with same with your token.
If I call
https://api.gurufocus.com/public/user/{abcd1234?}/stock/WMT/financials
(where abcd1234 is a known fake token), I get a 404 Not Found. I believe this is because the question mark breaks the URL into a path component
https://api.gurufocus.com/public/user/{abcd1234
and arguments
}/stock/WMT/financials
However, when I do:
https://api.gurufocus.com/public/user/abcd1234/stock/WMT/financials
I get a 401 Unauthorized, meaning that it parsed a token out of it, found the endpoint, and the token is not sufficient (which I know) to authorize access.

How to make a token request in R?

Trying to get an access token for an API so I can use R to pull the data. The documentation I'm using has the following instructions:
The following POST request shows the format of an Axon Evidence token request:
POST /api/oauth2/token HTTP/1.1
Host: api.evidence.com
Content-Type: application/x-www-form-urlencoded
grant_type: client_credentials&partner_id={partner ID}&client_id={client ID}&client_secret={client secret}
Note: ensure that you use URL-encoded forms for all parameter values. The client secret is base64 encoded and therefore usually end in an equals sign (=), which must be encoded as %3D. Additionally, if you use a manual REST client for testing, ensure that you know whether the client is automatically performing URL encoding. For more information, see Resources.
I'm trying to request a token using the httr package with the instructions above and it doesn't seem to be working. I'm using:
Token_Response <- POST("https://api.evidence.com/", body = "POST /api/oauth2/token HTTP/1.1
Host: api.evidence.com
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials&partner_id={partner ID}&client_id={client ID}&secret={client secret}")
Unfortunately I can't make this question reproducible for security reasons, but I'm hoping it's just a syntax problem on my end.
You don't usually work directly with the body or headers when using HTTR. The function can do most of the formatting of the request for you. A proper translation would look more like
url <- "https://api.evidence.com/"
partnerId <- "my partner ID"
clientId <- "my client ID"
clientSecret <- "my client secret"
token_Response <- POST(url,
encode="form",
body=list(
grant_type = "client_credentials",
partner_id = partnerId,
client_id = clientId,
secret = clientSecret
)
)
I used a few variables for clarity but you could put the values directly in the function call if you like.

Sending a SOAP XML manually and receiving a HTTP 500 error code and binary data in the response

I am using ruby to send a SOAP request to a very enterprisey bla bla service, so unfortunately I can not attach any samples, there's nobody to send any server-side logs, nobody knows whats wrong on the provider side or how the actual HTTP requests need to look like (except a single XML example I got, but no HTTP headers), the docs are very Microsoft-centric with C# examples and whatnot ("instantiate AbstractFactoryFactory..." and whatnot), long live enterprise software.
But the bottom line is, eventually I took one of their own XMLs from their logs and sent it via HTTP to the endpoint from the WSDL and sent it to their host using the Savon gem raw XML option and got a HTTP 500 error from their host and a bunch of non-ascii binary data inside - literally, no ASCII characters are in the body.
I guessed that maybe Savon does some bad magic or that the XML option is not working as expected and I tried sending the same request via Faraday, but got the same thing,
the HTTP response headers says it's a HTTP response, XML encoded, from an ASP.NET host:
"content-type"=>"text/xml; charset=utf-8",
"server"=>"Microsoft-IIS/7.5",
"x-aspnet-version"=>"2.0.50727",
"x-powered-by"=>"ASP.NET",
but again, a 440 bytes worth of binaries in the response:
method=:post,
body=
"\x1F\x8B\b\x00\x00...
etc.
Am I missing some weird aspect of the SOAP specification and I need to do something to decode this data or has their server gone bonkers from my XML, HTTP headers or something else and I need to ping the provider?
Update 1
I noticed that their original XML had UTF-16 encoding set, so I tried encoding the raw string to UTF-16, then had Savon spew errors at me about bad data, then I updated encoding in the Savon client config. But I still get HTTP 500 error and binaries as response and if I try to log anything Savon reports a bug:
Encoding::CompatibilityError: incompatible encoding regexp match (US-ASCII regexp with UTF-16 string)
from /home/bbozo/.rvm/gems/ruby-2.2.4/gems/savon-2.11.1/lib/savon/log_message.rb:13:in `to_s'
Faraday basically reported the same behavior, an binary blob.
Update 2
I tried piping the encoding to every known encoding, and got nothing, even though the HTTP headers imply the encoding is UTF-8, it obviously isn't
Encoding.name_list.map{ |e_in| [ e_in, ( response.body.dup.force_encoding(e_in).encode('utf-8') rescue 'incompatible' ) ] }
There is nothing that would indicate the encoding in the WSDL files, the API spec doesn't even mention encoding except that the request XMLs need to be UTF-8 encoding, I tried encoding the body, changing the XML encoding definition, HTTP headers, but still I get the same binary blob, with the same heading (\x1F\x8B\b\x00\x00) - so it's not some weird encryption either.
Compression maybe?
I tried with https for good measure and nothing.
Question
Am I missing some weird aspect of the SOAP specification and I need to do something to decode this data or has their server gone bonkers from my XML, HTTP headers or something else and I need to ping the provider?
The response body was compressed! In the end I just gunzipped it and there it was,
How to decompress Gzip string in ruby?

Paw base Oauth encoding

I am trying to do oauth 2 with Paw. It doesn't appear to be encoding my Client ID and Client Secret correctly. My server expects username:password format. I have tried it in the same field, separate, with same results.
Example
Input = VjE6MHJ:SjZvR24=
Base 64 encode result = `VmpFNk1ISjpTalp2UjI0PQ==
Paw result = VmpFNk1ISjpTalp2UjI0JTNE
The base 64 result works and returns a token, but Paw does not as it is encoded wrong.
It's a late answer, sorry! It seems like the only difference is that the = sign at the end of your password, is URL encoded by Paw, while your server doesn't expects a URL encoded value.
According to the OAuth 2 spec / 2.3.1. Client Password, we need to URL encode the Client ID and Client Secret. Then the question is, should we also encode the = sign? It's not very clear from the spec. Have you tested with other clients (maybe client implementation of OAuth, in JS/Swift/Python/Java)?

Resources