Context
I want to access an API from R and have the correct credentials to do so.
The issue is that Authentication requires three arguments to be passed (Username, Password and Organisation).
I am not familiar with API calls from R and I do not know how to pass this "organisation" argument.
Example
An example of the API call to request a token is as follows (taken from the documentation):
curl -X POST "https://URL.com" \
-H "Content-Type: application/json" \
--data "{\"user_name\": \"<username>\" \
\"password\": \"<mypassword>\",\
\"organisation\": \"<organisation>\"}"
Using the crul package, I have tried:
require("crul")
z <- crul::HttpClient$new(url = "https://THEURL.com",
auth(user = "myuser",
pwd = "mypwd"))
z$get()
Which returns:
<crul response>
url: https://THEURL.com
request_headers:
User-Agent: libcurl/7.59.0 r-curl/3.3 crul/0.8.0
Accept-Encoding: gzip, deflate
Accept: application/json, text/xml, application/xml, */*
response_headers:
status: HTTP/1.1 403 Forbidden
content-type: application/json
content-length: 211
connection: keep-alive
x-amzn-errortype: IncompleteSignatureException
Im assuming that x-amzn-errortype is referring to the missing organisation authentication variable.
Questions
How can I pass the variable "Organisation" to the function?
Or is there a better way to do this?
I think it will be way easier for you to use the package httr when you deal with HTTP requests. I think your example request could be:
library(httr)
POST("https://THEURL.com",
body = list(user_name = "<username>", password = "<mypassword>", organisation = "<organisation>"),
encode = "json")
Related
I'm trying to access JSON data (in R) from a REST API.
To authenticate myself, I need to use a POST method in https://dashboard.server.eu/login. The data that needs to be sent are email and password:
library(httr)
login <- list(
email = "my#email.com",
password = "mypass"
)
res <- POST("https://dashboard.server.eu/login", body = login, encode = "form", verbose())
When executing the above, I get this output:
-> POST /login HTTP/1.1
-> Host: dashboard.server.eu
-> User-Agent: libcurl/7.59.0 r-curl/3.3 httr/1.4.1
-> Accept-Encoding: gzip, deflate
-> Cookie: session=10kq9qv1udf0107F4C70RY14fsum41sq50
-> Accept: application/json, text/xml, application/xml, */*
-> Content-Type: application/x-www-form-urlencoded
-> Content-Length: 53
->
>> email=my%40email.com&password=mypass
<- HTTP/1.1 200 OK
<- access-control-allow-headers: Accept, Authorization, Content-Type, If-None-Match
<- access-control-allow-methods: HEAD, GET, POST, PUT, DELETE
<- cache-control: no-cache
<- content-encoding: gzip
<- content-type: application/json; charset=utf-8
<- date: Mon, 09 Mar 2020 14:58:31 GMT
<- set-cookie: session=10kq9qv1udf0107F4C70RY14fsum41sq50; HttpOnly; SameSite=Strict; Path=/
<- vary: origin,accept-encoding
<- x-microserv: NS4yNi4xODQuMjE3
<- x-poweredby: Poetry
<- Content-Length: 2346
<- Connection: keep-alive
The doc of the site says that, in case of success, a JSON res is returned and contains a string token in res.data._id.
I don't find it... even looking at every list (and sub-lists) of res.
How am I supposed to find the token?
Following the doc, and an example in AngularJS, I'm then supposed to do:
// Create JSON Object with your token
let authorizeObject = {
'Authorization': 'Session ' + token,
'content-type': 'application/json;charset=UTF-8',
'accept': 'application/json,text/plain',
};
// Create header from the previous JSON Object
let header = {'headers':authorizeObject};
// Use the header in your http request...
$http.get('https://dashboard.server.eu/', header)
Any hint on making this dream become true?
UPDATE -- With cURL, I could check that there is a _id key/value returned…
With the command:
curl -k -X POST "https://dashboard.server.eu/login" \
-d '{ "email" : "my#email.com", "password" : "mypass" }' \
-H "Content-Type: application/json"
I get the output:
{
"_id": "697v2on4ll0107F4C70RYhosfgtmhfug",
"isAuthenticated": true,
"user": {
"_id": "5dd57868d83cfc000ebbb273",
"firstName": "me",
"lastName": "Me",
...
So, the session token is indeed somewhere...
Does this help to help me?
Looking at the image of res in your question, the message is there, under content - it's just that the content is stored as a vector of raw bytes, which is why you didn't recognise it as json.
Since any file type can be sent by http, the contents in an httr response object are stored in raw format rather than a character string for various reasons - perhaps most importantly because many binary files will contain a 0x00 byte, which isn't allowed in a character string in R.
In your case, we can not only tell that res$content is text, but that it is your "missing" json. The first six bytes of res$content are shown in your image, and are 7b, 22, 5f, 69, 64, 22. We can convert these to a character string in R by doing:
rawToChar(as.raw(c(0x7b, 0x22, 0x5f, 0x69, 0x64, 0x22)))
[1] "{\"_id\""
This matches the first six characters of your expected json string.
Therefore if you do:
httr::content(res, "text")
or
rawToChar(res$content)
You will get your json as a character string.
I'm trying to use the sendgrid v3 API to purge bounces, it works fine when using CURL in CLI, here is the command:
curl -v -X DELETE -d '{"delete_all": true}' -H "Content-Type: application/json" -H "Authorization: Bearer SG.mykey" "https://api.sendgrid.com/v3/suppression/bounces"
But when trying to launch it with Symfony2 / Guzzle, I am getting a bad request error, however the request seems OK, here is the output of (string) $request:
"""
DELETE /v3/suppression/bounces HTTP/1.1\r\n
Host: api.sendgrid.com\r\n
Authorization: Bearer SG.mykey\r\n
User-Agent: Guzzle/3.9.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.17\r\n
Accept: application/json\r\n
Content-Length: 20\r\n
\r\n
{"delete_all": true}
"""
And the exception:
[Guzzle\Http\Exception\ClientErrorResponseException]
Client error response
[status code] 400
[reason phrase] BAD REQUEST
[url] https://api.sendgrid.com/v3/suppression/bounces
When using the GET method, it works correctly and it returns me all the bounces.
Here is the guzzle code:
$request = $this->httpClient->delete('/v3/suppression/bounces', null, '{"delete_all": true}');
$response = $request->send();
The http client is a service initialized with the https://api.sendgrid.com base URL.
Your problem I think is that you are sending 3 params to delete when it only has two, instead what you need to do is pass the body in options array.
$response = $this->httpClient->delete(
'/v3/suppression/bounces',
[
'body' => json_encode(['delete_all', true]),
'Authorization' => 'Basic ' . base64_encode($username . ':' . $password),
'content-type' => 'application/json'
]
);
Guzzle options docs
Answering to myself. The problem was pretty obvious: The content-type header was not set, the "accept" one was. I didn't care about this header because you don't have to pass it when using the GET method for this API. So now when debugging my request object I have:
"""
DELETE /v3/suppression/bounces HTTP/1.1\r\n
Host: api.sendgrid.com\r\n
Authorization: Bearer SG.mykey\r\n
Content-Type: application/json\r\n
Content-Length: 20\r\n
User-Agent: Guzzle/3.9.3 curl/7.35.0 PHP/5.5.9-1ubuntu4.17\r\n
Accept: application/json\r\n
\r\n
{"delete_all": true}
"""
I'm trying to make a GET call in R using httr and I keep getting an authorization 401 error.
R code:
testfunction2 <- function()
{
set_config(verbose())
locus_url <- "https://api.locusenergy.com/v3/clients/5599"
r <- GET(url = "https://api.locusenergy.com/v3/clients/5599",
query=list(authorization="Bearer c935845d8fc1124757e66ce04d2c75d0"),
Accept="application/json")
}
The results:
> print(testfunction2())
-> GET /v3/clients/5599
authorization=Bearer%20c935845d8fc1124757e66ce04d2c75d0 HTTP/1.1
-> User-Agent: libcurl/7.39.0 r-curl/0.9.1 httr/1.0.0
-> Host: api.locusenergy.com
-> Accept-Encoding: gzip, deflate
-> Cookie: AWSELB=D91FBFE1087EF6EBC125A126777051237474A8A060B6095B8E3C16151308453F8556B2A2E90CB2178F365FAA8AA8C29B124D15CA3EB859CFE615428E8D55C393ABB5B436BF
-> Accept: application/json, text/xml, application/xml, */*
->
<- HTTP/1.1 401 Unauthorized
<- Content-Type: application/json
<- Date: Sun, 16 Aug 2015 05:02:27 GMT
<- Server: Apache-Coyote/1.1
<- transfer-encoding: chunked
<- Connection: keep-alive
<-
I expect it to return a 200 code (rather than 401, that implies authorization error.)
I know the token is correct because it works if I use the Postman (google add-in) and Python. The token won't work for you because I changed it since I can't share it.
Python Code:
import http.client
conn = http.client.HTTPSConnection("api.locusenergy.com")
headers = {
'authorization': "Bearer 935845d8fc1124757e66ce04d2c75d0"
}
conn.request("GET", "/v3/clients/5599", headers=headers)
res = conn.getresponse()
data = res.read()
print(data)
results from Python
b'{"statusCode":200,"partnerId":4202,"tz":"US/Arizona","firstName":"xxx","lastName":"xxxx","email":"xxxx#aol.com","id":5599}'
So, again the question is what am I doing wrong in R or can you give me any hints? This won't be reproducible for you because the token expired and I can't share it.
Could it be the space in the authorization? authorization="Bearer 935845d8fc1124757e66ce04d2c75d0"? Are there any hints in the verbose output of the get call in R?
For reference, this is the site's API page:
https://developer.locusenergy.com/
The site requires OAUTH2 authentication to return the token. I didn't include that code but I verified the token works with Python and Postman.
Right now you are passing your authorization values in the query string of the httr code and not in the http header as you are doing in the python code. Instead use
GET(url = "https://api.locusenergy.com/v3/clients/5599",
accept_json(),
add_headers(Authorization="Bearer c935845d8fc1124757e66ce04d2c75d0")
)
In Windows cURL I can post a web request similar to this:
curl --dump-header cook.txt ^
--data "RURL=http=//www.example.com/r&user=bob&password=hello" ^
--user-agent "Mozilla/5.0" ^
http://www.example.com/login
With type cook.txt I get a response similar to this:
HTTP/1.1 302 Found
Date: Thu, ******
Server: Microsoft-IIS/6.0
SERVER: ******
X-Powered-By: ASP.NET
X-AspNet-Version: 1.1.4322
Location: ******
Set-Cookie: Cookie1=; domain=******; expires=****** ******
******
******
Cache-Control: private
Content-Type: text/html; charset=iso-8859-1
Content-Length: 189
I can manually read cookie lines like: Set-Cookie: AuthCode=ABC... (I could script this of course). So I can use AuthCode for subsequent requests.
I am trying do the same in R with RCurl and/or httr (still don't know which one is better for my task).
When I try:
library(httr)
POST("http://www.example.com/login",
body= list(RURL="http=//www.example.com/r",
user="bob", password="hello"),
user_agent("Mozilla/5.0"))
I get a response similar to this:
Response [http://www.example.com/error]
Status: 411
Content-type: text/html
<h1>Length Required</h1>
By and large I know about 411-error and I could try to fix the request; but I do not get it in cURL, so I am doing something wrong with the POST command.
Can you help me in translating my cURL command to RCurl and/or httr?
httr automatically preserves cookies across calls to the same site, as illustrated by these two calls to http://httpbin.org
GET("http://httpbin.org/cookies/set?a=1")
# Response [http://httpbin.org/cookies]
# Status: 200
# Content-type: application/json
# {
# "cookies": {
# "a": "1"
# }
# }
GET("http://httpbin.org/cookies")
# Response [http://httpbin.org/cookies]
# Status: 200
# Content-type: application/json
# {
# "cookies": {
# "a": "1"
# }
# }
Perhaps the problem is that you're sending your data as application/x-www-form-urlencoded, but the default in httr is multipart/form-data, so use multipart = FALSE in your POST call.
Based on Juba suggestion, here is a working RCurl template.
The code emulates a browser behaviour, as it:
retrieves cookies on a login screen and
reuses them on the following page requests containing the actual data.
### RCurl login and browse private pages ###
library("RCurl")
loginurl ="http=//www.*****"
mainurl ="http=//www.*****"
agent ="Mozilla/5.0"
#User account data and other login pars
pars=list(
RURL="http=//www.*****",
Username="*****",
Password="*****"
)
#RCurl pars
curl = getCurlHandle()
curlSetOpt(cookiejar="cookiesk.txt", useragent = agent, followlocation = TRUE, curl=curl)
#or simply
#curlSetOpt(cookiejar="", useragent = agent, followlocation = TRUE, curl=curl)
#post login form
web=postForm(loginurl, .params = pars, curl=curl)
#go to main url with real data
web=getURL(mainurl, curl=curl)
#parse/print content of web
#..... etc. etc.
#This has the side effect of saving cookie data to the cookiejar file
rm(curl)
gc()
Here is a way to create a post request, keep and reuse the resulting cookies with RCurl, for example to get web pages when authentication is required :
library(RCurl)
curl <- getCurlHandle()
curlSetOpt(cookiejar="/tmp/cookies.txt", curl=curl)
postForm("http://example.com/login", login="mylogin", passwd="mypasswd", curl=curl)
getURL("http://example.com/anotherpage", curl=curl)
This is the Google Apps Script that makes the request:
UrlFetchApp.fetch('https://user:password#sitename.com/api', {
method: 'put',
headers: { 'Accept': '*/*' },
payload: 'foo=bar&baz=qux'
});
Which can be successfully posted to http://requestb.in/, for examination:
PUT /1bfk94g1 HTTP/1.1
User-Agent: Mozilla/5.0 (compatible; GoogleDocs; script; +http://docs.google.com)
Host: requestb.in
Content-Type: application/x-www-form-urlencoded
Content-Length: 43
Connection: keep-alive
Accept-Encoding: gzip
Accept: */*
foo=bar&baz=qux
But fails when the request URL is https://user:password#sitename.com/api. The only error information shown is Bad request: https://user:password#sitename.com/api.
I've constructed a curl command which yields the exact same HTTP request:
curl -XPUT \
-H 'Accept-Encoding: gzip' \
--user-agent 'Mozilla/5.0 (compatible; GoogleDocs; script; +http://docs.google.com)' \
-d foo='bar' \
-d baz='qux'\
https://user:password#sitename.com/api
Which successfully posts to https://user:password#sitename.com/api. How can two identical requests have different outcomes? Am I missing something with regards to my Google Apps Script? I've tried using the debugger, but it didn't help.
It appears that UrlFetchApp doesn't yet support requests with credentials in the URL. The Authorization HTTP header needs to be built manually. The following works:
var response = UrlFetchApp.fetch('https://sitename.com/api', {
headers: {
'Authorization': 'Basic ' + Utilities.base64Encode(user + ':' + password)
}
});