Dealing with gzip encoded GET/OAUTH response in R - r

I'm new to both: R and OAUTH. I've learned a little using coursera examples on github API where OAUTH request gave plaintext response but now I'm trying to do something that is practicall for me and access EVE-Online CREST OAUTH API but instead of what I got when I tried github API (im using "httr" libary):
Response [https://api.github.com/users/jtleek/repos]
Date: 2014-12-14 08:57
Status: 200
Content-type: application/json; charset=utf-8
Size: 154 kB
[
{
"id": 12441219,
"name": "ballgown",
"full_name": "jtleek/ballgown",
"owner": {
"login": "jtleek",
"id": 1571674,
"avatar_url": "https://avatars.githubusercontent.com/u/1571674?v=3",
"gravatar_id": "",
...
I got this BINARY BODY response:
Response [https://crest-tq.eveonline.com/market/10000002/orders/buy/?type=https://crest-tq.eveonline.com/types/185/]
Date: 2014-12-14 08:05
Status: 200
Content-type: application/vnd.ccp.eve.MarketOrderCollection-v1+json; charset=utf-8
Size: 7.61 kB
<BINARY BODY>
And frankly I have no idea what to do with it. I'm preety sure its gzip (I used chrome extension postman to access the same information and header says its encoded with gzip) but I dont know how to uncompress it, maybe there is standard way of dealing with binary/gzip response but my google foo have failed me.
Here is exact code I'm running:
library(httr)
myapp <- oauth_app("my app name redacted", "my id redacted", "my secret redacted")
eve_token <- oauth2.0_token(oauth_endpoint(authorize = "https://login-tq.eveonline.com/oauth/authorize/",access = "https://login-tq.eveonline.com/oauth/token/"), myapp, scope = "publicData")
token <- config(token = eve_token)
req <- GET("https://crest-tq.eveonline.com/market/10000002/orders/buy/?type=https://crest-tq.eveonline.com/types/185/", token)
EDIT:
YES!!! :)
managed to figure it out :)
result <- content(req, type = "application/json; charset=utf-8")
while the reqular content(req) produced just raw binary data, the above translated it to json :)

Like I wrote above, what I needed to do was pass more information about content type and encoding used to content function like this:
result <- content(req, type = "application/json; charset=utf-8")
gzip part as its turned out was handled automagically, but the issue was strage content-type used by EVE API. when i explicitly passed desired content type R was able to read data as json without problem

Related

how to get data from the WTO API in R

library(httr)
library(jsonlite)
headers = c(
# Request headers
'Ocp-Apim-Subscription-Key' = '{subscription key}'
)
params = list()
# Request parameters
params['countries[]'] = '{array}'
resp <- GET(paste0("https://api.wto.org/tfad/transparency/procedures_contacts_single_window?"
, paste0(names(params),'=',params,collapse = "&")),
add_headers(headers))
if(!http_error(resp)){
jsonRespText<-fromJSON(rawToChar(content(resp,encoding = 'UTF-8')))$Dataset
jsonRespText
}else{
stop('Error in Response')
}
I don't know how to get response from an API in R. I have executed this code but the server is not responding...
If you examine the value of the resp object after running your code you'll notice a status code:
> resp
Response [https://tfadatabase.org/api/transparency/procedures_contacts_single_window?countries[]=%7Barray%7D]
Date: 2020-04-17 19:25
Status: 422
Content-Type: application/json
Size: 77 B
So the server actually did respond, it just didn't give you what you were hoping for. In the API documentation we can look up this code:
422 Unprocessable Entity
If a member cannot be found, or the request parameters are poorly
formed.
So I just went to the Query Builder and looked for a valid request URL and updated the code. It ran fine - i.e. Status 200.
This was the URL I used in the code:
https://api.wto.org/timeseries/v1/data?i=TP_A_0100&r=000&fmt=json&mode=full&lang=1&meta=false
and the value of resp was
Date: 2020-04-17 19:30
Status: 200
Content-Type: application/json; charset=utf-8
Size: 88 B
I cut out the subscription key in my results above. You can find the Query Builder here. Incidentally, in the Query Builder it automatically includes the subscription key and other "header" info in the URL. You can either remove that first and re-add it in your code, or just change your code to run GET() directly on their version of the URL.

Send a POST request in R package

I've been trying to use httr package in R to send a POST request
The website which I'm trying to send a POST request states to do the following:
POST /newwebservices/locationverifier.asmx/findLocation2 HTTP/1.1
Host: citizenatlas.dc.gov
Content-Type: application/x-www-form-urlencoded
Content-Length: length
str=string&f=string
where str takes an address in Washington DC such as "701 FARADAY PL NE, WASHINGTON, DC 20017" and f takes the format requested in this case I want "json"
I tried :
url = "/newwebservices/locationverifier.asmx HTTP/1.1"
body = list(
"Host" = "citizenatlas.dc.gov",
"Content-Type" = "application/x-www-form-urlencoded",
"Content-Length" = length,
"str"="3513 S St NW",
"f"="json"
)
httr::POST(url,body, verbose)
I also tried Get
This is what they state to put in
GET /newwebservices/locationverifier.asmx/findLocation2? str=string&f=string HTTP/1.1
Host: citizenatlas.dc.gov
or
GET("http://citizenatlas.dc.gov/newwebservices/locationverifier.asmx/findLocation2?str=3513 S St NW&f=json")
but nothing is working. By the way, I had no problem making the request using Mathematica.
I will appreciate any help. Thank You in Advance.
There are a few different things going on here. Below works:
url <- "http://citizenatlas.dc.gov/newwebservices/locationverifier.asmx/findLocation2"
body <- list(str = "3513 S St NW", f = "json")
httr::POST(url, body = body, encode = "json")
A few differences:
POST is to host + endpoint, not just endpoint
Details on response (content type, length) are not included
Encoding specified for body (per httr::POST documentation)
body and encode are named because they aren't in positions from documentation, so positional argument matching won't work

Calling a REST API in R

I recently discovered the dataforseo api and tryed to call it via R
library(httr)
username <- 'mygmailadress#gmail.com'
password <- 'mypassword'
dataforseo_api <- POST('https://api.dataforseo.com/v2/op_tasks_post/$data',
authenticate(username,password),
body = list(grant_type = 'client_credentials'),
type = "basic",
verbose()
)
This is the message I have received:
<- HTTP/1.1 401 Unauthorized
<- Server: nginx/1.14.0 (Ubuntu)
<- Date: Sun, 08 Jul 2018 13:31:34 GMT
<- Content-Type: application/json
<- Transfer-Encoding: chunked
<- Connection: keep-alive
<- WWW-Authenticate: Basic realm="Rest Server"
<- Cache-Control: no-cache, must-revalidate
<- Expires: 0
<- Access-Control-Allow-Origin: *
<- Access-Control-Allow-Methods: POST, GET, OPTIONS
<- Access-Control-Allow-Headers: Content-Type, Access-Control-Allow-Headers, Authorization, X-Requested-With
Do you know where my issue should come? Can you please help?
It looks like you're improperly configuring config. I don't see a config= in your code. The body is also not encoded correctly.
Also, in the API documentation I don't see anything about grant_type. It looks like an array of tasks should go there, e.g. something like:
{882394209: {'site': 'ranksonic.com', 'crawl_max_pages': 10}}
Response:
{'results_count': 1, 'results_time': '0.0629 sec.', 'results': {'2308949': {'post_id': 2308949, 'post_site': 'ranksonic.com',
'task_id': 882394209, 'status': 'ok'}}, 'status': 'ok'}
OK, so first off we need set_config or config=:
username <- 'Hack-R#stackoverflow.com' # fake email
password <- 'vxnyM9s7FAKESeIO' # fake password
set_config(authenticate(username,password), override = TRUE)
GET("https://api.dataforseo.com/v2/cmn_se")
Response [https://api.dataforseo.com/v2/cmn_se]
Date: 2018-07-08 16:20
Status: 200
Content-Type: application/json
Size: 551 kB
{
"status": "ok",
"results_time": "0.0564 sec.",
"results_count": 2187,
"results": [
{
"se_id": 37,
"se_name": "google.com.af",
"se_country_iso_code": "AF",
"se_country_name": "Afghanistan",
...
GET("https://api.dataforseo.com/v2/cmn_se/$country_iso_code")
Response [https://api.dataforseo.com/v2/cmn_se/$country_iso_code]
Date: 2018-07-08 15:48
Status: 200
Content-Type: application/json
Size: 100 B
{
"status": "ok",
"results_time": "0.0375 sec.",
"results_count": 0,
"results": []
GET("https://api.dataforseo.com/v2/cmn_se/$op_tasks_post")
Response [https://api.dataforseo.com/v2/cmn_se/$op_tasks_post]
Date: 2018-07-08 16:10
Status: 200
Content-Type: application/json
Size: 100 B
{
"status": "ok",
"results_time": "0.0475 sec.",
"results_count": 0,
"results": []
That was one thing. Also to POST data they need you to specify it as json, e.g. encode = "json". From their docs:
All POST data should be sent in the JSON format (UTF-8 encoding). The
keywords are sent by POST method passing tasks array. The data should
be specified in the data field of this POST array. We recommend to
send up to 100 tasks at a time.
Further:
The task setting is done using POST method when array of tasks is sent to
the data field. Each of the array elements has the following
structure:
then it goes on to list 2 required fields and many optional ones.
Note also that you can use reset_config() after as a better practice. If you're going to be running this a lot, sharing it, or using more than 1 computer I would also suggest to put your credentials in environment variables instead of your script for security and ease.
Another final word of advice is that you may want to just leverage their published Python client library and large compilation of examples. Since every new API request is something you'll be pioneering in R without their support, it may pay off to just do the data collection in Python.
This is an interesting API. If you get over to the Open Data Stack Exchange you should consider sharing it with that community.

Refresh Token for Access Token Google API: R Code

I am attempting to retrieve an access token using my refresh token, client id and client secret for the youtube api using R Code.
This is google's example of how to POST a request.
POST /o/oauth2/token HTTP/1.1 Host: accounts.google.com Content-Type: application/x-www-form-urlencoded client_id=21302922996.apps.googleusercontent.com&client_secret=XTHhXh1SlUNgvyWGwDk1EjXB&refresh_token=1/6BMfW9j53gdGImsixUH6kU5RsR4zwI9lUVX-tqf8JXQ&grant_type=refresh_token
This was my r code:
library(httr)
url<- paste("https://accounts.google.com/o/oauth2/token?client_id=", client_id, "&client_secret=", client_secret, "&refresh_token=", refresh_token, "&grant_type=access_token", sep="")
POST(url)
And I keep getting this response:
Response [https://accounts.google.com/o/oauth2/token?client_id=xxxxxxxxxx&client_secret=xxxxxxxx&refresh_token=xxxxxxxxxxxxxxxxxxxxxx&grant_type=refresh_token]
Date: 2015-09-02 16:43
Status: 400
Content-Type: application/json
Size: 102 B
{
"error" : "invalid_request",
"error_description" : "Required parameter is missing: grant_type"
Is there a better way to do this? Maybe using RCurl? If so, what would the format of the request be? I would appreciate help on this!
The RAdwords package has a function to retrieve the refresh token. If you don't want to add the entire package you can just add the following code to your script.
refreshToken = function(google_auth) {
# This function refreshes the access token.
# The access token deprecates after one hour and has to updated
# with the refresh token.
#
# Args:
# access.token$refreh_token and credentials as input
# Returns:
# New access.token with corresponding time stamp
rt = rjson::fromJSON(RCurl::postForm('https://accounts.google.com/o/oauth2/token',
refresh_token=google_auth$access$refresh_token,
client_id=google_auth$credentials$c.id,
client_secret=google_auth$credentials$c.secret,
grant_type="refresh_token",
style="POST",
.opts = list(ssl.verifypeer = FALSE)))
access <- rt
access
}

What am I doing wrong in this QBO v3 API (IPP) Attachments upload python request?

Intuit offers these instructions for uploading attachments (which become Attachable objects that can be associated with one or more transactions).
I believe I'm using python's requests module (via rauth's OAuth1Session module—see below for how I'm creating the session object) to generate these requests. Here's the code leading up to the request:
print request_type
print url
print headers
print request_body
r = session.request(request_type, url, header_auth,
self.company_id, headers = headers,
data = request_body, **req_kwargs)
result = r.json()
print json.dumps(result, indent=4)
and the output of these things:
POST
https://quickbooks.api.intuit.com/v3/company/0123456789/upload
{'Accept': 'application/json'}
Content-Disposition: form-data; name="Invoice 003"; filename="Invoice 003.pdf"
Content-Type: application/pdf
<#INCLUDE */MyDir/Invoice 003.pdf*#>
{
"Fault": {
"type": "SystemFault",
"Error": [
{
"Message": "An application error has occurred while processing your request",
"code": "10000",
"Detail": "System Failure Error: Cannot consume content type"
}
]
},
"time": "[timestamp]"
}
I have confirmed (by uploading an attachment through the QBO web UI and then querying the Attachable object through the API) that application/pdf is included in the list of acceptable file types.
At sigmavirus24's suggestion, I tried removing the Content-Type line from the headers, but I got the same result.
Here's how I'm creating the session object (which, again, is working fine for other QBO v3 API requests of every type you see in Intuit's API Explorer):
from rauth import OAuth1Session
def create_session(self):
if self.consumer_secret and self.consumer_key and self.access_token_secret and self.access_token:
session = OAuth1Session(self.consumer_key,
self.consumer_secret,
self.access_token,
self.access_token_secret,
)
self.session = session
else:
raise Exception("Need four creds for Quickbooks.create_session.")
return self.session
What might I be missing here?
EDIT: current area of exploration is here; I just formed the header you see (that has the "INCLUDE" string there) directly. Perhaps I should be using rauth to attach the file...
Without being able to see what code you're using with requests, I'm going to take a shot in the dark and tell you to remove setting your own Content-Type. You probably don't want that. It looks like you want multipart/form-data and requests will set that on its own if you stop fighting it.
It looks like you're missing the boundaries that QuickBooks is expecting (based on what you linked).
---------------------------acebdf13572468
Content-Disposition: form-data; name="file_content_01"; filename="IMG_0771.jpg"
Content-Type: image/jpeg
<#INCLUDE *Y:\Documents\IMG_0771.jpg*#>
---------------------------acebdf13572468--
The first and last line above seem to be what you're missing.

Resources