I want to call Amazon product advertising API from R. Given below is the quick guide of pa API 5.
https://webservices.amazon.com/paapi5/documentation/quick-start/using-curl.html
I tried to do the way it described here https://webservices.amazon.com/paapi5/documentation/sending-request.html using 'httr' but got thrown off on the signature version 4 signing process.
I tried using 'aws.signature' package for signature prior to calling the POST function, but the final output I am getting is status code 500.
Here is the code I have used
library(httr)
library(jsonlite)
library(aws.signature)
request_body=data.frame("Keywords"="Harry",
"Marketplace"= "www.amazon.com",
"PartnerTag"= "mytag-20",
"PartnerType"= "Associates",
"Access Key"="my_accesskey",
"Secret Key"="my_secret_key",
"service"="ProductAdvertisingAPIv1",
"Region"="us-east-1"
"Resources"="Offers.Listings.Price",
"SearchIndex"= "All")
request_body_json=toJSON(request_body,auto_unbox=T)
request_body_json=gsub("\\[|\\]","",request_body_json)
t=signature_v4_auth(
datetime = format(Sys.time(), "%Y%m%dT%H%M%SZ", tz = "UTC"),
region = NULL,
service="ProductAdvertisingAPIv1",
verb="POST",
"com.amazon.paapi5.v1.ProductAdvertisingAPIv1.SearchItems",
query_args = list(),
canonical_headers=c("Host: webservices.amazon.com",
"Content-Type: application/json; charset=UTF-8",
"X-Amz-Target: com.amazon.paapi5.v1.ProductAdvertisingAPIv1.SearchItems",
"Content-Encoding: amz-1.0",
"User-Agent: paapi-docs-curl/1.0.0"),
request_body=request_body_json,
signed_body = TRUE,
key = "access_key",
secret = "secret-key",
session_token = NULL,
query = FALSE,
algorithm = "AWS4-HMAC-SHA256",
force_credentials = FALSE,
verbose = getOption("verbose", FALSE)
)
result=POST("https://webservices.amazon.com/paapi5/searchitems",body=request_body_json,
add_headers(.headers=c("Host: webservices.amazon.com",
"Content-Type: application/json; charset=UTF-8",
paste("X-Amz-Date:",format(Sys.time(), "%Y%m%dT%H%M%SZ", tz = "UTC")),
"X-Amz-Target: com.amazon.paapi5.v1.ProductAdvertisingAPIv1.SearchItems",
"Content-Encoding: amz-1.0",
"User-Agent: paapi-docs-curl/1.0.0",
paste0("Authorization: AWS4-HMAC-SHA256 Credential=",t[["Credential"]],"SignedHeaders=content-encoding;host;x-amz-date;x-amz-target Signature=",t[["Signature"]])
)))
Appreciate it if anyone one can help with this. Thanks
Related
I was wondering whether anyone has found a way to how to use search_fullarchive() from the "rtweet" package in R with the new Twitter academic research project track?
The problem is whenever I try to run the following code:
search_fullarchive(q = "sunset", n = 500, env_name = "AcademicProject", fromDate = "202010200000", toDate = "202010220000", safedir = NULL, parse = TRUE, token = bearer_token)
I get the following error "Error: Not a valid access token". Is that because search_fullarchive() is only for paid premium accounts and that doesn't include the new academic track (even though you get full archive access)?
Also, can you retrieve more than 500 tweets (e.g., n = 6000) when using search_fullarchive()?
Thanks in advance!
I've got the same problem w/ Twitter academic research API. I think if you set n = 100 or just skip the argument, the command will return you 100 tweets. Also, the rtweet package does not (yet) support the academic research API.
Change your code to this:
search_fullarchive(q = "sunset", n = 500, env_name = "AcademicProject", fromDate = "202010200000", toDate = "202010220000", safedir = NULL, parse = TRUE, token = t, env_name = "Your Environment Name attained in the Dev Dashboard")
Also The token must be created like this:
t <- create_token(
app = "App Name",
'Key',
'Secret',
access_token = '',
access_secret = '',
set_renv = TRUE
)
I'm trying to access Bitfinex via api but struggling to properly authenticate my request. There is a Python example of what I want to do (https://gist.github.com/jordanbaucke/5812039) but I can't seem to get it to work in R.
key <- c("MY API KEY")
secret = c("MY API SECRET")
req <- GET("https://api.bitfinex.com/v1/balances",
authenticate(key, secret))
add_headers(X-BFX-APIKEY = key))
stop_for_status(req)
content(req)
Can someone tell me what I'm doing wrong?
/v1/balances is a Bitfinex authenticated endpoints, thus it requires the POST request with a proper handling of payload and headers.
Here is a working example from my own script:
library(httr)
key <- "..."
secret <- "..."
# payload JSON object, the request should refer to the URL
# nonce should always be greater than for previous calls
payload_json <- list(request = '/v1/account_infos', nonce = as.character(as.numeric(as.POSIXct(Sys.time()))))
# creating string from JSON payload
payload <- jsonlite::toJSON(payload_json, auto_unbox = TRUE)
# encoding payload string
payload_str <- base64enc::base64encode(charToRaw(as.character(payload)))
# adding three Bitfinex headers:
# X-BFX-APIKEY = key
# X-BFX-PAYLOAD = base64 encoded payload string
# X-BFX-SIGNATURE = sha384 encrypted payload string with the secret key
req <- POST("https://api.bitfinex.com/v1/account_infos",
add_headers('X-BFX-APIKEY' = key,
'X-BFX-PAYLOAD' = payload_str,
'X-BFX-SIGNATURE' = openssl::sha384(payload_str, key = secret))
)
content(req)
For creating "New Order" you only need to change the payload to something, like this:
payload_json <- list(request = '/v1/account_infos',
nonce = as.character(as.numeric(as.POSIXct(Sys.time()))),
symbol = 'BTCUSD',
amount = '0.3',
price = '1000',
exchange = 'bitfinex',
side = 'sell',
type = 'exchange market'
)
The rest of the code will work without changes.
For list of payload parameters check the Bitfinex API docs, e.g. "New Order".
I am using R, version 3.3.1. I am trying to scrap data from following web site:
http://plovila.pomorstvo.hr/
As you can see, it is a HTML form. I would like to choose "Tip objekta" (object type), for example "Jahta" (Yacht) and enter "NIB" (which is an integer, eg. 93567). You can try yourself; just choose "Jahta" and type 93567 in NIB field.
Method is POST, type application/x-www-form-urlencoded. I have tried 3 different approaches: using rvest, POST (httr package) and postForm (Rcurl). My rvest code is:
session <- html_session("http://plovila.pomorstvo.hr")
form <- html_form(session)[[1]]
form <- set_values(form, `ctl00$Content_FormContent$uiTipObjektaDropDown` = 2,
`ctl00$Content_FormContent$uiOznakaTextBox` = "",
`ctl00$Content_FormContent$uiNibTextBox` = 93567)
x <- submit_form(session, form)
If I run this code and get 200 status but I don't understand how can I get the table:
Additional step is to submit Detalji button and get additional information, but I can't see any information from x submit output.
I used the curlconverter package to take the "Copy as cURL" data from the XHR POST request and turn it automagically into:
httr::VERB(verb = "POST", url = "http://plovila.pomorstvo.hr/",
httr::add_headers(Origin = "http://plovila.pomorstvo.hr",
`Accept-Encoding` = "gzip, deflate",
`Accept-Language` = "en-US,en;q=0.8",
`X-Requested-With` = "XMLHttpRequest",
Connection = "keep-alive",
`X-MicrosoftAjax` = "Delta=true",
Pragma = "no-cache", `User-Agent` = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.34 Safari/537.36",
Accept = "*/*", `Cache-Control` = "no-cache",
Referer = "http://plovila.pomorstvo.hr/",
DNT = "1"), httr::set_cookies(ASP.NET_SessionId = "b4b123vyqxnt4ygzcykwwvwr"),
body = list(`ctl00$uiScriptManager` = "ctl00$Content_FormContent$ctl00|ctl00$Content_FormContent$uiPretraziButton",
ctl00_uiStyleSheetManager_TSSM = ";|635908784800000000:d29ba49:3cef4978:9768dbb9",
`ctl00$Content_FormContent$uiTipObjektaDropDown` = "2",
`ctl00$Content_FormContent$uiImeTextBox` = "",
`ctl00$Content_FormContent$uiNibTextBox` = "93567",
`__EVENTTARGET` = "", `__EVENTARGUMENT` = "",
`__LASTFOCUS` = "", `__VIEWSTATE` = "/wEPDwUKMTY2OTIzNTI1MA9kFgJmD2QWAgIDD2QWAgIBD2QWAgICD2QWAgIDD2QWAmYPZBYIAgEPZBYCZg9kFgZmD2QWAgIBDxAPFgYeDURhdGFUZXh0RmllbGQFD05heml2VGlwT2JqZWt0YR4ORGF0YVZhbHVlRmllbGQFDElkVGlwT2JqZWt0YR4LXyFEYXRhQm91bmRnZBAVBAAHQnJvZGljYQVKYWh0YQbEjGFtYWMVBAEwATEBMgEzFCsDBGdnZ2cWAQICZAIBDw8WAh4HVmlzaWJsZWdkFgICAQ8PFgIfA2dkZAICDw8WAh8DaGQWAgIBDw8WBB4EVGV4dGUfA2hkZAIHDzwrAA4CABQrAAJkFwEFCFBhZ2VTaXplAgoBFgIWCw8CCBQrAAhkZGRkZDwrAAUBBAUHSWRVcGlzYTwrAAUBBAUISWRVbG9za2E8KwAFAQQFBlNlbGVjdGRlFCsAAAspelRlbGVyaWsuV2ViLlVJLkdyaWRDaGlsZExvYWRNb2RlLCBUZWxlcmlrLldlYi5VSSwgVmVyc2lvbj0yMDEzLjMuMTExNC40MCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2tlbj0xMjFmYWU3ODE2NWJhM2Q0ATwrAAcACyl1VGVsZXJpay5XZWIuVUkuR3JpZEVkaXRNb2RlLCBUZWxlcmlrLldlYi5VSSwgVmVyc2lvbj0yMDEzLjMuMTExNC40MCwgQ3VsdHVyZT1uZXV0cmFsLCBQdWJsaWNLZXlUb2tlbj0xMjFmYWU3ODE2NWJhM2Q0ARYCHgRfZWZzZGQWBB4KRGF0YU1lbWJlcmUeBF9obG0LKwQBZGZkAgkPZBYCZg9kFgJmD2QWIAIBD2QWBAIDDzwrAAgAZAIFDzwrAAgAZAIDD2QWBAIDDzwrAAgAZAIFDzwrAAgAZAIFD2QWAgIDDzwrAAgAZAIHD2QWBAIDDzwrAAgAZAIFDzwrAAgAZAIJD2QWBAIDDzwrAAgAZAIFDzwrAAgAZAILD2QWBgIDDxQrAAI8KwAIAGRkAgUPFCsAAjwrAAgAZGQCBw8UKwACPCsACABkZAIND2QWBgIDDxQrAAI8KwAIAGRkAgUPFCsAAjwrAAgAZGQCBw8UKwACPCsACABkZAIPD2QWAgIDDxQrAAI8KwAIAGRkAhEPZBYGAgMPPCsACABkAgUPPCsACABkAgcPPCsACABkAhMPZBYGAgMPPCsACABkAgUPPCsACABkAgcPPCsACABkAhUPZBYCAgMPPCsACABkAhcPZBYGAgMPPCsACABkAgUPPCsACABkAgcPPCsACABkAhkPPCsADgIAFCsAAmQXAQUIUGFnZVNpemUCBQEWAhYLZGRlFCsAAAsrBAE8KwAHAAsrBQEWAh8FZGQWBB8GZR8HCysEAWRmZAIbDzwrAA4CABQrAAJkFwEFCFBhZ2VTaXplAgUBFgIWC2RkZRQrAAALKwQBPCsABwALKwUBFgIfBWRkFgQfBmUfBwsrBAFkZmQCHQ88KwAOAgAUKwACZBcBBQhQYWdlU2l6ZQIFARYCFgtkZGUUKwAACysEATwrAAcACysFARYCHwVkZBYEHwZlHwcLKwQBZGZkAiMPPCsADgIAFCsAAmQXAQUIUGFnZVNpemUCBQEWAhYLZGRlFCsAAAsrBAE8KwAHAAsrBQEWAh8FZGQWBB8GZR8HCysEAWRmZAILD2QWAmYPZBYCZg9kFgICAQ88KwAOAgAUKwACZBcBBQhQYWdlU2l6ZQIFARYCFgtkZGUUKwAACysEATwrAAcACysFARYCHwVkZBYEHwZlHwcLKwQBZGZkZIULy2JISPTzELAGqWDdBkCVyvvKIjo/wm/iG9PT1dlU",
`__VIEWSTATEGENERATOR` = "CA0B0334",
`__PREVIOUSPAGE` = "jGgYHmJ3-6da6PzGl9Py8IDr-Zzb75YxIFpHMz4WQ6iQEyTbjWaujGRHZU-1fqkJcMyvpGRkWGStWuj7Uf3NYv8Wi0KSCVwn435kijCN2fM1",
`__ASYNCPOST` = "true",
`ctl00$Content_FormContent$uiPretraziButton` = "Pretraži"),
encode = "form") -> res
You can see the result of that via:
content(res, as="text") # returns raw HTML
or
content(res, as="parsed") # returns something you can use with `rvest` / `xml2`
Unfortunately, this is yet another useless SharePoint website that "eGov" sites around the world have bought into as a good thing to do. That means you have to do trial and error to figure out which of those parameters is necessary since it's different on virtually every site. I tried a minimal set to no avail.
You may even have to issue a GET request to the main site first to establish a session.
But this should get you going in the right direction.
I'm following the official manual of opencpu package in R. In chapter 4.3 Calling a function It uses curl to test API:
curl http://your.server.com/ocpu/library/stats/R/rnorm -d "n=10&mean=100"
and the sample output is:
/ocpu/tmp/x032a8fee/R/.val
/ocpu/tmp/x032a8fee/stdout
/ocpu/tmp/x032a8fee/source
/ocpu/tmp/x032a8fee/console
/ocpu/tmp/x032a8fee/info
I can use curl to get similar result, but when I try to send this http request using httr package in R, I don't know how to replicate the result. Here is what I tried:
resp <- POST(
url = "localhost/ocpu/library/stats/R/rnorm",
body= "n=10&mean=100"
)
resp
the output is:
Response [HTTP://localhost/ocpu/library/stats/R/rnorm]
Date: 2015-10-16 00:51
Status: 400
Content-Type: text/plain; charset=utf-8
Size: 30 B
No Content-Type header found.
I guess I don't understand what's the equivalence of curl -d parameter in httr, how can I get it correct?
Try this :)
library(httr)
library(jsonlite)
getFunctionEndPoint <- function(url, format) {
return(paste(url, format, sep = '/'))
}
resp <- POST(
url = getFunctionEndPoint(
url = "https://public.opencpu.org/ocpu/library/stats/R/rnorm",
format = "json"),
body = list(n = 10, mean = 100),
encode = 'json')
fromJSON(rawToChar(resp$content))
I'm trying to POST data to JIRA Project using R and I keep getting: Error Bad Request. At first I thought it must be the JSON format that I created. So I wrote the JSON to file and did a curl command from console (see below) and the POST worked just fine.
curl -D- -u fred:fred -X POST -d #sample.json -H "Content-Type: application/json" http://localhost:8090/rest/api/2/issue/
Which brings the issue to my R code. Can someone tell me what am I doing wrong with the RCurl postForm?
Source:
library(RJSONIO)
library(RCurl)
x <- list (
fields = list(
project = c(
c(key="TEST")
),
summary="The quick brown fox jumped over the lazy dog",
description = "silly old billy",
issuetype = c(name="Task")
)
)
curl.opts <- list(
userpwd = "fred:fred",
verbose = TRUE,
httpheader = c('Content-Type' = 'application/json',Accept = 'application/json'),
useragent = "RCurl"
)
postForm("http://jirahost:8080/jira/rest/api/2/issue/",
.params= c(data=toJSON(x)),
.opts = curl.opts,
style="POST"
)
rm(list=ls())
gc()
Here's the output of the response:
* About to connect() to jirahost port 80 (#0)
* Trying 10.102.42.58... * connected
* Connected to jirahost (10.102.42.58) port 80 (#0)
> POST /jira/rest/api/2/issue/ HTTP/1.1
User-Agent: RCurl
Host: jirahost
Content-Type: application/json
Accept: application/json
Content-Length: 337
< HTTP/1.1 400 Bad Request
< Date: Mon, 07 Apr 2014 19:44:08 GMT
< Server: Apache-Coyote/1.1
< X-AREQUESTID: 764x1525x1
< X-AUSERNAME: anonymous
< Cache-Control: no-cache, no-store, no-transform
< Content-Type: application/json;charset=UTF-8
< Set-Cookie: atlassian.xsrf.token=B2LW-L6Q7-15BO- MTQ3|bcf6e0a9786f879a7b8df47c8b41a916ab51da0a|lout; Path=/jira
< Connection: close
< Transfer-Encoding: chunked
<
* Closing connection #0
Error: Bad Request
You might find it easier to use httr which has been constructed with
the needs of modern APIs in mind, and tends to set better default
options. The equivalent httr code would be:
library(httr)
x <- list(
fields = list(
project = c(key = "TEST"),
summary = "The quick brown fox jumped over the lazy dog",
description = "silly old billy",
issuetype = c(name = "Task")
)
)
POST("http://jirahost:8080/jira/rest/api/2/issue/",
body = RJSONIO::toJSON(x),
authenticate("fred", "fred", "basic"),
add_headers("Content-Type" = "application/json"),
verbose()
)
If that doesn't work, you'll need to supply the output from a successful
verbose curl on the console, and a failed httr call in R.