I am trying to get data from Amazon MWS API using GetMatchingProductForId operation.
When I use Amazon MWS Scratchpad it works perfectly fine.
I am now trying to replicate the urls that are sent in the HTTP POST request but I get a Signature error message.
I need to understand how the url request should be structured.
Below is the detail of the request in Amazon MWS Scratchpad, I ANONYMIZED private identifiers but that is the only thing I changed:
HTTP POST
POST /Products/2011-10-01?AWSAccessKeyId=ANONYMIZED
&Action=GetMatchingProductForId
&SellerId=ANONYMIZED
&SignatureVersion=2
&Timestamp=2018-09-28T05%3A45%3A43Z
&Version=2011-10-01
&Signature=ANONYMIZED
&SignatureMethod=HmacSHA256
&MarketplaceId=A13V1IB3VIYZZH
&IdType=EAN
&IdList.Id.1=9781933988665 HTTP/1.1
Host: mws.amazonservices.fr
x-amazon-user-agent: AmazonJavascriptScratchpad/1.0 (Language=Javascript)
Content-Type: text/xml
String to Sign
POST
mws.amazonservices.fr
/Products/2011-10-01
AWSAccessKeyId=ANONYMIZED&Action=GetMatchingProductForId&IdList.Id.1=9781933988665&IdType=EAN&MarketplaceId=A13V1IB3VIYZZH&SellerId=ANONYMIZED&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2018-09-28T05%3A45%3A43Z&Version=2011-10-01
=======
Now my question is, (and let's imagine my signature was created correctly), from the HTTP POST, what should the request look like ?
Here is my attempt:
https://mws.amazonservices.fr/Products/2011-10-01?AWSAccessKeyId=ANONYMIZED&Action=GetMatchingProductForId&SellerId=ANONYMIZED&SignatureVersion=2&Timestamp=2018-09-28T05%3A52%3A33Z&Version=2011-10-01&Signature=ANONYMIZED&SignatureMethod=HmacSHA256&MarketplaceId=A13V1IB3VIYZZH&IdType=EAN&IdList.Id.1=9781933988665
But what about '\n' escape characters that are in the scratchpad ? and what about 'HTTP/1.1' at the end, should I include that also ?
Thanks for your help.
I don't have an MWS account so I can't test the following, but this is one way you can do it:
# set this to your python2 binary; you'll need to do
# pip2 install boto
# from a command-line before using this code
Sys.setenv("RETICULATE_PYTHON"="/usr/bin/python2.7")
library(reticulate)
boto_mws_connection <- import("boto.mws.connection")
con <- boto_mws_connection$MWSConnection(
aws_access_key_id = ACCESS_KEY
aws_secret_access_key = AWS_SECRET
Merchant = MERCHANT_ID
)
con$get_matching_product_for_id(
MarketplaceId = "A13V1IB3VIYZZH",
IdType = "EAN",
IdList = c("9781933988665")
)
The HTTP/1.1 is usually created by your http client library. I am not familiar with R, but I googled and it seems there is a CURL package for R. CURL is the standard http library for a lot of languages including PHP. My PHP code to send an XML feed through curl looks like this:
$ch = curl_init();
curl_setopt($ch,CURLOPT_URL, 'https://mws.amazonservices.fr/Products/2011-10-01?.....your data and signature here...');
curl_setopt($ch,CURLOPT_POST, 1);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_POSTFIELDS, $xmlcontent);
curl_setopt($ch,CURLOPT_HTTPHEADER, array(
"Content-Type: text/xml",
"Content-MD5: ".base64_encode(md5($xmlcontent,true)),
"x-amazon-user-agent: TestScript/0.01")
);
$result = curl_exec($ch);
curl_close($ch);
By looking at this, it seems to me this should be easily translatable to the R interface for CURL.
Related
Brand new to API's and in a bit over my head. Reading a bit and searching around but still confused. I was given the following code to run in curl to generate a token using oauth2 with the password grant. I want to implement this code in R and running into some issues.
curl -H "Content-type: application/x-www-form-urlencoded" -H "Authorization: Basic xxxyyy123xxx==" -d "grant_type=password&username={username}&password={password}" https://{website}.com/rest/v2/oauth/token
When I run the above command in a terminal it generates a token which I have successfully used in R using this code:
token <- {character string generated by the above curl command}
call <- "https://{website}.com/rest/v2/services/APIService/getRegistryRecordByAccessionID?id={id#}"
df <- as.data.frame(fromJSON(content(httr::GET(call, add_headers(Authorization = paste("Bearer", token))), "text"), flatten = TRUE))
All I want is to generate the token in R instead of using another program like I have been. This is where my errors come up. The basic format I believe is:
token <- oauth2.0_token(endpoint = oauth_endpoint(
authorize = NULL,
access = "https://{website}.com/rest/v2/oauth/token"),
app = oauth_app(), #not sure how to set this up
use_basic_auth = T,
add_headers(Content-type = "application/x-www-form-urlencoded",
Authorization = "Basic xxxyyy123xxx==") #not sure if i need the authorization header here, I believe I need the content-type heder
)
From MrFlick's comments above, we can integrate the curl auhtorization commands into a POST using the httr package. We then look at the content of the object this returns which will include the access token in the first item in the list.
token <- content(httr::POST("https://{website}.com/rest/v2/oauth/token", body=list(grant_type="password", "username="{username}", password="{password}"), encode="form"))[[1]]
I am creating a Firebase HTTP function that uploads a file to Cloud Storage, creates a signed URL to the file, and then redirects the client to that URL. Using Postman with automatic redirect following turned on, the file is retrieved correctly. However, if I try to turn on redirects while using cURL (curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}'), the following error is returned by Cloud Storage:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
application/json
1602245678
/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I make the request with form encoded data instead, it works in cURL as well: curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
1602246219
/www.google.com/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I paste the URL into a browser or use cURL to download the signed URL, the file is also downloaded correctly.
I am using the following function to get the signed url:
async getSignedUrl(file: File, expireAt: number): Promise<string> {
const [url] = await file
.getSignedUrl({
action: "read",
expires: expireAt
});
return url
}
which returns a signed URL in the following format:
https://storage.googleapis.com/example.appspot.com/exampleBucket/exampleFile.txt?GoogleAccessId=[Access ID]&Expires=1602246219&Signature=[Signature] (I've noted that the value of "Expires" is the same value returned in the tag).
My suspicion is that Postman and cURL adds something to the request which results in a different signature, but I am not sure exactly what is going on.
What is happening when letting cURL follow the redirect or when creating a GET request in Postman, that leads to this difference in signature?
If I understood correctly, the issue arises in two scenarios
When hitting your CF through curl with
curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}')
According to the example in github in the docs Signed URL v4, 'Content-Type: application/octet-stream' should be used:
curl -X PUT -H 'Content-Type: application/octet-stream' --upload-file my-file '${url}'
I tried with the following with successfully result:
curl -X PUT -H 'Content-Type: application/octet-stream' -d '{"example": true}' 'https://storage.googleapis.com/...'
If I try with the content-type you shared with failed results.
2.
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
I tried a simple GET in postman using a Signed URL and it worked just fine
Command used in gsutil to get the signed URL:
gsutil signurl -d 10m key.json gs://BUCKET/aa.png
Then I tried a GET on postman and worked just fine.
I also tried with a Signed URL to upload a File in Postman and worked just fine.
My thoughts are that, according to Common MIME types
application/octet-stream is the default value for all other cases (not textual files).
When you set the content type as application/json you specify a JSON format, but not an object or file. That's why it works with the following, since you are not specifying the header content-type, the default is taken application/octet-stream
curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
Joss Barons answer helped me in the right direction, but it is not true that the Content-Type has to be application/octet-stream. That is only used for creating a signed url that can be used for uploading a file. In my case, when creating the signed url using the Cloud Storage SDK for node, I didn't specify a Content-Type, so when sending a GET request to the signed url, it must not contain a Content-Type header.
I need to perform a get request from nifi to couchbase. The curl command is:
curl http://HOST:PORT/query/service -d "statement=select item.Date from bucket unnest bucket as item" -u USER:PASSWORD
I tried using InvokeHttp and ExecuteStreamCommand but it keeps returning errors(status code 400). The full error message is:
{ "requestID": "bff62c0b-36fd-401d-bca0-0959e0944323", "errors":
[{"code":1050,"msg":"No statement or prepared value"}], "status":
"fatal", "metrics": {"elapsedTime": "113.31µs","executionTime":
"74.321µs","resultCount": 0,"resultSize": 0,"errorCount": 1
It's important to say that I prefer that the http request will be triggered by an incoming flowfile. I tried using the processors in various of ways but non of them worked.
When I run the command from the nifi server it works fine.
Thanks for the help
the -d parameter of the curl utility forces HTTP POST command
and application/x-www-form-urlencoded mime-type.
so, in the nifi InvokeHTTP select the following parameters
HTTP Method = POST
Remote URL = <your url here>
Basic Authentication Username = <username>
Basic Authentication Password = <password>
Content-Type = application/x-www-form-urlencoded
and the body of the flow file should be
statement=select item.Date from bucket unnest bucket as item
I don't know nifi, but based on the error message, the "statement=" part of the request isn't being included, or you are not sending the request as a POST command.
I want to make a request to an API which expects a JSON to be sent in the Header field. I'm unable to do it in Python Request library.
I'm able to do it in cURL.
cURL code:
curl -v -X POST https://content.dropboxapi.com/2/files/download
--header "Authorization: Bearer abcdefgh12343567"
--header "Dropbox-API-Arg: {\"path\": \"/folder/file.mp4\"}" -o file.mp4
Python code:
import requests
import simplejson
r = requests.post(
'https://content.dropboxapi.com/2/files/download',
headers={
'Authorization':'Bearer abcdefgh12343567',
'Dropbox-API-arg': simplejson.dumps({'path': '/folder/file.mp4'})
})
Here the Header contains a JSON string.
I'm trying to use Dropbox's files/download API documented here.
Even though the request is sent, the JSON value seems to be wrong.
The above code seems to work correctly now. I'm not sure about what happened - I was receiving HTTP 409 error earlier.
I was doubtful about how Python Requests Library would handle JSON string inside the header. So, I created an API to see how the server would read the headers made by both cURL and Python Requests. Both headers are identical.
I can access a REST API using curl by sending a system command and using pipe as below
filename fn pipe "curl -k -u &user.:&pass. 'https://blahblah.com/rest/api/content/108428908/child/attachment' ";
data new;
infile fn;
input;
put _infile_;
run;
This works, but I'd like to use PROC HTTP so I get the response back in a .json file rather than in the log, and I can use SAS password encryption. Here's my PROC HTTP code:
filename out "/blah/blahblah/output.json";
proc http
url = 'https://blahblah.com/rest/api/content/108428908/child/attachment'
method = "GET"
webusername = "&user"
webpassword = "&pass"
out = out
;
run;
This works as far as giving me a json file, but the json file I get back says "User not permitted to view attachments on content", which is the response I was getting from curl before adding the -k option (i.e. --insecure).
So how can I do the equivalent of -k in PROC HTTP? (Tell SAS not to check the SSL Cert). Modifying any options outside of those which can be specified within the .SAS file is not an option, as I don't have the access to change those.
Is setting the SSLREQCERT flag an option? According to this it looks like you can use the ALLOW option to tell the server to allow connections that fail the SSL handshake.
The most correct answer here is to NOT ignore the SSL issue and instead add the certificate to the appropriate trust manager/key store on the machine where you are running PROC HTTP. The method by which to do this varies by OS and SAS version. This documentation page will provide some insight into how to do this.