I want to make a request to an API which expects a JSON to be sent in the Header field. I'm unable to do it in Python Request library.
I'm able to do it in cURL.
cURL code:
curl -v -X POST https://content.dropboxapi.com/2/files/download
--header "Authorization: Bearer abcdefgh12343567"
--header "Dropbox-API-Arg: {\"path\": \"/folder/file.mp4\"}" -o file.mp4
Python code:
import requests
import simplejson
r = requests.post(
'https://content.dropboxapi.com/2/files/download',
headers={
'Authorization':'Bearer abcdefgh12343567',
'Dropbox-API-arg': simplejson.dumps({'path': '/folder/file.mp4'})
})
Here the Header contains a JSON string.
I'm trying to use Dropbox's files/download API documented here.
Even though the request is sent, the JSON value seems to be wrong.
The above code seems to work correctly now. I'm not sure about what happened - I was receiving HTTP 409 error earlier.
I was doubtful about how Python Requests Library would handle JSON string inside the header. So, I created an API to see how the server would read the headers made by both cURL and Python Requests. Both headers are identical.
Related
I am creating a Firebase HTTP function that uploads a file to Cloud Storage, creates a signed URL to the file, and then redirects the client to that URL. Using Postman with automatic redirect following turned on, the file is retrieved correctly. However, if I try to turn on redirects while using cURL (curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}'), the following error is returned by Cloud Storage:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
application/json
1602245678
/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I make the request with form encoded data instead, it works in cURL as well: curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
1602246219
/www.google.com/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I paste the URL into a browser or use cURL to download the signed URL, the file is also downloaded correctly.
I am using the following function to get the signed url:
async getSignedUrl(file: File, expireAt: number): Promise<string> {
const [url] = await file
.getSignedUrl({
action: "read",
expires: expireAt
});
return url
}
which returns a signed URL in the following format:
https://storage.googleapis.com/example.appspot.com/exampleBucket/exampleFile.txt?GoogleAccessId=[Access ID]&Expires=1602246219&Signature=[Signature] (I've noted that the value of "Expires" is the same value returned in the tag).
My suspicion is that Postman and cURL adds something to the request which results in a different signature, but I am not sure exactly what is going on.
What is happening when letting cURL follow the redirect or when creating a GET request in Postman, that leads to this difference in signature?
If I understood correctly, the issue arises in two scenarios
When hitting your CF through curl with
curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}')
According to the example in github in the docs Signed URL v4, 'Content-Type: application/octet-stream' should be used:
curl -X PUT -H 'Content-Type: application/octet-stream' --upload-file my-file '${url}'
I tried with the following with successfully result:
curl -X PUT -H 'Content-Type: application/octet-stream' -d '{"example": true}' 'https://storage.googleapis.com/...'
If I try with the content-type you shared with failed results.
2.
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
I tried a simple GET in postman using a Signed URL and it worked just fine
Command used in gsutil to get the signed URL:
gsutil signurl -d 10m key.json gs://BUCKET/aa.png
Then I tried a GET on postman and worked just fine.
I also tried with a Signed URL to upload a File in Postman and worked just fine.
My thoughts are that, according to Common MIME types
application/octet-stream is the default value for all other cases (not textual files).
When you set the content type as application/json you specify a JSON format, but not an object or file. That's why it works with the following, since you are not specifying the header content-type, the default is taken application/octet-stream
curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
Joss Barons answer helped me in the right direction, but it is not true that the Content-Type has to be application/octet-stream. That is only used for creating a signed url that can be used for uploading a file. In my case, when creating the signed url using the Cloud Storage SDK for node, I didn't specify a Content-Type, so when sending a GET request to the signed url, it must not contain a Content-Type header.
I want to upload a jpg file to a server using python post request. I used a curl command that works fine:
curl --cacert C:\cacert.pem -u user:key -F "data=#picture.jpg" https://serverurl/users/pictures
I am using the following python request but it gives me a status 500 response.
picture = {'data' : ('picture',open('picture.jpg','rb'))}
req = requests.post(
'https://serverurl/users/pictures',
files=picture, auth=('user','key'))
Your curl command is POSTing a form with "data=#picture.jpg" while you are uploading a local file using requests. Have you read the doc?
Try something different:
session = requests.Session()
session.post('https://serverurl/users/pictures',
data={'data': 'picture.jpg'}, cert='C:\cacert.pem', auth=('user':'pass'))
I need to perform a get request from nifi to couchbase. The curl command is:
curl http://HOST:PORT/query/service -d "statement=select item.Date from bucket unnest bucket as item" -u USER:PASSWORD
I tried using InvokeHttp and ExecuteStreamCommand but it keeps returning errors(status code 400). The full error message is:
{ "requestID": "bff62c0b-36fd-401d-bca0-0959e0944323", "errors":
[{"code":1050,"msg":"No statement or prepared value"}], "status":
"fatal", "metrics": {"elapsedTime": "113.31µs","executionTime":
"74.321µs","resultCount": 0,"resultSize": 0,"errorCount": 1
It's important to say that I prefer that the http request will be triggered by an incoming flowfile. I tried using the processors in various of ways but non of them worked.
When I run the command from the nifi server it works fine.
Thanks for the help
the -d parameter of the curl utility forces HTTP POST command
and application/x-www-form-urlencoded mime-type.
so, in the nifi InvokeHTTP select the following parameters
HTTP Method = POST
Remote URL = <your url here>
Basic Authentication Username = <username>
Basic Authentication Password = <password>
Content-Type = application/x-www-form-urlencoded
and the body of the flow file should be
statement=select item.Date from bucket unnest bucket as item
I don't know nifi, but based on the error message, the "statement=" part of the request isn't being included, or you are not sending the request as a POST command.
I am using below curl
curl http://XXXXX:6800/schedule.json -d project=stackoverflow -d spider=careers.stackoverflow.com -d setting=DOWNLOAD_DELAY=2 -d arg1=val1
How can i convert this to Http request? Can any one help me?
Thanks,
In curl the -d option means --data, just the data to post to the http server,in the same way that a browser does when a user has filled in an HTML form and presses the submit button.
This will cause curl to pass the data to the server using the content-type application/x-www-form-urlencoded.
For example you use python, you can form the requests lik:
import requests
data = {
'project': 'stackoverflow',
'spider': 'careers.stackoverflow.com',
'setting': 'DOWNLOAD_DELAY=2'
'arg1': 'val1'
}
headers = {'content-type': 'application/x-www-form-urlencoded'}
requests.post('http://XXXX:6800/schedule.json', data=data, headers=headers)
For more infomation, you can look the curl man page.
I am trying to send an http PUT request to my lighttpd server with a COMPRESSED file abc.xml.gz in its body. using curl command by using the --compressed option in curl and -H "Content-Encoding: gzip" to indicate encoding format.
My question is two pronged :
Is there anything else that i need to include in my curl command?
What option, if there's any, should I use in lighttpd.conf at the server to support input decompression i.e. to decompress the request object?
To my knowledge mod_compress in lighttpd only supports compressing the contents of the response and not decompressing the request.