I am trying to send an http PUT request to my lighttpd server with a COMPRESSED file abc.xml.gz in its body. using curl command by using the --compressed option in curl and -H "Content-Encoding: gzip" to indicate encoding format.
My question is two pronged :
Is there anything else that i need to include in my curl command?
What option, if there's any, should I use in lighttpd.conf at the server to support input decompression i.e. to decompress the request object?
To my knowledge mod_compress in lighttpd only supports compressing the contents of the response and not decompressing the request.
Related
I am creating a Firebase HTTP function that uploads a file to Cloud Storage, creates a signed URL to the file, and then redirects the client to that URL. Using Postman with automatic redirect following turned on, the file is retrieved correctly. However, if I try to turn on redirects while using cURL (curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}'), the following error is returned by Cloud Storage:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
application/json
1602245678
/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I make the request with form encoded data instead, it works in cURL as well: curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
<?xml version='1.0' encoding='UTF-8'?>
<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your Google secret key and signing method.</Message>
<StringToSign>GET
1602246219
/www.google.com/example.appspot.com/exampleBucket/exampleFile.txt</StringToSign>
</Error>
If I paste the URL into a browser or use cURL to download the signed URL, the file is also downloaded correctly.
I am using the following function to get the signed url:
async getSignedUrl(file: File, expireAt: number): Promise<string> {
const [url] = await file
.getSignedUrl({
action: "read",
expires: expireAt
});
return url
}
which returns a signed URL in the following format:
https://storage.googleapis.com/example.appspot.com/exampleBucket/exampleFile.txt?GoogleAccessId=[Access ID]&Expires=1602246219&Signature=[Signature] (I've noted that the value of "Expires" is the same value returned in the tag).
My suspicion is that Postman and cURL adds something to the request which results in a different signature, but I am not sure exactly what is going on.
What is happening when letting cURL follow the redirect or when creating a GET request in Postman, that leads to this difference in signature?
If I understood correctly, the issue arises in two scenarios
When hitting your CF through curl with
curl -L -H "Content-Type: application/json" "https://us-central1-example.cloudfunctions.net/exampleFunction" -d '{"example": true}')
According to the example in github in the docs Signed URL v4, 'Content-Type: application/octet-stream' should be used:
curl -X PUT -H 'Content-Type: application/octet-stream' --upload-file my-file '${url}'
I tried with the following with successfully result:
curl -X PUT -H 'Content-Type: application/octet-stream' -d '{"example": true}' 'https://storage.googleapis.com/...'
If I try with the content-type you shared with failed results.
2.
If I try to manually make a GET request to the URL in Postman, I get an equivalent error:
I tried a simple GET in postman using a Signed URL and it worked just fine
Command used in gsutil to get the signed URL:
gsutil signurl -d 10m key.json gs://BUCKET/aa.png
Then I tried a GET on postman and worked just fine.
I also tried with a Signed URL to upload a File in Postman and worked just fine.
My thoughts are that, according to Common MIME types
application/octet-stream is the default value for all other cases (not textual files).
When you set the content type as application/json you specify a JSON format, but not an object or file. That's why it works with the following, since you are not specifying the header content-type, the default is taken application/octet-stream
curl -L "https://us-central1-example.cloudfunctions.net/exampleFunction" -d "example=true"
Joss Barons answer helped me in the right direction, but it is not true that the Content-Type has to be application/octet-stream. That is only used for creating a signed url that can be used for uploading a file. In my case, when creating the signed url using the Cloud Storage SDK for node, I didn't specify a Content-Type, so when sending a GET request to the signed url, it must not contain a Content-Type header.
I want to test proxy server. In order to make https request, browser sends CONNECT method beforehand (e.g. like Firefox does, when proxy is specified).
I can not achieve/send the same result in curl:
Following has root slash /www.example.com:443:
curl -X CONNECT http://proxy_host:proxy_port/www.example.com:443
Following will not work (without slash):
curl -X CONNECT http://proxy_host:proxy_portwww.example.com:443
Following is not what I want:
curl -X CONNECT http://proxy_host:proxy_port/some_path
So the first line of HTTP data should be CONNECT www.example.com:443 HTTP/1.1 but not CONNECT /www.example.com:443 HTTP/1.1 like curl sends in this case.
Maybe this question also related some-how, if I would know how to not send path.
NOTE! I do not want to use curl -x http://proxy_host:proxy_port https://www.example.com, because this option/flag -x does not work with custom SSL certificates --cacert ... --key ... --cert ....
Any ideas how to send plain header data or not specify path, or specify host and port as a path?
(-X simply replaces the string in the request so of course setting it to CONNECT will not issue a proper CONNECT request and will certainly not make curl handle it correctly.)
curl will do a CONNECT by itself when connecting to a TLS server through a HTTP proxy, and even though you claim -x breaks the certificate options that is an incorrect statement. The --cacert and other options work the same even when the connection is done through a HTTP proxy.
You can also make curl do a CONNECT trough a HTTP(S) proxy for other protocols by using -p, --proxytunnel - also in combination with -x.
I need to perform a get request from nifi to couchbase. The curl command is:
curl http://HOST:PORT/query/service -d "statement=select item.Date from bucket unnest bucket as item" -u USER:PASSWORD
I tried using InvokeHttp and ExecuteStreamCommand but it keeps returning errors(status code 400). The full error message is:
{ "requestID": "bff62c0b-36fd-401d-bca0-0959e0944323", "errors":
[{"code":1050,"msg":"No statement or prepared value"}], "status":
"fatal", "metrics": {"elapsedTime": "113.31µs","executionTime":
"74.321µs","resultCount": 0,"resultSize": 0,"errorCount": 1
It's important to say that I prefer that the http request will be triggered by an incoming flowfile. I tried using the processors in various of ways but non of them worked.
When I run the command from the nifi server it works fine.
Thanks for the help
the -d parameter of the curl utility forces HTTP POST command
and application/x-www-form-urlencoded mime-type.
so, in the nifi InvokeHTTP select the following parameters
HTTP Method = POST
Remote URL = <your url here>
Basic Authentication Username = <username>
Basic Authentication Password = <password>
Content-Type = application/x-www-form-urlencoded
and the body of the flow file should be
statement=select item.Date from bucket unnest bucket as item
I don't know nifi, but based on the error message, the "statement=" part of the request isn't being included, or you are not sending the request as a POST command.
I want to make a request to an API which expects a JSON to be sent in the Header field. I'm unable to do it in Python Request library.
I'm able to do it in cURL.
cURL code:
curl -v -X POST https://content.dropboxapi.com/2/files/download
--header "Authorization: Bearer abcdefgh12343567"
--header "Dropbox-API-Arg: {\"path\": \"/folder/file.mp4\"}" -o file.mp4
Python code:
import requests
import simplejson
r = requests.post(
'https://content.dropboxapi.com/2/files/download',
headers={
'Authorization':'Bearer abcdefgh12343567',
'Dropbox-API-arg': simplejson.dumps({'path': '/folder/file.mp4'})
})
Here the Header contains a JSON string.
I'm trying to use Dropbox's files/download API documented here.
Even though the request is sent, the JSON value seems to be wrong.
The above code seems to work correctly now. I'm not sure about what happened - I was receiving HTTP 409 error earlier.
I was doubtful about how Python Requests Library would handle JSON string inside the header. So, I created an API to see how the server would read the headers made by both cURL and Python Requests. Both headers are identical.
Is there a command line utility where you can simply set up an HTTP request and have the trace simply output back to the console?
Also specifying the method simply would be a great feature instead of the method being a side effect.
I can get all the information I need with cURL but I can't figure out a way to just display it without dumping everything to files.
I'd like the output to show the sent headers the received headers and the body of the message.
There must be something out there but I haven't been able to google for it. Figured I should ask before going off and writing it myself.
I dislike answering my own question but c-smile's answer lead me down the right track:
Short answer shell script over cURL:
curl --dump-header - "$#"
The - [dash] meaning stdout is a convention I was unaware of but also works for wget and a number of other unix utilities. It is apparently not part of the shell but built into each utility. The wget equivalent is:
wget --save-headers -qO - "$#"
Did you try wget:
http://www.gnu.org/software/wget/manual/wget.html#Wgetrc-Commands ?
Like wget --save-headers ...
To include the HTTP headers in the output (as well as the server response), just use curl’s -i/--include option. For example:
curl -i "http://www.google.com/"
Here’s what man curl says about this setting:
-i/--include
(HTTP) Include the HTTP-header in the output. The HTTP-header
includes things like server-name, date of the document, HTTP-
version and more...
If this option is used twice, the second will again disable
header include.
Try http, e.g.
http -v example.org
Further into at https://httpie.org
It even includes a page to try online:
https://httpie.org/run
Telnet has for long been a well-known (though now forgotten, I guess) tool for looking at a web page. The general idea is to telnet to the http port, to type an http 1.1 GET command, and then to see the served page on the screen.
A good detailed explanation is http://support.microsoft.com/kb/279466
A Google search yields a whole bunch more.
Use telnet on port 80
For example:
telnet telehack.com 80
GET / HTTP/1.1
host: telehack.com
<CR>
<CR>
<CR> means Enter