We have an existing API where 3rd parties can push data. The API calls are authenticated with HMAC using the same scheme as described in Ruby's ApiAuth library which I use to verify signed requests. We now need to support multipart/form-data file uploads.
I'm attempting to write a bash script as an example API call using cURL. I already have one that works without file uploads (POST request with only JSON data) here.
The part I'm stuck on is generating the $content_md5 for the multipart request. I understand the content of a multipart request has content sections separated by the boundary string, as described here.
Problem 1: cURL generates it's own boundary string and appends it to my content-type header
Problem 2: Should I MD5 the entire request body with boundary strings and section headers included?
So basically, I need to be able to know what the boundary string is so that I can generate a string that looks like the content sections and boundaries described in the http multipart format, lines 7 through 23 so that I may MD5 it.
Is there a way to do this with only cURL? Is there a better way to construct such an HMAC signed multipart request?
I was hoping to do it with cURL to present as a generalized example to 3rd party devs that will integrate with our API so they can sign their requests in whatever language they're using.
Related
I'm trying to download a firmware.bin file that is produced in a private Github repository. I have the code that is finding the right asset url to download the file and per Github instructions the accept header needs to be set to accept: application/octet-stream in order to get the binary file. I'm only getting JSON in response. If I run the same request through postman I'm getting a binary file as the body. I've tried downloading it using HTTPClient and I get the same JSON request. It seems the headers aren't being set as requested to tell Github to send the binary content as I'm just getting JSON. As for the ArduinoOTA abstraction, I can't see how to even try to set headers and in digging into the esp_https_ota functions and http_client functions there doesn't appear to be a way to set headers for any of these higher level abstractions because the http_config object has no place for headers as far as I can tell. I might file a feature request to allow for this, but am new to this programming area and want to check to see if I'm missing something first.
Code returns JSON, not binary. URL is github rest api url to the asset (works in postman)
HTTPClient http2;
http2.setAuthorization(githubname,githubpass);
http2.addHeader("Authorization","token MYTOKEN");
http2.addHeader("accept","application/octet-stream");
http2.begin( firmwareURL, GHAPI_CERT); //Specify the URL and certificate
With the ESP IDF HTTP client you can add headers to an initialized HTTP client using function esp_http_client_set_header().
esp_http_client_handle_t client = esp_http_client_init(&config);
esp_http_client_set_header(client, "HeaderKey", "HeaderValue");
err = esp_http_client_perform(client);
If using the HTTPS OTA API, you can register for a callback which gives you a handle to the underlying HTTP client. You can then do the exact same as in above example.
I'm working in a Go Lang REST API repo.
I'm wanting to build an endpoint that will take in a file (as part of the form-data, so I suppose I'll use request.FormFile('my-file-key')). This endpoint should also take in a body of a JSON model (which i suppose would be decoded with something like this:
var myData model.MyModel
json.NewDecoder(request.Body).Decode(&myData)
But I'm running into a lot of issues. Is it even possible to send both a body and a file in the form-data with a http request?
If I try to send both I get errors from FormFile saying that it can't find the file of the key name (but if I send the exact same request without a body, this error doesn't happen). I guessing it's having trouble decoding the request.
What you need is a multipart request. One part can be JSON data, and the other part the file data.
If you're using a Go client to prepare the request, you need to use the mime/multipart package to create a Writer, then use CreatePart to create the JSON part, then the file part, and submit the request to the server.
On the decoding side: since the body is JSON you cannot parse it as a form. You have to use a multipart.Reader to read from the body after you parse the headers. Again, from that reader you get a Part, and read the data from that part. You'll get two parts, one for the JSON data and one for the file data.
I'm currently writing a little library to standardize the use of HttpURLConenction's in my android-projects.
In one of my projects I'm communicating with a server using Http Digest to authenticate the user. The default java HttpUrlConnection doesn't support digest but I've managed to write a simplified digest auth (qop=auth) which is working pretty well.
For future projects I want to enable my library to use auth-int. Therefor I need to modify the construction of the A2 Hash and include the HTTP entity body (see RFC 7616 Section 3.4.3).
To do so I thought about extracting the full HTTP request from the URLConnection and then throwing out the unneeded stuff. Unfortunately I was unable to find a way to do this.
Extracting the single headerfields is possible by .getContentEncoding(), .getContentLength(), .getContentType() etc. . But with this I can't ensure that the order of the entity I'm reconstructing with the get-methods is the same as in the request and this might lead to a 401.
To sum it up:
Is there a way to extract the full request (or better only the entity body) from a HttpURLConnection?
I want to redirect my users browser using HTTP code 303 to a GET URL that I secure using HMAC. Because the request will come from the users browser, I will not have fore-knowledge of the request headers. So I am generating the HMAC hash using the values of the HTTP method and URL only. For example, the URL I want the browser to do to might be:
GET /download
?name=report.pdf
&include=http://url1
&include=http://url2
This create report.pdf for me, containing the contents of all the urls specified using the include query param.
My HMAC code will change this URL to be
GET /download
?name=report.pdf
&include=http://url1
&include=http://url2
&hmac-algorithm=simple-hmac
&hmac-signature=idhihhoaiDOICNK
I can issue HTTP 303 to the user using this URL, and the user will get their report.pdf.
As I am not including the request headers in the signature, I am wondering two things:
1) Can a would-be attacker take advantage of the fact that I am not signing the request headers?
2) Is there a better way to achieve what I am trying to do?
When I realised that what I am talking about here is a signed URL, I checked the Amazon Docs and found "REST Authentication Example 3: Query String Authentication Example" in this document: http://s3.amazonaws.com/doc/s3-developer-guide/RESTAuthentication.html.
This example is about a signed URL for use through a browser. About signing the headers, the document says:
You know that when the browser makes the GET request, it won't provide a Content-Md5 or a Content-Type header, nor will it set any x-amz- headers, so those parts are all kept empty.
In other words, Amazon leave the headers out of the signature.
Amazon make no mention of potential security holes, so until I hear otherwise (or get hacked :) ), I will assume my approach above is fine.
I currently upload to a webservice on an nginx server using the upload module (http://www.grid.net.ru/nginx/upload.en.html) from a custom desktop application doing a simple multipart-form POST that sends a file in one part and a base64 encoded XML with the file's metadata in another part.
The server receives this POST, passes it to my webservice which reads the metadata, processes the file and all is good.
What I want to do now is use the upload module's upload_resumable directive to do the POST in several chunks to minimize disconnection chances and allow resume. I can currently do this following the protocol described here: http://www.grid.net.ru/nginx/resumable_uploads.en.html
One sends byte ranges of the file along with some headers to identify the chunk and the session in several posts and once all the parts have been uploaded, nginx will compose the final POST containing the file name and path and pass it to your upload_pass location (which in my case CGIs to a django app).
However, I am not clear on how one would send a multipart post with this method since the protocol indicates that the body of the POST must be the bytes indicated in the byte range. I need the final post to also contain the XML I wrote about above.
I can think of sending the XML as the first bytes of the body and a header that indicates how many bytes belong to it but that would mean extra handling of the final file to remove that header and the final files are potentially in the GB size range.
Any other ideas?
Since the protocol supported by nginx specifically states that the post should not be multipart I ended up sending the file in the body, and the rest of the parameters encoded in the URL. Not the prettiest URLs but it works.