I currently upload to a webservice on an nginx server using the upload module (http://www.grid.net.ru/nginx/upload.en.html) from a custom desktop application doing a simple multipart-form POST that sends a file in one part and a base64 encoded XML with the file's metadata in another part.
The server receives this POST, passes it to my webservice which reads the metadata, processes the file and all is good.
What I want to do now is use the upload module's upload_resumable directive to do the POST in several chunks to minimize disconnection chances and allow resume. I can currently do this following the protocol described here: http://www.grid.net.ru/nginx/resumable_uploads.en.html
One sends byte ranges of the file along with some headers to identify the chunk and the session in several posts and once all the parts have been uploaded, nginx will compose the final POST containing the file name and path and pass it to your upload_pass location (which in my case CGIs to a django app).
However, I am not clear on how one would send a multipart post with this method since the protocol indicates that the body of the POST must be the bytes indicated in the byte range. I need the final post to also contain the XML I wrote about above.
I can think of sending the XML as the first bytes of the body and a header that indicates how many bytes belong to it but that would mean extra handling of the final file to remove that header and the final files are potentially in the GB size range.
Any other ideas?
Since the protocol supported by nginx specifically states that the post should not be multipart I ended up sending the file in the body, and the rest of the parameters encoded in the URL. Not the prettiest URLs but it works.
Related
I'm trying to download a firmware.bin file that is produced in a private Github repository. I have the code that is finding the right asset url to download the file and per Github instructions the accept header needs to be set to accept: application/octet-stream in order to get the binary file. I'm only getting JSON in response. If I run the same request through postman I'm getting a binary file as the body. I've tried downloading it using HTTPClient and I get the same JSON request. It seems the headers aren't being set as requested to tell Github to send the binary content as I'm just getting JSON. As for the ArduinoOTA abstraction, I can't see how to even try to set headers and in digging into the esp_https_ota functions and http_client functions there doesn't appear to be a way to set headers for any of these higher level abstractions because the http_config object has no place for headers as far as I can tell. I might file a feature request to allow for this, but am new to this programming area and want to check to see if I'm missing something first.
Code returns JSON, not binary. URL is github rest api url to the asset (works in postman)
HTTPClient http2;
http2.setAuthorization(githubname,githubpass);
http2.addHeader("Authorization","token MYTOKEN");
http2.addHeader("accept","application/octet-stream");
http2.begin( firmwareURL, GHAPI_CERT); //Specify the URL and certificate
With the ESP IDF HTTP client you can add headers to an initialized HTTP client using function esp_http_client_set_header().
esp_http_client_handle_t client = esp_http_client_init(&config);
esp_http_client_set_header(client, "HeaderKey", "HeaderValue");
err = esp_http_client_perform(client);
If using the HTTPS OTA API, you can register for a callback which gives you a handle to the underlying HTTP client. You can then do the exact same as in above example.
I am bit new to HTTP/2. I have learned that using HTTP/2 we can send multiple requests to the server without waiting for previous responses. Well I want to send an image file to the server which is large (more then 500 MB). There are following ways as listed here
Base64 encode the file, at the expense of increasing the data size by around 33%.
Send the file first in a multipart/form-data POST, and return an ID to the client. The client then sends the metadata with the ID, and
the server re-associates the file and the metadata.
Send the metadata first, and return an ID to the client. The client then sends the file with the ID, and the server re-associates
the file and the metadata.
I donont want to use first solution because it will increase the file size by 33%. I want to use 3rd solution.
As I am using HTTP/2 so my questions are
Can I send metadata and image simultaneously without waiting for ID from the server ?
If yes, then how can I implement ? Like do I have to do multithreading at server end for a client or how can I associate metadata and image with each other ?
If no then should I go for conventional style of HTTP/1.1 ?
I am using Restful and JSON for communication. More specifically I am using C# command-line client to send image and Asp.Net as server.
You can use multipart/related type to make a request with related mime-types. i.e in your case you can send an image along with a JSON body data.
You can refer Google drive file upload api which has a very similar implementation.
I'm developing a web page with a form which returns a PDF document based on the form data. Currently I use the HTTP response fields
Content-Type: application/pdf
Content-Disposition: attachment; filename="foo.pdf"
However, since the field Content-Disposition is non-standard and doesn't work in all browsers I'm looking for a different approach. Do I have to save the PDF document on the server? What is the modus operandi?
Edit: By "doesn't work in all browsers" I mean that with some browsers the filename is not set to foo.pdf. Dillo, for instance, just sets the default filename (in the download dialog) to the basename of the URL path (plus query string).
Do I have to save the PDF document on the server?
No. As far as the HTTP client is concerned it, the inner workings of the server are completely opaque to it. All it sees is a TCP stream of bytes from the server and how exactly that stream is produced doesn't matter as long as it matches the specified Content-Type.
Just send the PDF right after the HTTP headers and you're done with.
Update due to comment
So if you're wondering how to supply a filename without using a header field: Just augment the URL with it. I.e. something like
http://${DOMAIN}/${PDF_GENERATOR}/${DESIRED_FILENAME}
In the HTTP server add a rewrite rule to simply omit the filename part and redirect to just
http://${DOMAIN}/${PDF_GENERATOR}
The HTTP client does not see that, all it see is some URL ending with a "filename", that it can present the user as a default for saving.
On my web server I have a video file named 03.mp4.
I have a page (videoserver.aspx) to serve that file using below code
Response.ContentType = "application/octet-stream";
Response.AppendHeader("Content-Disposition", "attachment; filename=video.mp4");
Response.TransmitFile(Server.MapPath("03.mp4"));
Response.End();
Whats the difference between these 2 calls?
1: http://localhost/media/03.mp4
2: http://localhost/media/videoserver.aspx?q=03
When I point to those URLs directly in my browser, it prompts me a Save dialog in both the cases.
I have another web page that has a SWFObject. It consumes a video as input. Ok. When I feed it URL 1, it loads the video.
When I feed it URL 2, it doesn't load the video.
Why this difference? I prefer URL 2 as you can dynamically change the videos you are serving to consumers based on the query-string.
A lot of video players, including the new HTML5 <video> element, require support for so-called byte range requests using the HTTP Range header. This is normally already built in a bit self-respected HTTP server. Basically, to inform the client that the requested URL supports byte range requests, the server is supposed to return Accept-Ranges: bytes on the response and to be able to process all incoming Range requests by serving exactly the requested byte ranges back to the response as per the specification (see the first link on the Range header for detail).
So if you choose to take the HTTP response handling fully in your own hands instead of letting the HTTP server do the job it is designed for, you have to take this carefully into account.
Hence it proves I am a newbie to SWFObject.
The SWFObject I was referring to was dished out by Camtasia and it accepts a mp4 file thru FLashVars.
The question is "why did it not accept URL 2 while it accepted URL 1?". To which the answer is, URL 2 was not ending with .mp4.
And solution to my problem then was, create a handler that would accept */media/*.mp4 path and return the appropriate file's content, which in my case is fetched from DB.
I am setting up a back end API in a script of mine that contacts one of my sites by sending XML to my web server in the form of POST data. This script will be used by many and I want to limit the bandwidth waste for people that accidentally turn the feature on without a proper access key.
I will be denying requests that do not have the correct access key by maybe generating a 403 access code.
Lets say the POST data is ~500kb of data. Does the server receive all 500kb of data when this attempt is made regardless of the status code?
How about if I made the url contain the key mydomain/api/123456789 and generate 403 status on all bad access keys.
Does the POST data still get sent/received regardless or is it negotiated before the data is finally sent.
Thanks in advance!
Generally speaking, the entire request will be sent, including post data. There is often no way for the application layer to return a response like a 403 until it has received the entire request.
In reality, it will depend on the language/framework used and how closely it is linked to the HTTP server. Section 8.2.2 of RFC2616 HTTP/1.1 specification has this to say
An HTTP/1.1 (or later) client sending
a message-body SHOULD monitor the
network connection for an error status
while it is transmitting the request.
If the client sees an error status, it
SHOULD immediately cease transmitting
the body. If the body is being sent
using a "chunked" encoding (section
3.6), a zero length chunk and empty trailer MAY be used to prematurely
mark the end of the message. If the
body was preceded by a Content-Length
header, the client MUST close the
connection.
So, if you can find a language environemnt closely linked with the HTTP server (for example, mod_perl), you could do this in a way which does comply with standards.
An alternative approach you could take is to make an initial, smaller request to obtain a URL to use for the larger POST. The application can then deny providing the URL to clients without an appropriate key.
Here is great book about RESTful Web Services, where it's explained how HTTP works: http://oreilly.com/catalog/9780596529260
You can consider any request as envelope, where on top of it it's written address (URL), some properties (HTTP Headers) and inside it there's some data (if request is initiated by post method). So as you might guess you can't receive envelope partially.
Oh I forgot, it's when you are using HTTP Post with standard HTTP header "application/x-www-form-urlencoded" but if you are uploading files (correspondingly using ""multipart/form-data") Django gives you control over streamed chunks of files using Middleware classes: http://docs.djangoproject.com/en/dev/topics/http/middleware/