I am trying to change the body of a HTTP response (for debugging purposes) using Charles webproxy. However, the response is GZIPed and when I modify something in the body (using the Tools-->Rewrite options) two problems emerge:
The Content-Length header is not updated to reflect the new data I've added to the response. I have to manually update it myself.
Even if I update the Content-Length manually, it seems the new GZIP body is invalid (ie, it is not GZIPed correctly).
I can't just remove the Content-Encoding: gzip from the request because I don't have control over the code that consumes the response. That code is always expecting a gzip body and it fails to parse the data if I remove the GZIP encoding from the response by removing the Content-Encoding: gzip from the request.
How can I modify a GZIPed response body using Charles?
Related
I have a website that caches data, it uses a content-delivery-network called akamai, and this is the response header. 'cache-control': 'must-revalidate, max-age=600'. This means, re-validate after 600 seconds (stale). If i want the cdn to query the origin server each request, i can do this... cache-control: no-cache. When i send this request, i get the same response header... indicating that it isn't being re-validated? Is it actually not being re-validated, or is it being re-validated? Since the website is well-known, it is safe to say that the website is correctly responding to headers.
What you've observed is correct behavior.
Your Cache-Control request header applies to this request, while the Cache-Control response applies to future requests. Whether or not your client wants a fresh response to this request will not and should not change the server's general directions as to how its resources can be cached.
As long as you use no-cache in your requests you should not get a cached response.
Background
A server can send a Content-Encoding header to indicate if, and how, the content of the response body has been compressed. E.g.
Content-Encoding: gzip
A server can also send a Content-Type header to indicate the media type, and optionally provide the standard used to encode the content. E.g.
Content-Type: text/html; charset=utf-8
Therefore, it seems that the encoding of the content is specified in the Content-Type header, and not the Content-Encoding header.
Question
During the design of the HTTP standard, what's the rationale behind the naming of the Content-Encoding header? (Over, say Content-Compression) Is this a similar case of bad naming like the 401 Unauthorized response code?
I want to send large amount of json over http to sever.
If I use "Content-Encoding":"GZIP" in my httpClient, does it automatically convert the request body to compressed format?
No, the RFC 7231 describes content encoding. If you are sending Content-Encoding you need to make sure that the content is in that encoding.
If you send Content-Encoding: gzip and the message in plain text you will (quite rightly) receive an HTTP 400. The body of a gzip message will always start with 0x1f 0x8b and if the server does not find that int he POST request it is right to complain.
Another reason for this is that you need an appropriate Content-Length header. This will not be the length of the original JSON, it must be the length (in bytes) of the gzipped JSON.
You need to perform the gzip of the JSON before sending anything since you need to know what to place in Content-Length beforehand.
Extra note: If the JSON is that huge (e.g. several gigabytes) you probably will need Transfer-Encoding: chunked, which comes with its own complications. (You do not send Content-Length but add the length of the chuck to the body itself.)
If it automatically does this, is 100% dependent on which http client you are using and if they implemented it that way. Usually setting a header will not automatically encode it, at least in the clients I regularly use.
My reading of the definition of the 'private' directive for the Cache-Control header is that it will prevent any part of the response from being cached by intermediate proxies. So based on that, it sounds like if I'm using the 'private' directive then there's no need to also use a 'no-cache="set-cookie"' directive to tell intermediate proxies to suppress caching of the Set-Cookie header.
However, in section 4.2.3 in this document, it says:
The origin server should send the following additional HTTP/1.1
response headers, depending on circumstances:
To suppress caching of the Set-Cookie header: Cache-control: no-cache="set-cookie".
and one of the following:
To suppress caching of a private document in shared caches: Cache-control: private.
[...]
and I see a ton of examples online that have both directives.
So do I really need both of those to prevent intermediate proxies from caching a Set-Cookie header? I've been doing some testing, and it seems like Internet Explorer is responding to the 'no-cache="set-cookie"' directive by issuing a full request every subsequent time, so I'd rather not include it if it's not necessary.
Cache-Control: Private will stop intermediary caches from storing the content, so the set-cookie directive isn't applicable in this case.
HTTP 1.1 responses can be chunked (spec). At the end of the last chunk the server can send a "trailer", which contains additional headers.
The question is: can you include a Location header in the trailer, and will the browser react by making a redirect?
The problem is that Location header to work should be with specific response code 3xx so in a standard response you can't use it see http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html
Yes.
To have the Location header, the server should be responding with the 201 or one of the 3XX status codes. It should also include content, perhaps a message in HTML explaining what happened and giving a handy hyperlink to the resource. The content would need to be chunked and the Transfer-Encoding header should be present and have the value "chunked".
If all this were true, then the Trailer header could be added with a value of "Location", and the Location header could then trail the content.
Here's the real issue: This seems silly and pointless. What use case might you have where you will redirect a client to a new location, yet you don't know that location until after the content is complete? I can't think of a reason. But maybe you have one? If there's not a good use case, then I don't think you should do this.
Edit: I thought of a reason. Shiflett gives an example of chunked transfer encoding where the first chunk of HTML sent to the client says, "We are completing your transaction." Time passes while the transaction is completed. Then the second and final chunk of HTML is sent to the client that says, "Ok, your transaction is now complete." (HTTP Dev's Handbook p97) Combine this idea with a 201 where a new file is created. It might be that the location of the new file is unknown until the very end of the server's processing. Thus it would want to use chunked transfer encoding, and it would want to put the Location header in the trailers.
Second Edit: Yes, you can add it because the spec specifically forbids only the following header fields: Transfer-Encoding, Content-Length, and Trailer.