If the post is using "multipart/form-data" content-type, and each part could be a file or other content type.
If I want to use GZIP, should the GZIP apply to the entire post body of all parts altogether or could it be possible to choose some file use gzip content-encoding while some file don't.
Is there any standard for it or only common practice ?
Thanks
For example, can I add Content-Encoding:gzip at below 'file1' part
Host: localhost:8081
Connection: keep-alive
Content-Length: 317
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.84 Safari/537.36
Cache-Control: no-cache
Origin: chrome-extension://fhbjgbiflinjbdggehcddcbncdddomop
Postman-Token: 7143164d-0da5-0e1d-112e-91f2a21c22c2
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryPZAv0gGlJrA4ABu2
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9,zh-CN;q=0.8,zh;q=0.7
------WebKitFormBoundaryPZAv0gGlJrA4ABu2
Content-Disposition: form-data; name="key1"
value1
------WebKitFormBoundaryPZAv0gGlJrA4ABu2
Content-Disposition: form-data; name="file1"; filename="sample_file.txt"
Content-Type: text/plain
Content-Encoding: gzip ---------------------------IS IT OK TO ADD GZIP HERE?
This is a sample file content!
------WebKitFormBoundaryPZAv0gGlJrA4ABu2--
If you specify "Content-Encoding: gzip" in the HTTP response header section, it will apply to the full response.
I don't believe there's a way to apply compression to individual parts.
Either apply Content-Encoding compression to the whole message (end-to-end), or have the payload compressed on the transport layer, Transfer-Encoding (potentially hop-by-hop). What you can not do is compression on each part.
The multipart/form-data media type does not support any MIME header
fields in parts other than Content-Type, Content-Disposition, and (in
limited circumstances) Content-Transfer-Encoding. Other header
fields MUST NOT be included and MUST be ignored.
Emphasis mine. Source: https://www.rfc-editor.org/rfc/rfc7578#section-4.8
Related
when investigating network behavior, I usually use postman for sending HTTP requests, however
I need the option to send a raw HTTP request (via clear text), or at least only the headers, and it seems that postman does not support to edit your request via clear HTTP text. (buy the way the opposite is possible, you can read the raw http text of the requests you constructed in postman but you can't edit them)
for example:
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: he-IL,he;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 21114
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Cookie: csrftoken=0alLaljTasofjCWZv7gcmukXuz6gMxfzlWpV691hzZZ1hTBcdVJ3mH8ozRDnO6hu; tk_or=%22%22; tk_lr=%22%22; session_id_12211=ff6a58b0baf98005748ce5a3c6a732aef33b750f; splunkweb_csrf_token_12211=10024448868272708216; token_key=10024448868272708216; experience_id=4852e1c6-726b-1ab3-bafa-f0a735d3f708; splunkd_12211=NjcrwAj_TLgz5JalVh2HTynLdbp_CPnfHFKi8qmsODiH40HI2urbPvAvJ9uvDKKoM3nATXEkS6dGytD0TvfiOtAUGJhk7Od25on_gJcZrQwcePQZ8HQaCmGScm^RXmOdDa^KVvN
Host: localhost:12211
Origin: http://localhost:12211
Pragma: no-cache
sec-ch-ua: "Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
X-Requested-With: XMLHttpRequest
X-Splunk-Form-Key: 10024448898272708216
does postman allow editing the raw HTML? if not, there is other tool that can?
Preflight response returns headers
server: uvicorn
access-control-allow-origin: *
access-control-allow-methods: DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT
access-control-max-age: 600
access-control-allow-headers: Accept, Accept-Language, Content-Language, Content-Type
content-length: 2
content-type: text/plain; charset=utf-8
for request with headers
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Firefox/102.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Access-Control-Request-Method: PATCH
Access-Control-Request-Headers: content-type
Referer: http://localhost:8080/
Origin: http://localhost:8080
Connection: keep-alive
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-site
But Firefox (102.0) rejects this with reason CORS Method Not Found.
POST and DELETE methods work fine, only PATCH is rejected. In Chrome, this request passes without problems.
In FastAPI, I have a middleware configured as following:
middleware=[Middleware(CORSMiddleware, allow_origins=["*"], allow_methods=["*"])],
Does anyone know what the problem may be? I also don't send any content/data in this PATCH request, I don't know if that's important.
I run a webstore on Magento ver. 2.1.8 eCommerce platform.
Recently, I came across a issue where front-end of my website was using a completely different colour skin and layout and couldn't figure out why this is happening.
Purging the Varnish cache solved the issue for some time as the website would randomly changed the skin again after some time. I had no idea how Varnish was caching the content that should never be displayed.
I also use Acunetix as a Server Side Scanner which scans my website every week for any malicious code or gaps in security. I found one of the POST request that it sent:
POST /themesettings/index/paneltool/ HTTP/1.1
Content-Length: 1193
Content-Type: multipart/form-data; boundary=-----Boundary_JTWCAHJSKP
Referer: https://www.domain.co.uk/
Cookie: PHPSESSID=sadfq345r234324dfasd; mage-messages= vespaneltool=a%3A6%3A%7Bs%3A52%3A%22ves_themesettings_general%2Fgeneral_settings%2Fdirection%22%3Bs%3A3%3A%22rtl%22%3Bs%3A49%3A%22ves_themesettings_general%2Fgeneral_settings%2Flayout%22%3Bs%3A8%3A%22boxed-lg%22%3Bs%3A52%3A%22ves_themesettings_general%2Fgeneral_settings%2Fmax_width%22%3Bs%3A5%3A%22960px%22%3Bs%3A59%3A%22ves_themesettings_general%2Fgeneral_settings%2Fmax_width_custom%22%3Bs%3A8%3A%22gqwtkdks%22%3Bs%3A47%3A%22ves_themesettings_general%2Fgeneral_settings%2Fskin%22%3Bs%3A8%3A%22blue.css%22%3Bs%3A55%3A%22ves_themesettings_header%2Fgeneral_settings%2Fheader_layout%22%3Bs%3A13%3A%22default.phtml%22%3B%7D; _vwo_uuid_v2=EC7CC959823F97596222AB508A6BB8BE|53a815cb661ea346311131469aaeb1c2; PHPSESSID=oaibesqi4980brc3udl1gdrfb0
Host: www.domain.co.uk
Connection: Keep-alive
Accept-Encoding: gzip,deflate
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.21 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.21
Accept: */*
-------Boundary_JTWCAHJSKP
Content-Disposition: form-data; name="btn-save"
Apply
-------Boundary_JTWCAHJSKP
Content-Disposition: form-data; name="userparams[ves_themesettings_general/general_settings/direction]"
rtl
-------Boundary_JTWCAHJSKP
Content-Disposition: form-data; name="userparams[ves_themesettings_general/general_settings/layout]"
boxed-md
-------Boundary_JTWCAHJSKP
Content-Disposition: form-data; name="userparams[ves_themesettings_general/general_settings/max_width]"
1024px
-------Boundary_JTWCAHJSKP
Content-Disposition: form-data; name="userparams[ves_themesettings_general/general_settings/max_width_custom]"
12345'"\'\");|]*%00{%0d%0a<%00>%bf%27'ð©
-------Boundary_JTWCAHJSKP
Content-Disposition: form-data; name="userparams[ves_themesettings_general/general_settings/skin]"
aquamarine.css
-------Boundary_JTWCAHJSKP
Content-Disposition: form-data; name="userparams[ves_themesettings_header/general_settings/header_layout]"
default2.phtml
-------Boundary_JTWCAHJSKP
Content-Disposition: form-data; name="vespanel"
1
-------Boundary_JTWCAHJSKP
Content-Disposition: form-data; name="vesreset"
0
-------Boundary_JTWCAHJSKP--
The above is the configuration of the wrong designed that was cached by Varnish. E.g. aquamarine.css is the wrong CSS file cached, it should be red.css.
Is it possible that after this post request, Varnish cached this settings? Also, what is the best way to overcome this problem? Should I create a rule in Acunetix to avoid this URL? Or create a NginX configuration to block access to it?
Try something like the following to just force Varnish to not attempt to use (or store into) the cache for POST requests:
sub vcl_recv {
if (req.method == 'POST') {
set req.hash_always_miss = true;
}
}
I would like to make a file download resumable using byte-range requests.
The problem is that my existing download action is responding on a POST method and I would like to keep it that way.
But it seems from my early tests that Chrome turns interrupted POST requests for file downloads into GET requests when the user tries to resume and thus the resuming of the download fails.
Am I missing something?
Is this something related to the HTTP specs that only allow GET requests to be resumed?
Or is it simply a design flaw in Chrome (and maybe other browsers as well) that makes it forget the original HTTP method used?
UPDATE:
Here are the request/response data:
Initial POST request:
POST http://localhost:35547/Download?Guid=396b4697-e275-4396-818c-548bf8c0a281 HTTP/1.1
Host: localhost:35547
Connection: keep-alive
Content-Length: 0
Cache-Control: max-age=0
Origin: http://localhost:35547
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer: http://localhost:35547/File/396b4697-e275-4396-818c-548bf8c0a281
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: __RequestVerificationToken=LuPgM05MHrsuyskgfhsrHVUs; ASP.NET_SessionId=gfiulghfuygisghkf; .ASPXAUTH=FGDJHGDHSDFB15AFDE6371CGJHDFGFBHD; fileDownload=true
Initial response (to the request above):
HTTP/1.1 200 OK
Cache-Control: private, s-maxage=0
Content-Type: application/zip
Server: Microsoft-IIS/7.5
X-AspNetMvc-Version: 5.2
Content-Disposition: attachment; filename="FILE-396b4697e2754396818c548bf8c0a281.zip"
X-AspNet-Version: 4.0.30319
Set-Cookie: fileDownload=true; path=/
X-Powered-By: ASP.NET
Date: Wed, 09 Nov 2016 11:13:50 GMT
Content-Length: 1885473
PK.......... ZIP file data .............................................
After the interruption, this is the request that the browser does on resume (notice the GET method used):
GET http://localhost:35547/Download?Guid=396b4697-e275-4396-818c-548bf8c0a281 HTTP/1.1
Host: localhost:35547
Connection: keep-alive
Referer: http://localhost:35547/File/396b4697-e275-4396-818c-548bf8c0a281
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8
Cookie: __RequestVerificationToken=.............
(Some data from security-related cookies have been shortened and altered)
Am I missing something ?
It depends, upon how have you analyzed the behavior of google chrome. Ideal way will be use any proxy or to use packet sniffer such as Wireshark to see what request method, is used by chrome in subsequent request.
Is this something related to the HTTP specs that only allow GET
requests to be resumed?
As of now, there's no mention in spec of HTTP protocol, that only GET requests can be resumed.
Or is it simply a design flaw in Chrome (and maybe other browsers as
well) that makes it forget the original HTTP method used?
Yes, It's the flaw of google chrome. Make sure that you check it on the latest version of Google chrome with all the update patches. Also check it on other browsers.
For more info about HTTP protocol, refer to https://www.ietf.org/rfc/rfc2616.txt.
Refer to following request, for serving partial response : https://en.wikipedia.org/wiki/Byte_serving
Edit
For more updated info regarding HTTP info, refer to :-
https://www.rfc-editor.org/rfc/rfc7230
Despite what the title may suggest, this is not related to the common "preventing double post request" issue.
In my application, I add some data on outgoing HTTP traffic, and with some some websites, I randomly encounter an HTTP POST request that has a double-header, resulting in a server termination, as I assume the server doesn't understand the request and decides to cut me off. As the title says, I'm literally seeing two POST headers in a single request. This only happens when I append some custom HTTP fields to the header. For example, I came across this today when I followed a surveygizmo.com link, as seen in the trace at the bottom of this post.
I cannot understand if it's the browser that's doing something funky because it noticed I've modified some data, or it's something in my LSP application that causes this to happen.
When I debug my application, I only see the intercepted request the first time, which is when I inject the custom data. After that, I don't see the request anywhere except in Wireshark, so it's not like I can remediate the double headers by deleting the redundant data.
Things to note looking at the trace:
The data I'm appending is 'Custom-FieldN:'
Two almost-identical headers
Three double-CRLF's in one single request header (how is that possible?)
The Request:
POST http://www.surveygizmo.com/s3/1212345/Who-Are-You HTTP/1.1
Host: www.surveygizmo.com
Custom-Field1: UserNameBob
Custom-Field2: 2578291789
proxy-connection: keep-alive
Content-Length: 836
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: http://www.surveygizmo.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryaQraA7ZABICMT6jO
Referer: http://www.surveygizmo.com/s3/1212345/Who-Are-You
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,ja;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: sg-response-979523-1212345=0%3B1369154430_519ba37e477bc8.35524744%3B1369154430%3BPartial
POST http://www.surveygizmo.com/s3/1212345/Who-Are-You HTTP/1.1
Host: www.surveygizmo.com
Custom-Field1: UserNameBob
Custom-Field2: 2578291789
proxy-connection: keep-alive
Content-Length: 836
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: http://www.surveygizmo.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryaQraA7ZABICMT6jO
Referer: http://www.surveygizmo.com/s3/1212345/Who-Are-You
accept-encoding: gzip,deflate
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,ja;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: sg-response-979523-1212345=0%3B1369154430_519ba37e477bc8.35524744%3B1369154430%3BPartial
------WebKitFormBoundaryaQraA7ZABICMT6jO
Content-Disposition: form-data; name="sg_navchoice"