Despite what the title may suggest, this is not related to the common "preventing double post request" issue.
In my application, I add some data on outgoing HTTP traffic, and with some some websites, I randomly encounter an HTTP POST request that has a double-header, resulting in a server termination, as I assume the server doesn't understand the request and decides to cut me off. As the title says, I'm literally seeing two POST headers in a single request. This only happens when I append some custom HTTP fields to the header. For example, I came across this today when I followed a surveygizmo.com link, as seen in the trace at the bottom of this post.
I cannot understand if it's the browser that's doing something funky because it noticed I've modified some data, or it's something in my LSP application that causes this to happen.
When I debug my application, I only see the intercepted request the first time, which is when I inject the custom data. After that, I don't see the request anywhere except in Wireshark, so it's not like I can remediate the double headers by deleting the redundant data.
Things to note looking at the trace:
The data I'm appending is 'Custom-FieldN:'
Two almost-identical headers
Three double-CRLF's in one single request header (how is that possible?)
The Request:
POST http://www.surveygizmo.com/s3/1212345/Who-Are-You HTTP/1.1
Host: www.surveygizmo.com
Custom-Field1: UserNameBob
Custom-Field2: 2578291789
proxy-connection: keep-alive
Content-Length: 836
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: http://www.surveygizmo.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryaQraA7ZABICMT6jO
Referer: http://www.surveygizmo.com/s3/1212345/Who-Are-You
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,ja;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: sg-response-979523-1212345=0%3B1369154430_519ba37e477bc8.35524744%3B1369154430%3BPartial
POST http://www.surveygizmo.com/s3/1212345/Who-Are-You HTTP/1.1
Host: www.surveygizmo.com
Custom-Field1: UserNameBob
Custom-Field2: 2578291789
proxy-connection: keep-alive
Content-Length: 836
Cache-Control: max-age=0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Origin: http://www.surveygizmo.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryaQraA7ZABICMT6jO
Referer: http://www.surveygizmo.com/s3/1212345/Who-Are-You
accept-encoding: gzip,deflate
Accept-Language: en-GB,en;q=0.8,en-US;q=0.6,ja;q=0.4
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: sg-response-979523-1212345=0%3B1369154430_519ba37e477bc8.35524744%3B1369154430%3BPartial
------WebKitFormBoundaryaQraA7ZABICMT6jO
Content-Disposition: form-data; name="sg_navchoice"
Related
when investigating network behavior, I usually use postman for sending HTTP requests, however
I need the option to send a raw HTTP request (via clear text), or at least only the headers, and it seems that postman does not support to edit your request via clear HTTP text. (buy the way the opposite is possible, you can read the raw http text of the requests you constructed in postman but you can't edit them)
for example:
Accept: text/javascript, text/html, application/xml, text/xml, */*
Accept-Encoding: gzip, deflate, br
Accept-Language: he-IL,he;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 21114
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Cookie: csrftoken=0alLaljTasofjCWZv7gcmukXuz6gMxfzlWpV691hzZZ1hTBcdVJ3mH8ozRDnO6hu; tk_or=%22%22; tk_lr=%22%22; session_id_12211=ff6a58b0baf98005748ce5a3c6a732aef33b750f; splunkweb_csrf_token_12211=10024448868272708216; token_key=10024448868272708216; experience_id=4852e1c6-726b-1ab3-bafa-f0a735d3f708; splunkd_12211=NjcrwAj_TLgz5JalVh2HTynLdbp_CPnfHFKi8qmsODiH40HI2urbPvAvJ9uvDKKoM3nATXEkS6dGytD0TvfiOtAUGJhk7Od25on_gJcZrQwcePQZ8HQaCmGScm^RXmOdDa^KVvN
Host: localhost:12211
Origin: http://localhost:12211
Pragma: no-cache
sec-ch-ua: "Chromium";v="106", "Google Chrome";v="106", "Not;A=Brand";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36
X-Requested-With: XMLHttpRequest
X-Splunk-Form-Key: 10024448898272708216
does postman allow editing the raw HTML? if not, there is other tool that can?
I turned on the fidder capture and entered username and password of my application and clicked login button. Now the username and password is displaying in the fiddler.
I am using primefaces. I dont want to show the password.
I added method="post" in form. but it doesnt work. Please help.
Fiddler Capure is below:
POST http://localhost:8186/myapp/ui/login.xhtml HTTP/1.1
Host: localhost:8186
Connection: keep-alive
Content-Length: 391
Accept: application/xml, text/xml, /; q=0.01
Origin: http://localhost:8186
X-Requested-With: XMLHttpRequest
Faces-Request: partial/ajax
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://localhost:8186/myapp/ui/login.xhtml
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cookie: JSESSIONID=025A4CEED3F44EEA80E08B00154EC1EB
javax.faces.partial.ajax=true&javax.faces.source=myloginForm%3AloginButton&javax.faces.partial.execute=%40all&javax.faces.partial.render=myloginForm%3AouterPanel+myloginForm%3AstatusMsgPanel&myloginForm%3AloginButton=myloginForm%3AloginButton&myloginForm=myloginForm&myloginForm%3AuserName=user1&myloginForm%3Apassword=pass1&javax.faces.ViewState=5967922235798284125%3A-8394345289058332812
You should understand that any HTTP message by default isn't encrypted. That means that all the data is moved via insecure connection. You may even notice a warning by the browser that your connection isn't safe and you should not share any private data. The solution for this is SSL/TLS encryption. It allows you to encrypt your connection from both sides so the information would be much harder to get.
Depending of your programming language, this part should be done from the server side.
I would like to make a file download resumable using byte-range requests.
The problem is that my existing download action is responding on a POST method and I would like to keep it that way.
But it seems from my early tests that Chrome turns interrupted POST requests for file downloads into GET requests when the user tries to resume and thus the resuming of the download fails.
Am I missing something?
Is this something related to the HTTP specs that only allow GET requests to be resumed?
Or is it simply a design flaw in Chrome (and maybe other browsers as well) that makes it forget the original HTTP method used?
UPDATE:
Here are the request/response data:
Initial POST request:
POST http://localhost:35547/Download?Guid=396b4697-e275-4396-818c-548bf8c0a281 HTTP/1.1
Host: localhost:35547
Connection: keep-alive
Content-Length: 0
Cache-Control: max-age=0
Origin: http://localhost:35547
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer: http://localhost:35547/File/396b4697-e275-4396-818c-548bf8c0a281
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: __RequestVerificationToken=LuPgM05MHrsuyskgfhsrHVUs; ASP.NET_SessionId=gfiulghfuygisghkf; .ASPXAUTH=FGDJHGDHSDFB15AFDE6371CGJHDFGFBHD; fileDownload=true
Initial response (to the request above):
HTTP/1.1 200 OK
Cache-Control: private, s-maxage=0
Content-Type: application/zip
Server: Microsoft-IIS/7.5
X-AspNetMvc-Version: 5.2
Content-Disposition: attachment; filename="FILE-396b4697e2754396818c548bf8c0a281.zip"
X-AspNet-Version: 4.0.30319
Set-Cookie: fileDownload=true; path=/
X-Powered-By: ASP.NET
Date: Wed, 09 Nov 2016 11:13:50 GMT
Content-Length: 1885473
PK.......... ZIP file data .............................................
After the interruption, this is the request that the browser does on resume (notice the GET method used):
GET http://localhost:35547/Download?Guid=396b4697-e275-4396-818c-548bf8c0a281 HTTP/1.1
Host: localhost:35547
Connection: keep-alive
Referer: http://localhost:35547/File/396b4697-e275-4396-818c-548bf8c0a281
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8
Cookie: __RequestVerificationToken=.............
(Some data from security-related cookies have been shortened and altered)
Am I missing something ?
It depends, upon how have you analyzed the behavior of google chrome. Ideal way will be use any proxy or to use packet sniffer such as Wireshark to see what request method, is used by chrome in subsequent request.
Is this something related to the HTTP specs that only allow GET
requests to be resumed?
As of now, there's no mention in spec of HTTP protocol, that only GET requests can be resumed.
Or is it simply a design flaw in Chrome (and maybe other browsers as
well) that makes it forget the original HTTP method used?
Yes, It's the flaw of google chrome. Make sure that you check it on the latest version of Google chrome with all the update patches. Also check it on other browsers.
For more info about HTTP protocol, refer to https://www.ietf.org/rfc/rfc2616.txt.
Refer to following request, for serving partial response : https://en.wikipedia.org/wiki/Byte_serving
Edit
For more updated info regarding HTTP info, refer to :-
https://www.rfc-editor.org/rfc/rfc7230
To send an HTTP request through a socket to a server, i would do something like this:
GET / HTTP/1.0
Host: www.example.com
User-agent: SomeBot
...
How would you go about programmatically defining an HTTPS request? I'm not looking for any programming language specific answer, something that teaches me the essence of HTTPS.
My research:
When i goto https://www.google.co.in/webhp?tab=ww&ei=sg7MUvKgGoX_rAeKxoGIDg&ved=0CBQQ1S4 this is what i get:
GET https://www.google.co.in/webhp?tab=ww&ei=sg7MUvKgGoX_rAeKxoGIDg&ved=0CBQQ1S4 HTTP/1.1
:host: www.google.co.in
x-chrome-variations: COy1yQEIlLbJAQiftskBCKS2yQEIqLbJAQiptskBCL62yQEI8YPKAQ==
accept-encoding: gzip,deflate,sdch
accept-language: en-US,en;q=0.8
user-agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36
:path: /webhp?tab=ww&ei=sg7MUvKgGoX_rAeKxoGIDg&ved=0CBQQ1S4
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
:version: HTTP/1.1
cache-control: max-age=0
cookie: PREF=ID=c032cbb31701d0d8:U=3a8fed312bb2ee57:FF=0:LD=en:TM=1374381891:LM=1376055657:S=BooLSkeTxOsbOYls; NID=67=HDIT9zwo-KKhljgRnJMz4u_5L_qpj3FvsN9Y47dWZmByQRS4N8QYs64IcEjFYphs6YpbrmvgsejwaL5YwxzbkY_qYaKU7wBfDA9N955NznF7IIyeHxcQ5UX8Dm999AElAKdkyswNbwUx1WJZo5vEuIaqC4Hdw4AkjsdwmFjY4ujPiEAj72z93QpCLleM-NXOK8N5YWn8DqiteGrEZUQ3FdPK3vkfDet_GF3CcBnkiYWxXON6R8Kum8BWaJGtm9h5dA; SID=DQAAANQAAAB2hOHWGXo76aWm_lgruhW0NH_zbU26rK7YMM_uiyMRvIBoyiEb3Gn_j2AhtmM4v6a74DinFMAOIjq5N4g4JcAAXaMEXz1dUz8MVup_nt1udNM0hpvybeWPxE1xK8rvdL2ra9moRW58jRzzA0HdpmkrH_t2ZIQ7GhqJlxp6lOS_jfvmeeb3REYFp6Q08hRYvCRDmhYFQ7NSt_Ua_3EWu4d_o125kvZ0x0bwm7JDKEcO3S-b6SJ4KnAGIWYjQKPdirgIFEUm1vApvIr4hoa4Z01rBt9YTmhwdEG5KvJmjusPkQ; HSID=AoEbPqSO97tEXhBOd; SSID=Abkp9uP00vi4wX19_; APISID=idPvNkfOQ-W9vefw/AjRgJIuDHZMDnME-B; SAPISID=5UH5pOlPn4c_31En/AcwfUulAqos_McwmH
:scheme: https
referer: https://www.google.co.in/
:method: GET
I see that it is pretty understandable, yet slightly alien...
During development of an IIS module for basic authentication, I stocked to a problem. The module is working fine when browsing the pages, but when calling web-services it seems that request does not reach the module and some in-the-middle module takes control of request.
using fiddler, I found out when Content-type in http request header is set to application/json that in the middle module/handler is triggered. so following request does not work:
when working fine, the server should ask client to send the user credentials by setting the WWW-Authenticate header in response
GET /WebServices/service.asmx/someMethod?param=test HTTP/1.1
Host: localhost
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.152 Safari/537.22
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Content-Type: application/json
asdfasdf
asdfasdfasdf
response: notice the jsonerror header in response
HTTP/1.1 401 Unauthorized
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/7.5
jsonerror: true
X-Powered-By: ASP.NET
Date: Mon, 11 Mar 2013 23:49:02 GMT
Content-Length: 105
{"Message":"Authentication failed.","StackTrace":null,"ExceptionType":"System.In
validOperationException"}
where this one works fine: notice that there is no content-type
GET /WebServices/service.asmx/someMethod?param=test HTTP/1.1
Host: localhost
Connection: keep-alive
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.152 Safari/537.22
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-US,en;q=0.8
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
asdfasdf
asdfasdfasdf
and the correct response is: notice the WWW-Authenticate header in response
HTTP/1.1 401 Unauthorized
Location: http://localhost/WebServices/service.asmx/someMethod?param=test
Server: Microsoft-IIS/7.5
WWW-Authenticate: Basic
X-Powered-By: ASP.NET
Date: Mon, 11 Mar 2013 23:59:48 GMT
Content-Length: 0
Well, that in-the-middle module was ScriptModule where we had both 3.5 and 4.0 version being added in the config. inspecting them through dotpeek, I found that the script module checks request's content-type against being application/json and then tries to handle the request as a REST request or webservice call.
By removing them, nothing special happened. I assume that they are to be used when script manager or Microsoft Specific AJAX services are used. You can find more about it in
ASP.Net Ajax Programming Tricks