Server does not receive HTTP headers that are sent by the browser? - http

Now this is extremely weird. Somehow my api backend does not receive the http headers (specifically the authorization header), that are clearly sent by the web application, as one can see in the dev tools. But when I try to send a request from GraphiQL or Postman with a authorization header the server DOES get them. How is this possible?
Log of received http headers in my backend:
map[Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9] Accept-Encoding:[gzip, deflate, br] Accept-Language:[de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7] Cache-Control:[no-cache] Connection:[keep-alive] Pragma:[no-cache] Sec-Fetch-Dest:[document] Sec-Fetch-Mode:[navigate] Sec-Fetch-Site:[none] Sec-Fetch-User:[?1] Upgrade-Insecure-Requests:[1] User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.163 Safari/537.36]]

I found out what was the problem... It was sending the OPTIONS request first, which is sent without this header. My api denied the request because of the missing auth header and so the final POST request never happened.

Related

A server responds differently to the same request sent from a browser and Fiddler

Opening an URL in an incognito tab of Chrome => the server responds with HTTP 200 OK code and a HTML page.
Replaying the same exact request using Fiddler => the server responds with HTTP 301 redirection to the same URL.
Closing the incognito tab, opening a new one, opening the same URL again => HTTP 200 OK code again...
I've verified that the request being sent from the browser is exactly the same every time you open that URL (well it can't be different because it's just opening an URL in an incognito tab)
How can the server distinguish the same request sent from a browser and from Fiddler?
First time I see such issue and I don't have any idea on how to dig deeper here...
The request looks like this:
GET <url> HTTP/1.1
Host: <host>
Connection: keep-alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding: gzip, deflate, sdch, br
Accept-Language: en-US,en;q=0.8
It seems this server has dropped the TLSv1.0 support and .NET Framework uses it by default.
There is a detailed answer on how to set the default SSL cipher to TLSv1.2 here: Are there .NET implementation of TLS 1.2?
By the way, it IS possible to enable TLSv1.2 support in .NET 4.0 apps using a registry trick described in one of the answers to that question. That is if the installed IE supports it of course.

HTTP 1.1 message protocol and server sent events

As I understand it, in 1.1 normally the browser issues a request and the server makes a single message response. The browser will not issue a new request until it has received the response to the previous message. So what ever response it receives is always interpreted as a response to the last message. Is my understanding correct?
When I open a page in Firefox, the server application parses the following request:
HGet / http/1.1
HeaderField(Host, localhost:8080)
HeaderField(User-Agent, Mozilla/5.0 (X11; Ubuntu; Linux x86_64;rv:41.0) Gecko/20100101 Firefox/41.0)
HeaderField(Accept, text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8)
HeaderField(Accept-Language, en-GB,en;q=0.5)
HeaderField(Accept-Encoding, gzip, deflate)
HeaderField(Connection, keep-alive)
After responding with the page this would normally be followed by a /favicon.ico request and things precede as expected. But now I have inserted the following line into my javascript to enable server sent events:
var evtSource = new EventSource("/");
which produces a second request:
HGet / http/1.1
HeaderField(Host, localhost:8080)
HeaderField(User-Agent, Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:41.0) Gecko/20100101 Firefox/41.0)
HeaderField(Accept, text/event-stream)
HeaderField(Accept-Language, en-GB,en;q=0.5)
HeaderField(Accept-Encoding, gzip, deflate)
HeaderField(Referer, http://localhost:8080/)
HeaderField(Connection, keep-alive)
HeaderField(Pragma, no-cache)
HeaderField(Cache-Control, no-cache
So can I now send multiple messages when ever I want from the server? (Leaving aside timeout issues) If so how does the browser know to which request, the message (coming from the server) is a response? Does it rely on the contentType header field? Should I use a different uri in the event source? As I'm learning to keep things simple I'm not using encryption, which stops me using HTTP 2. but later I intend to use https. My preference for using the same uri for the normal get and post requests as for the Server Sent Events is that I don't want to put unnecessary information in the unencrypted response line.
Edit: my confusion came from forgetting that http 1.1 browsers will open multiple connections if they need them. So in my simple setup the browser only has one connection open, it converts that one into a Server Sent Event connection and then makes future requests on a new connection with a different client side port number.
So can I now send multiple messages whenever I want from the server?
Yes.
how does the browser know to which request, the message (coming from the server) is a response?
It's not a request. It is an open HTTP connection. So the browser is only receiving events on the open connection that it is holding open.
Make sense?

HTTP Error 400 : The request verb is invalid in IIS 8.5

I have hosted ASP NET 4.5 application in Windows Server 2012 R2 (IIS Version 8.5.9600.16384). But once in a while I am getting an error like Bad Request - Invalid Verb HTTP Error 400. The request verb is invalid while going from one page to another. There is no special sequence or specific page where I am getting this error.
httperr log file shows me an error like
2015-01-21 04:48:18 MyIP 55452 ServerIP 80 - - - 400 - Verb -
2015-01-21 04:48:20 MYIP 55454 SerevrIP 80 - - - 400 - Verb -
I checked this post http://support.microsoft.com/kb/828726 but here it is applicable for Microsoft Internet Information Services (IIS) 6.0. Can I do this hotfix for iis 8.5 as well. I have searched about this issue over the internet but could not get substantial information. So, can anyone help me resolve this issue.
Using Fiddler I got the Raw Data of the request as
POST http://example.com/Project.aspx?prj=5566 HTTP/1.1
Host: example.com
Connection: keep-alive
Content-Length: 30634
Cache-Control: max-age=0
Authorization: Negotiate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Origin: http://example.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.99 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Referer: http://example.com/Project.aspx?prj=5566
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8
Cookie: ASP.NET_SessionId=4oymd5odmcaluiuzc24tbx22; _gat=1; _ga=GA1.2.1888408676.1418017144
The Request Filtering Setting of Windows Server 2012 R2 is
Any help would be appreciated.
Try adding
EnableViewState="false"
to your aspx page. It looks like Chrome adds the viewstate where IIS expects the request verb.
You might have some luck with this: Fix HTTP.sys
400 Status Code
400 status means “Bad Request”. So this is not a server-side issue. There is something wrong with the request. Is not well-formed before reaching to IIS
If the substatus code was something other than 0, it would be easier to troubleshoot. For example, If it was 1, it would mean “Invalid Destination Header”. If it was 2, that would mean “Invalid Depth Header”. Here is the full list of status codes.
If you see value 64 in sc-win32-status, it means there is a network-related issue. It refers to “The specified network name is no longer available”. After sending the response, IIS waits for ACK package from the client. If the client resets the connection instead of sending this package, IIS logs 64 code since it’s not a graceful connection close.
Check HTTPERR log for the same timestamp. In my case, I saw Timer_EntityBody error which means “The connection expired before the request entity body arrived”.
Recommendations
Check if there are software like antivirus or network endpoint
protection in the server. Uninstall them and monitor the system for a
while to see if the issue occurs again (Note: disabling them
sometimes doesn't mean they don't run in the background anymore.
Uninstalling is the ultimate way of eliminating them from the
troubleshooting process)
Work with the third-party product support to
see if there is anything blocking the requests to be fully sent.
Application might be initiating the connection but not completing it
Check with your Networking team to analyze the network between the
web server and client. Ideally, a network trace would be very helpful
Source: Status Code 400 with 64 in sc-win32-status column

How to force full HTTP delivery when client requests Range?

Let's say a client makes a request like the following (pulled from iOS):
GET /test.mp4 HTTP/1.1
Host: example.com:80
Range: bytes=0-1
X-Playback-Session-Id: 3DFA3BE3-CB22-4EC5-808F-B59A735DCECE
Accept-Encoding: identity
Accept: */*
Accept-Language: en-us
Connection: keep-alive
User-Agent: AppleCoreMedia/1.0.0.11B554a (iPad; U; CPU OS 7_0_4 like Mac OS X; en_us)
There other such requests out there, I believe Chrome might test the waters by asking for blank Range.
How can the server respond to any such request so that it does not need to honor Range , but rather treat it as a standard HTTP delivery, and the client will play the file?
Sending a regular header response and the data as though the client were not asking for Range does not seem to work.
EDIT: Conversely, if the client does not request a Range, is it okay to respond with HTTP 206 with full filesize in Content-Length and also Content-Range header (which client will ignore)?
If the server does not support the Range header, it would send a normal 200 reply to send the entire file. If the server supports the Range header, it would send a 206 or 416 reply, depending on whether the requested range can be satisfied or not. This is covered in RFC 2616 Section 14.35.
It is not OK to respond with 206 if the client did not request a Range.
Try responding with HTTP 1.0 - it doesn't support range requests at all.
Maybe the client will treat such a reply more gracefully.

http client got bad request page from google

I sent the following http request to google, and it returned a bad request page. Was there any wrong in my request? I was implementing a proxy server with C++. I redirected clients' requests to servers they want to connect to. Before redirecting, I inserted "\r\nConnection: close" to the request. Was the position I inserted to wrong? Thanks. (I use "###" to surround the request)
###GET http://www.google.com.tw/ HTTP/1.1
Host: www.google.com.tw
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:11.0) Gecko/20100101 Firefox/11.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: zh-tw,en-us;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Proxy-Connection: keep-alive
Connection: close
Cookie:***
###
What you have in there is not correct per spec, although I wouldn't be surprised if some servers actually responded to it (but not Google's).
Proxy-Connection is a misnomer and not needed at all.
The GET request should provide a relative path, not an absolute one. To be clear: the client does need to send a full address in the GET header, but the proxy needs to extract it and rewrite it such that GET carries the path, and Host header carries the hostname.
To try a couple of simple experiments, simply telnet google.com 80 and copy paste your request followed by few CRLF's.

Resources