Say I make an HTTP request to: foosite.com but the port I actually send the request to is 6103 and I DON'T put that port in the Host header for example:
GET /barpage HTTP/1.1
Host: foosite.com
Method: GET
Should http server then recognize that I'm trying to talk to it on port 6103? Or since it was omitted in the request header am I gambling on if the server actually recognizes this?
I ask that question to say this: I've found that browsers, at least firefox + chrome, put the port in the Host header. But the Java app I'm using does not. And when the port is not passed in the Host the server responds back thinking I'm on port 80. So who do I need to badger? The server operator, or the Java programmer?
See section 14.23 of the HTTP spec which specifies that the port # should be included if it's not the default port (80 for HTTP, 443 for HTTPS).
UPDATED for modern day browsers:
Browsers (and curl) will add the port only when it is not the standard port, as required by the HTTP spec and noted in #superfell's answer.
Browsers this day (2013), will actually strip the port from the Host Header when the port is the standard (http port 80, https port 443). Some clients, which use their own method, like the Baidu Spider, include the port number even when the port is 80.
Whether this is proper or not, I don't know. The spec doesn't say whether it's OK or not to include the port number when the port used IS the default.
To answer your comment, servers will do whatever they need to do to comply with the spec, and the spec suggests only the cases WHEN it's needed. Because of this, I feel It's not really a question of how the server deals with it - it's more how the client issues the request: includes the port number in the Host Header, or not.
RFC2616 states that
A "host" without any trailing port information implies the default
port for the service requested (e.g., "80" for an HTTP URL). For
example, a request on the origin server for
http://www.w3.org/pub/WWW/ would properly include:
GET /pub/WWW/ HTTP/1.1
Host: www.w3.org
This means that https://example.com would not need a trailing port as well since the default port is known for https. I have checked the HTTP requests from Firefox, Chrome and Edge and found that none of them added the port number for the host header when the domain protocole was https. For sure the port number is added when the port number was also added to the URL. The following screenshots below come from Google chrome
Sample headers of an actual request to a hopefully non existent server 'http://myhost.com:3003/content/page.htm'
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: en-US;q=0.9,en;q=0.8,nb;q=0.7,de;q=0.6
Connection: keep-alive
Host: myhost.com:3244
Referer: http://myhost.com:3244/content/page.htm
The RFC https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html requires some training to read.
Section 14:24 not so easy to translate all elements to the simple reality:
Host = "Host" ":" host [ ":" port ] ;
Host Header Syntax:
Host: :
if its not default than put port after host:
Host: example.com:1337
Related
we know that we use tcp in http. In tcp connection Origin and destination are specify.according to this sentence why we specify host in Http header?
for example:
GET /index.html HTTP/1.1\r\n
Host: www-net.cs.umass.edu\r\n
User-Agent: Firefox/3.6.10\r\n
Accept: text/html,application/xhtml+xml\r\n
Accept-Language: en-us,en;q=0.5\r\n
Accept-Encoding: gzip,deflate\r\n
Accept-Charset: ISO-8859-1,utf-8;q=0.7\r\n
Keep-Alive: 115\r\n
Connection: keep-alive\r\n
\r\n
The reason it is specified is to specify the name of the host. Since the request itself is made to the IP address in the IP header, the web server has no real way to know which domain was requested.
Under HTTP 1.0, this was not needed. After HTTP 1.0 was in existence for a number of years, the prevalence of virtual hosts become very common. Virtual hosting, in case you are unaware, allows you to host multiple sites on a single server, even the same IP, responding to each domain differently.
Since virtual hosting had become very common, the HTTP 1.1 standard made the Host header a requirement, simplifying the logic necessary to figure out which site the request should be routed to.
I have HaProxy terminating SSL and passing the requests back to Varnish which then either serves the cached page or requests from Nginx. However, Varnish seems to be treating the request from HaProxy as HTTP/1 not HTTP/2 and failing to serve.
I can see in the Nginx logs the following when I try to hit a page:
" while reading PROXY protocol, client: 127.0.0.1, server: 127.0.0.1:8181
2016/08/11 06:53:31 [error] 5682#0: *1 broken header: "GET / HTTP/1.1
Host: www.example.com
User-Agent: curl/7.50.2-DEV
Accept: */*
X-Forwarded-For: IP_Removed
Accept-Encoding: gzip
X-Varnish: 32777
I've found something that relates to this here which states that the reason for this is that Nginx does not work with v2 PROXY only v1. So, as a result of this I've forced the use of protocol 1 in HaProxy using the send-proxy rather than send-proxy-v2 switch. But when it gets to Varnish I think that Varnish is converting this in some way to protocol 2 which is causing it to then fail to communicate properly with Nginx.
I have removed Varnish from the equation and connected HaProxy direct to Nginx and it works perfectly via HTTP/2. The problem is something is happening in the Varnish stack and the likely suspect is the proxy protocol v2 being used by Varnish.
So, to cut a long story short, how do I force Varnish to adhere to PROXY1 rather than PROXY2 protocol? I've tried adding PROXY1 into the launch daemon options but Varnish won't accept that. Any help is appreciated. Thanks!
UPDATE - I tested HaProxy > Nginx with the send-proxy-v2 switch on the HaProxy backend and it causes the identical problem to when Varnish is introduced into the stack. Switching back to send-proxy on HaProxy fixes the issue. So, I'm convinced that the issue is Varnish using protocol 2 rather than protocol 1. But how to tell it not to?
I understand that Varnish isn't HTTP/2 or does SSL but it should be passing the protocol back as is to Nginx no?
No.
But first, let's clarify. HTTP/2 and Proxy protocol V2 have absolutely nothing to do with each other. Remove HTTP/2 from your mind, as it is not applicable here in any sense.
Your question is, in fact, this:
If HAProxy is sending Proxy Protocol V1 to Varnish, and Nginx is configured behind Varnish to expect Proxy Protocol V1, why does Nginx complain of broken headers? Does Varnish not forward Proxy Protocol V1 to the backend? Does it for some reason send Proxy Protocol V2, instead?
And the answer to that question is that Varnish isn't sending either one. Neither V1 nor V2.
The only thing you need the Proxy protocol for is so that an HTTP-aware component can receive the client IP address (and port) from a upstream, non-HTTP-aware component, such as HAProxy using mode tcp or Amazon ELB with a listener in TCP mode, either of which is typically doing SSL offloading for you and not HTTP request routing, so it needs an alternative mechanism of passing the client address.
The first HTTP-aware component can take that address and set it in an HTTP header, customarily X-Forwarded-For, for the benefit of the remaining components in the stack. As such, there's no reason for Varnish to forward the Proxy protocol onward. It isn't doing that in your example, and there is no obvious reason why Varnish would even be capable of forwarding the Proxy protocol.¹
And this brings us to the error. You are misdiagnosing the problem that Nginx is reporting. The broken header error means that Nginx is receiving something other than Proxy protocol V1. With Varnish in the loop, there is no Proxy protocol header² present at all in the request to Nginx -- and when a listener is configured to expect the Proxy protocol header, that header is mandatory.
If a component is configured to expect Proxy protocol V1 and it is not present, that is always an error. But "not present" means exactly that. A V1 header is not present. That does not mean V2 is. It isn't.
So, I'm convinced that the issue is Varnish using protocol 2 rather than protocol 1.
You have convinced yourself incorrectly. Proxy V2 into Nginx -- as you have tried with HAProxy -- is an error, and no Proxy protocol header at all -- as you are seeing from Varnish -- is an error, as explained above. Both are misconfigurations, though of a different type. What you have done here is duplicated the error but for an entirely different reason.
If you are sending all requests through Varnish, then configure Varnish to set X-Forwarded-For in the forwarded request using the information it learns from the incoming Proxy protocol mesaage. Remove Proxy protocol from the Nginx configuration.
Or configure HAProxy to operate in HTTP mode and let it insert the header using option forwardfor.
¹ Clearly, from the error, Varnish is just sending ordinary HTTP headers -- nothing that looks like Proxy protocol. I don't think it even supports the option of sending Proxy protocol to the origin server, but somebody say something if I've overlooked that capability.
² I would assert that the Proxy protocol "header" is not properly called a header, given what that implies. It is a preamble, not a header, though it was unfortunately called a "header" in the standard. It's most certainly not an HTTP header.
If you upgrade Varnish to 5.0 it can send PROXY Protocol version 1 to NGINX by setting ".proxy_header = 1"
I am using haproxy in front of my web-server for ssl termination.
I am forwarding request on port 81 if request is https and 80 if request is normal http-
backend b1_http
mode http
server bkend_server
backend b1_https
mode http
server bkend_server:81
Problem is, when haproxy sends request to back-end, it sends HTTP_HOST header as request.domain.com:81.
Is it possible in haproxy that I can send https request to back-end at specific port without appending the port in HTTP_HOST request header?
There are two issues, here.
First, there is no HTTP_HOST header. The header is Host:. It sounds like HTTP_HOST is something being generated internally by your web server or framework.
Second, HAProxy doesn't modify the Host: header just because your back-end is listening on a port other than 80. It doesn't actually modify the Host: header at all, unless explicit configured to, using a mechanism like reqirep ^Host: ... or http-request set-header host ....
You can confirm this with a packet capture. You should find that whatever HTTP_HOST is, the value is necessarily being generated internally on the back-end system itself, because it's not coming from HAProxy.
Im in the process of implementing a RFC 7239 compatible forwarded header in an internal reverse proxy. In brief, the specification states that the values of the host and by components should be as follows:
host - the value of the host header as received by the proxy
by - the user agent facing interface of the proxy
Whats the difference between these two? If i have a proxy server facing the internet on the address http://myexampleserver.com, as i understand it both host and by would have the same value?
No, the original request will contain the requested site's DNS address as host header, not the proxy's DNS address.
The RFC mentions this host value's intended use:
This can be used, for example, by the origin
server if a reverse proxy is rewriting the "Host" header field to
some internal host name.
So, for example:
User agent requests http://example.com/foo through proxy http://yourexampleproxy/. Request will contain GET / http://example.com/foo and Host: example.com.
Your proxy translates, by configuration, the Host: example.com header to Host: some-internal-foo, and adds the Forwarded: host=example.com;by=yourexampleproxy header, so the origin server can inspect it.
I need help. I need log(safe full request) in iis 5(with headers etc) or look on it in proxy,fiddler etc
I use fiddler/ I have web config
Do you know HOW TO see ALL REQUEST,request from ALL PORTS and applications?
Can y recommend me proxy or http debugger?
I can not see request to my website in fiddler((((((
Maybe i need add propertyes in iis? I have tcp port in iis(default web site, and this site is default in root of wwwroot
TCP port 80? and proxy address 8888????????????? MAYBE
Maybe i need change port of tcp to 8888?
I add to web.comfig
In fiddler is option Monitor all connections i check it. I don't use filters
I have web site on my own machine and fiddler on it
GMAIL send request on my site( get rss)
74.125.16.68 - W3SVC1 HOUSE 217.76.185.140 80 GET /24.rss - 200 1398 202 HTTP/1.1 217.76.185.140
I don't see this query in fiddler, but i see
GET /mail/channel/bind?at=xn3j35onw91kr1q7zyzwjdx2653kr7&VER=6&it=548812&RID=rpc&SID=8787BE0499898773&CI=1&AID=78&TYPE=html&zx=bf28pr-v4mnm7&DOMAIN=mail.google.com&t=1 HTTP/1.1
Accept: /
Referer: http://mail.google.com/mail/?ui=2&
IN FIDDLEr i see
POST /mail/?ui=2&ik=ba4ed7ee39&view=cps&q=http%3A%2F%2F217.76.185.140%2F25.rss&cps=r&rt=j HTTP/1.1
answer
Content-Type: text/javascript; charset=UTF-8
while(1);
[[["v","NEGbCBCSn-c.en.","8","55f6abc2045673de",,]
,["di",749,"",""]
,["ub",[["^i",1240327921200]
,["^f",1240327921200]
CHEERS
Unfortunately, I have no experience with fiddler (although I hear it is cool).
Try wireshark: here: http://www.wireshark.org/
It is a network packet analyzer with filtering, so you could set the filters to capture the messages you want.