I have HaProxy terminating SSL and passing the requests back to Varnish which then either serves the cached page or requests from Nginx. However, Varnish seems to be treating the request from HaProxy as HTTP/1 not HTTP/2 and failing to serve.
I can see in the Nginx logs the following when I try to hit a page:
" while reading PROXY protocol, client: 127.0.0.1, server: 127.0.0.1:8181
2016/08/11 06:53:31 [error] 5682#0: *1 broken header: "GET / HTTP/1.1
Host: www.example.com
User-Agent: curl/7.50.2-DEV
Accept: */*
X-Forwarded-For: IP_Removed
Accept-Encoding: gzip
X-Varnish: 32777
I've found something that relates to this here which states that the reason for this is that Nginx does not work with v2 PROXY only v1. So, as a result of this I've forced the use of protocol 1 in HaProxy using the send-proxy rather than send-proxy-v2 switch. But when it gets to Varnish I think that Varnish is converting this in some way to protocol 2 which is causing it to then fail to communicate properly with Nginx.
I have removed Varnish from the equation and connected HaProxy direct to Nginx and it works perfectly via HTTP/2. The problem is something is happening in the Varnish stack and the likely suspect is the proxy protocol v2 being used by Varnish.
So, to cut a long story short, how do I force Varnish to adhere to PROXY1 rather than PROXY2 protocol? I've tried adding PROXY1 into the launch daemon options but Varnish won't accept that. Any help is appreciated. Thanks!
UPDATE - I tested HaProxy > Nginx with the send-proxy-v2 switch on the HaProxy backend and it causes the identical problem to when Varnish is introduced into the stack. Switching back to send-proxy on HaProxy fixes the issue. So, I'm convinced that the issue is Varnish using protocol 2 rather than protocol 1. But how to tell it not to?
I understand that Varnish isn't HTTP/2 or does SSL but it should be passing the protocol back as is to Nginx no?
No.
But first, let's clarify. HTTP/2 and Proxy protocol V2 have absolutely nothing to do with each other. Remove HTTP/2 from your mind, as it is not applicable here in any sense.
Your question is, in fact, this:
If HAProxy is sending Proxy Protocol V1 to Varnish, and Nginx is configured behind Varnish to expect Proxy Protocol V1, why does Nginx complain of broken headers? Does Varnish not forward Proxy Protocol V1 to the backend? Does it for some reason send Proxy Protocol V2, instead?
And the answer to that question is that Varnish isn't sending either one. Neither V1 nor V2.
The only thing you need the Proxy protocol for is so that an HTTP-aware component can receive the client IP address (and port) from a upstream, non-HTTP-aware component, such as HAProxy using mode tcp or Amazon ELB with a listener in TCP mode, either of which is typically doing SSL offloading for you and not HTTP request routing, so it needs an alternative mechanism of passing the client address.
The first HTTP-aware component can take that address and set it in an HTTP header, customarily X-Forwarded-For, for the benefit of the remaining components in the stack. As such, there's no reason for Varnish to forward the Proxy protocol onward. It isn't doing that in your example, and there is no obvious reason why Varnish would even be capable of forwarding the Proxy protocol.¹
And this brings us to the error. You are misdiagnosing the problem that Nginx is reporting. The broken header error means that Nginx is receiving something other than Proxy protocol V1. With Varnish in the loop, there is no Proxy protocol header² present at all in the request to Nginx -- and when a listener is configured to expect the Proxy protocol header, that header is mandatory.
If a component is configured to expect Proxy protocol V1 and it is not present, that is always an error. But "not present" means exactly that. A V1 header is not present. That does not mean V2 is. It isn't.
So, I'm convinced that the issue is Varnish using protocol 2 rather than protocol 1.
You have convinced yourself incorrectly. Proxy V2 into Nginx -- as you have tried with HAProxy -- is an error, and no Proxy protocol header at all -- as you are seeing from Varnish -- is an error, as explained above. Both are misconfigurations, though of a different type. What you have done here is duplicated the error but for an entirely different reason.
If you are sending all requests through Varnish, then configure Varnish to set X-Forwarded-For in the forwarded request using the information it learns from the incoming Proxy protocol mesaage. Remove Proxy protocol from the Nginx configuration.
Or configure HAProxy to operate in HTTP mode and let it insert the header using option forwardfor.
¹ Clearly, from the error, Varnish is just sending ordinary HTTP headers -- nothing that looks like Proxy protocol. I don't think it even supports the option of sending Proxy protocol to the origin server, but somebody say something if I've overlooked that capability.
² I would assert that the Proxy protocol "header" is not properly called a header, given what that implies. It is a preamble, not a header, though it was unfortunately called a "header" in the standard. It's most certainly not an HTTP header.
If you upgrade Varnish to 5.0 it can send PROXY Protocol version 1 to NGINX by setting ".proxy_header = 1"
Related
I have been working towards switching the communication protocol for our application from HTTP/1.1 to HTTP/2.
The communication flow is some thing like this:
Client talks to an Amazon Application load balancer over HTTP/2
Application load balancer talks to a reverse proxy (HAProxy) over HTTP/1.1
Reverse proxy then talks to the webserver over HTTP/1.1
I wanted all of this to be HTTP/2 but due to a limitation of the load balancer (https://forums.aws.amazon.com/thread.jspa?threadID=332847) the communication between it and the reverse proxy can either be HTTP/2 or HTTP/1.1 but not both. I need to support both because there is a WebSocket connection that is opened over HTTP/1.1.
I have an option to make the communication between HAProxy and the Webserver to be HTTP/2 as our Webserver support it.
So the flow becomes:
Client -> ALB (HTTP/2)
ALB -> HAProxy (HTTP/1.1)
HAProxy -> Webserver (HTTP/2)
I wanted to understand two things
If this is possible with HAProxy?
If this is a good move? will this give me any performance improvements?
Thanks in advance!
Cheers
Technically Ha Proxy can do HTTP / 2.0 end to end (version like 2.0 or newer https://www.haproxy.com/fr/blog/haproxy-2-0-and-beyond/#end-to-end-http-2)
And in 2.4 you can do HTTP/2 WebSockets (https://www.haproxy.com/fr/blog/announcing-haproxy-2-4/)
My first thought about HTTP/2.0 for multiplexing is interesting to reduce latency. Latency which is usually between the client and your fisrt instance (here ALB).
I don't know if you have latencies between HaProxy -> Webserver but if you think you have it. This brings a nice performance increase between HAProxy and your backend servers, since it condenses multiple connections down to a single connection. Otherwise do not expect a big improvement.
Moreover it could depend if you use TLS between Haproxy and webservers.
But options like headers compression and persistent TCP Connections are interesting so this makes sense to use it.
I saw a website that offers HTTP Reverse proxy and TCP Reverse Proxy, I know what TCP Reverse Proxy is, but I have no clue what the difference is between HTTP & TCP Reverse proxy.
Try to explain it to someone that would like to buy it, because i want to know how it works. Those website's that offers it usually says "1 Protected Domain".
Contrary to a simple TCP reverse proxy a HTTP proxy can choose the target HTTP server based on the content of the HTTP request, i.e. based on the target host name (Host header), the path, Cookies, the User-Agent etc. It can also include additional headers like X-Forwarded-For to include the clients original IP address.
I am using haproxy in front of my web-server for ssl termination.
I am forwarding request on port 81 if request is https and 80 if request is normal http-
backend b1_http
mode http
server bkend_server
backend b1_https
mode http
server bkend_server:81
Problem is, when haproxy sends request to back-end, it sends HTTP_HOST header as request.domain.com:81.
Is it possible in haproxy that I can send https request to back-end at specific port without appending the port in HTTP_HOST request header?
There are two issues, here.
First, there is no HTTP_HOST header. The header is Host:. It sounds like HTTP_HOST is something being generated internally by your web server or framework.
Second, HAProxy doesn't modify the Host: header just because your back-end is listening on a port other than 80. It doesn't actually modify the Host: header at all, unless explicit configured to, using a mechanism like reqirep ^Host: ... or http-request set-header host ....
You can confirm this with a packet capture. You should find that whatever HTTP_HOST is, the value is necessarily being generated internally on the back-end system itself, because it's not coming from HAProxy.
Imagine a webbrowser that makes an HTTP request to a remote server, such as site.example.com
If the browser is then configured to use a proxy server, let's call it proxy.example.com using port 8080, in which ways are the request now different?
Obviously the request is now sent to proxy.example.com:8080, but there must surely be other changes to enable the proxy to make a request to the original url?
RFC 7230 - Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing, Section 5.3.2. absolute-form:
When making a request to a proxy, other than a CONNECT or server-wide
OPTIONS request (as detailed below), a client MUST send the target
URI in absolute-form as the request-target.
absolute-form = absolute-URI
The proxy is requested to either service that request from a valid
cache, if possible, or make the same request on the client's behalf
to either the next inbound proxy server or directly to the origin
server indicated by the request-target. Requirements on such
"forwarding" of messages are defined in Section 5.7.
An example absolute-form of request-line would be:
GET http://www.example.org/pub/WWW/TheProject.html HTTP/1.1
So, without proxy, the connection is made to www.example.org:80:
GET /pub/WWW/TheProject.html HTTP/1.1
Host: www.example.org
With proxy it is made to proxy.example.com:8080:
GET http://www.example.org/pub/WWW/TheProject.html HTTP/1.1
Host: www.example.org
Where in the latter case the Host header is optional (for HTTP/1.0 clients), and must be recalculated by the proxy anyway.
The proxy simply makes the request on behalf of the original client. Hence the name "proxy", the same meaning as in legalese. The browser sends their request to the proxy, the proxy makes a request to the requested server (or not, depending on whether the proxy wants to forward this request or deny it), the server returns a response to the proxy, the proxy returns the response to the original client. There's no fundamental difference in what the server will see, except for the fact that the originating client will appear to be the proxy server. The proxy may or may not alter the request, and it may or may not cache it; meaning the server may not receive a request at all if the proxy decides to deliver a cached version instead.
Imagine we have http client, some proxy and web-server that serves as backend. The proxy is configured to cache the responses of the backend.
A request arrives, the proxy transfers it to the backend, the latter responses, the proxy caches the response and sends it to the client.
Imagine the backend has set some cache-related headers in its response to the proxy's request. For example:
Cache-Control: no-cache (or)
Cache-Control: max-age=100000 (or)
Expires: 'Next Friday'
A question is: Will the next request from the client be processed by the proxy in compliance with that headers?
Another flavour of the question: Is there a way for a proxy to understand that the resource is stale, except for its own static resource-lifetime settings?
Third variation: Can the client force the proxy to load the fresh resource version if the proxy's version of the resource is not considered stale by the proxy?
My question may look a little bit overgeneralized, not specific enough. I'll try to address this issue by working with browser + nginx proxy + nginx web-server setup. If my setup works correctly, in case some resource has been cached by proxy and nginx's proxy_cache_valid timeout is still on - nothing can prevent the proxy from serving a stale response; whatever I do the requests do not hit the backend.
It looks like nginx decides whether the cache is stale only basing on proxy_cache_valid setting, and the headers of backend's response do not matter at all. I'm wondering whether my guess is correct and whether it can be incorrect for some other http proxy setups, serving as reverse proxy such as nginx, office network internal proxy such as squid, internet public proxy.