I have been working towards switching the communication protocol for our application from HTTP/1.1 to HTTP/2.
The communication flow is some thing like this:
Client talks to an Amazon Application load balancer over HTTP/2
Application load balancer talks to a reverse proxy (HAProxy) over HTTP/1.1
Reverse proxy then talks to the webserver over HTTP/1.1
I wanted all of this to be HTTP/2 but due to a limitation of the load balancer (https://forums.aws.amazon.com/thread.jspa?threadID=332847) the communication between it and the reverse proxy can either be HTTP/2 or HTTP/1.1 but not both. I need to support both because there is a WebSocket connection that is opened over HTTP/1.1.
I have an option to make the communication between HAProxy and the Webserver to be HTTP/2 as our Webserver support it.
So the flow becomes:
Client -> ALB (HTTP/2)
ALB -> HAProxy (HTTP/1.1)
HAProxy -> Webserver (HTTP/2)
I wanted to understand two things
If this is possible with HAProxy?
If this is a good move? will this give me any performance improvements?
Thanks in advance!
Cheers
Technically Ha Proxy can do HTTP / 2.0 end to end (version like 2.0 or newer https://www.haproxy.com/fr/blog/haproxy-2-0-and-beyond/#end-to-end-http-2)
And in 2.4 you can do HTTP/2 WebSockets (https://www.haproxy.com/fr/blog/announcing-haproxy-2-4/)
My first thought about HTTP/2.0 for multiplexing is interesting to reduce latency. Latency which is usually between the client and your fisrt instance (here ALB).
I don't know if you have latencies between HaProxy -> Webserver but if you think you have it. This brings a nice performance increase between HAProxy and your backend servers, since it condenses multiple connections down to a single connection. Otherwise do not expect a big improvement.
Moreover it could depend if you use TLS between Haproxy and webservers.
But options like headers compression and persistent TCP Connections are interesting so this makes sense to use it.
Related
What are common solutions getting gRPC app running when there is a requirement to run through a some sort of proxy which does not support HTTP/2 toward origin, rather towards client side.
Were you people got this kind of setup done somehow?
The setup via proxy would create a flow similar to this:
Client <--- HTTP/2 ---> Proxy <--- HTTP/1.1 ---> gRPC Server.
Currently, it's not possible -- but stay tuned: Piotr Sikora from Google is trying to get HTTP2 upstreams supported on nginx, even though things are proceeding slower than one would expect:
https://github.com/grpc/grpc.github.io/issues/230#issuecomment-306974585
https://forum.nginx.org/list.php?29 (look for Piotr)
I'm trying to load balance go servers in AWS beanstalk that uses GRPC/Protobuf for data serialization. Beanstalk makes offers nginx as reverse proxy for client-server communication which makes use of http1.1 protocol. This is resulting in bogus messages exchanged between proxy and server but client messages never seem to reach the server as intended. Any clean ideas would help here.
Nginx doesnt support http/2 to backend yet. Some of us are working on a fix for this but will take another quarter before we could get to upstream it. You can either wait for that or use Envoy (https://github.com/lyft/envoy) in front which supports grpc and http/2 natively. Hope this helps.
How do I proxy requests to NTLM-protected websites, like TeamFoundation and SharePoint? I keep getting 401 authentication errors.
According to this Microsoft TechNet article, you can't.
Microsoft NTLM uses stateful HTTP, which is a violation of the HTTP/1.1 RFC. It relies on authentication (an affair which involves a handshake with a couple of initial 401 errors) and subsequent connections to be done through the exact same connection from client to server. This makes HTTP proxying nearly impossible, since each request would usually go through either a new or a random connection picked from a pool of open connections. It can be done though.
NGiNX apparently supports this through the "ntlm" option, but this is part of their commercial offering. Apache HTTPD seems to have a couple of experimental patches for this, but this requires rebuilding Apache. TinyProxy doesn't support this either. HAProxy to the rescue!
Here is an example of a running configuration which works - it's a fairly simple setup with a single backend server:
backend backend_tfs
server static teamfoundation.mycompany.com:8080 check maxconn 3
mode http
balance roundrobin
option http-keep-alive
option prefer-last-server
timeout server 30s
timeout connect 4s
frontend frontend_tfs
# You probably want something other than 127.0.0.1 here:
bind 127.0.0.1:8080 name frontend_tfs
mode http
option http-keep-alive
timeout client 30s
default_backend backend_tfs
The important options here are http-keep-alive and prefer-last-server.
One more thing for my scenerio;
If you are using ssl both sides(the iis servers and haproxy), the ssl must be same for iis and haproxy server. Otherwise ntlm doesn't work when you want to go iis from haproxy.
Maybe can help someone who has the same problem.
I have HaProxy terminating SSL and passing the requests back to Varnish which then either serves the cached page or requests from Nginx. However, Varnish seems to be treating the request from HaProxy as HTTP/1 not HTTP/2 and failing to serve.
I can see in the Nginx logs the following when I try to hit a page:
" while reading PROXY protocol, client: 127.0.0.1, server: 127.0.0.1:8181
2016/08/11 06:53:31 [error] 5682#0: *1 broken header: "GET / HTTP/1.1
Host: www.example.com
User-Agent: curl/7.50.2-DEV
Accept: */*
X-Forwarded-For: IP_Removed
Accept-Encoding: gzip
X-Varnish: 32777
I've found something that relates to this here which states that the reason for this is that Nginx does not work with v2 PROXY only v1. So, as a result of this I've forced the use of protocol 1 in HaProxy using the send-proxy rather than send-proxy-v2 switch. But when it gets to Varnish I think that Varnish is converting this in some way to protocol 2 which is causing it to then fail to communicate properly with Nginx.
I have removed Varnish from the equation and connected HaProxy direct to Nginx and it works perfectly via HTTP/2. The problem is something is happening in the Varnish stack and the likely suspect is the proxy protocol v2 being used by Varnish.
So, to cut a long story short, how do I force Varnish to adhere to PROXY1 rather than PROXY2 protocol? I've tried adding PROXY1 into the launch daemon options but Varnish won't accept that. Any help is appreciated. Thanks!
UPDATE - I tested HaProxy > Nginx with the send-proxy-v2 switch on the HaProxy backend and it causes the identical problem to when Varnish is introduced into the stack. Switching back to send-proxy on HaProxy fixes the issue. So, I'm convinced that the issue is Varnish using protocol 2 rather than protocol 1. But how to tell it not to?
I understand that Varnish isn't HTTP/2 or does SSL but it should be passing the protocol back as is to Nginx no?
No.
But first, let's clarify. HTTP/2 and Proxy protocol V2 have absolutely nothing to do with each other. Remove HTTP/2 from your mind, as it is not applicable here in any sense.
Your question is, in fact, this:
If HAProxy is sending Proxy Protocol V1 to Varnish, and Nginx is configured behind Varnish to expect Proxy Protocol V1, why does Nginx complain of broken headers? Does Varnish not forward Proxy Protocol V1 to the backend? Does it for some reason send Proxy Protocol V2, instead?
And the answer to that question is that Varnish isn't sending either one. Neither V1 nor V2.
The only thing you need the Proxy protocol for is so that an HTTP-aware component can receive the client IP address (and port) from a upstream, non-HTTP-aware component, such as HAProxy using mode tcp or Amazon ELB with a listener in TCP mode, either of which is typically doing SSL offloading for you and not HTTP request routing, so it needs an alternative mechanism of passing the client address.
The first HTTP-aware component can take that address and set it in an HTTP header, customarily X-Forwarded-For, for the benefit of the remaining components in the stack. As such, there's no reason for Varnish to forward the Proxy protocol onward. It isn't doing that in your example, and there is no obvious reason why Varnish would even be capable of forwarding the Proxy protocol.¹
And this brings us to the error. You are misdiagnosing the problem that Nginx is reporting. The broken header error means that Nginx is receiving something other than Proxy protocol V1. With Varnish in the loop, there is no Proxy protocol header² present at all in the request to Nginx -- and when a listener is configured to expect the Proxy protocol header, that header is mandatory.
If a component is configured to expect Proxy protocol V1 and it is not present, that is always an error. But "not present" means exactly that. A V1 header is not present. That does not mean V2 is. It isn't.
So, I'm convinced that the issue is Varnish using protocol 2 rather than protocol 1.
You have convinced yourself incorrectly. Proxy V2 into Nginx -- as you have tried with HAProxy -- is an error, and no Proxy protocol header at all -- as you are seeing from Varnish -- is an error, as explained above. Both are misconfigurations, though of a different type. What you have done here is duplicated the error but for an entirely different reason.
If you are sending all requests through Varnish, then configure Varnish to set X-Forwarded-For in the forwarded request using the information it learns from the incoming Proxy protocol mesaage. Remove Proxy protocol from the Nginx configuration.
Or configure HAProxy to operate in HTTP mode and let it insert the header using option forwardfor.
¹ Clearly, from the error, Varnish is just sending ordinary HTTP headers -- nothing that looks like Proxy protocol. I don't think it even supports the option of sending Proxy protocol to the origin server, but somebody say something if I've overlooked that capability.
² I would assert that the Proxy protocol "header" is not properly called a header, given what that implies. It is a preamble, not a header, though it was unfortunately called a "header" in the standard. It's most certainly not an HTTP header.
If you upgrade Varnish to 5.0 it can send PROXY Protocol version 1 to NGINX by setting ".proxy_header = 1"
I'm just curious if I'm missing something in http2 that would make it more efficient in service-to-service communication, for example in a microservice architecture.
Are its improvements just related to end-users (browsers)?
If you are issuing many concurrent requests between microservices, then there's benefit from connection multiplexing. You do not need to manage TCP connection pools on the client, and restrict the number of incoming TCP connections at the service side.
Some services might benefit from server push, though it really depends on what the service does.
Headers compression can be useful if you have high traffic volumes to the service with repeated meta-data. More information can be found here.
In summary, yes, it is designed more with end users in mind, but there's value for RESTful microservices as well, especially due to connection multiplexing.
HTTP/2 adds an additional aspect to service-to-service communication that was not mandatory with HTTP/1.1. And that is security in form of SSL/TLS.
Although not required by the RFC standard, almost all HTTP/2 servers and clients will only support HTTP/2 over TLS, which makes encryption de facto mandatory.
So if you want to offer and consume microservices over HTTP/2, you have to think about ways to create, manage and distribute SSL-certificates to servers and clients.
Consequently, moving to HTTP/2 means introducing a new stack of technology, e.g. a public key infrastructure, to your service eco system.
Another way to make your services HTTP/2-ready for your service consumers would be to place a reverse proxy between your HTTP/2-enabled consumers and your HTTP/1.1 services.
The proxy would terminate the HTTP/2 connections from the consumers and translate them into HTTP/1.1 requests for your servers (and vise-versa).
This would implement a separation of concern, where your services would only be responsible for their business-logic stuff, while the proxies would handle the certificates and encryption. But again, you would add more complexity to your system.
More Complexity, but also better use of network resources
More complexity is what you are paying with. But you get a smarter consumption of network resources for it. With HTTP/1.1 you can have multiple TCP connections between one client and a server. And opening multiple connection is almost always necessary to overcome HTTP/1.1's performance drawbacks.
Establishing TCP connections is an expensive task, though. In order to create them DNS lookup, TCP handshake and SSL handshake are necessary.
HTTP/2 limits the number of open TCP-connections between one client and one server to exactly one (1). But at the same time, HTTP/2 brings us connection multiplexing, i.e. you can have multiple HTTP conversations simultaneously over the same TCP connection (HTTP/1.1: 1 TCP-connection = 1 HTTP connection).