I need your help.
I have implemented a haproxy configuration which correctly manages both http and websocket backends, except in one specific scenario.
Here below a summary about how this stuff works:
When I connect to :2703/webapp, haproxy correctly redirects to one of the two http configured backends (webapp-lb1 or webapp-lb2).
When I connect to :2703/webapp/events, haproxy correctly redirects to one of the two websocket configured backends (websocket-lb1 or websocket-lb2)
Webapp is a servlet running in apache tomcat.
When I stop one of the two backend tomcats, haproxy correctly switches to the other one (for both the http and the websocket).
On the contrary, when I try to simulate an outage of one of the http backends by stopping the webapp via the tomcat manager, haproxy reports a HTTP Status 404 error but does not switch to the other backend.
Being that I explicitly configured the http-check expect status 302 directive, I would expect that - in case of a 404 status - haproxy switches to the other backend.
I had a look at the haproxy official documentation and I also tested the http-check disable-on-404 configuration but this is not what I need, as the haproxy behavior remains exactly the same of the above one.
For info, activating the http-check disable-on-404, haproxy detects the DOWN of the backend I stopped but does nothing (which as far as I understand, is exactly what we have to expect from the http-check disable-on-404 configuration in case of 404 status); here below the haproxy log when this option is enabled:
Jul 23 14:19:23 localhost haproxy[4037]: Server webapp-lb/webapp-lb2 is stopping, reason: Layer7 check conditionally passed, code: 404, info: "Not Found", check duration: 0ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Here below an extract of my haproxy configuration:
frontend haproxy-webapp
bind *:2703
monitor-uri /haproxy_check
stats enable
stats uri /haproxy_stats
stats realm Strictly Private
stats auth admin:zqxwcevr
acl is_websocket url_beg /webapp/events
use_backend websocket-lb if is_websocket
default_backend webapp-lb
log global
backend webapp-lb
server webapp-lb1 192.168.136.129:8888 maxconn 400 check cookie webapp-lb1
server webapp-lb2 192.168.136.130:8888 maxconn 400 check cookie webapp-lb2
balance roundrobin
cookie JSESSIONID prefix nocache
log global
#http-check disable-on-404
option httpchk GET /webapp
http-check expect status 302
backend websocket-lb
server websocket-lb1 192.168.136.129:8888 maxconn 400 check
server websocket-lb2 192.168.136.130:8888 maxconn 400 check
balance roundrobin
log global
Please give me a hint as I am spending ages in reading documentation and forums with no success.
Thanks!
Related
I have Direct Connect over a fast pipe to an AWS VPC and I'd like to use a cluster of HAProxy instances in my VPC to reverse-proxy one or more S3 buckets. This is so my users on premises can enjoy the increased bandwidth.
I guess the main question is whether this is doable, with the follow-on, "Is there a better solution for this than HAProxy?" I don't want to use an explicit proxy like squid because my only use-case for this is S3.
Assuming HAProxy is fine, I did a quick dummy setup for one bucket as a POC. When I connect directly to the bucket without credentials (simply to test connectivity), I see the "Access Denied" XML response I expect. But when I connect to the reverse-proxy, it seems to redirect me to https://aws.amazon.com/s3/. How am I screwing this up?
Here's my config (replace MY_BUCKET with any bucket name):
global
daemon
maxconn 256
defaults
mode http
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
default_backend servers
backend servers
server server1 MY_BUCKET.s3.amazonaws.com:80 maxconn 100
UPDATE:
Per Pedro's request, here is the configuration I found that makes this work. As you can see, it's extremely bare bones. (I'm using an EC2 instance with two CPUs.)
global
daemon
nbproc 2
nbthread 1
defaults
mode tcp
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
frontend http-in
bind *:80
default_backend S3
backend S3
server server1 s3.amazonaws.com:80
Proxying from HAProxy to Amazon S3 is definitely possible! Below is a copy of our production config file. We are using HAProxy 2.2 but this should be backward compatible to older versions as well.
resolvers dns
parse-resolv-conf
hold valid 10s
frontend wwoof
mode http
bind *:80
default_backend s3
backend s3
mode http
http-request set-header Host your-bucket.s3-website.eu-west-3.amazonaws.com
http-request del-header Authorization
http-response del-header x-amz-id-2
http-response del-header x-amz-request-id
server s3 your-bucket.s3-website.eu-west-3.amazonaws.com:80 resolvers dns check inter 5000
Good day,
I want to connect to my kafka server from the internet. Kafka installed on the virtual server and all servers hidden behind a nginx.
I updated kafka settings (server.properties).
Added: listeners=PLAINTEXT://:9092
I can connect to kafka server from local network via ip address 10.0.0.1:9092, but unable connect from internet by domain name.
Response from kafka: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic test-topic not present in metadata after 60000 ms.
Nginx: [26/Nov/2019:12:38:25 +0100] "\x00\x00\x00\x14\x00\x12\x00\x02\x00\x00\x00\x00\x00" 400 166 "-" "-" "request_time=1.535" "upstream_response_time=-" "upstream_connect_time=-" "upstream_header_time=-"
nginx conf:
server {
listen 9092;
server_name site.name;
# Max Request size
client_max_body_size 20m;
location / {
proxy_pass http://10.0.0.1:9092;
}
}
Does anyone know what the problem is?
Kafka doesn't use http protocol for communication, so it can't be fronted by an HTTP reverse proxy.
You'll have to use nginx stream definition blocks for TCP proxying
(I've not tried this personally)
https://docs.nginx.com/nginx/admin-guide/load-balancer/tcp-udp-load-balancer/
unable connect from internet by domain name.
Sounds like an issue with your advertised.listeners configuration. Note that there is no clear way to "hide" Kafka behind a proxy since your clients are required to communicate directly with each broker individually (therefore defeating the purpose of having
Ngnix unless you want to use one Nginx server or open a new port, per broker), and would therefore also require Kafka to know that it would need to "advertise" the proxy rather than its own address.
If you really want to expose Kafka to the public web, you should really be using SSL/SASL listeners, not PLAINTEXT
If you want to use HTTP, then you can install Kafka REST Proxy, then put Nginx in front of that. Then your clients would use http rather than standard kafka libraries
I have a simple setup for HAProxy to a back-end server available over an IPSec VPN. When I connect directly to the back-end server using Curl, the requests goes through successfully, but when I use HAProxy to the same back-end, over the VPN, the request is dropped with a 503 ERROR. From the logs, it seems the connection is being aborted prematurely, but I cannot decipher why. Also, both requests work when I use remote servers available over the Internet as back-ends, where a VPN is not involved. Am I missing a specific config or something for HAProxy over a VPN?
Note: I have set no health check on purpose for the back-end
HAProxy config:
defaults
mode http
# option httplog
log global #use log set in the global config
log-format \"[Lo:%ci/%cp; Re:%si/%sp] [Proxy - %bi:%bp/%fi:%fp] [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r\"
option dontlognull
option http-keep-alive
option forwardfor except 127.0.0.0/8
option redispatch
retries 2
timeout http-request 10s #maximum allowed time to wait for a complete HTTP request
timeout queue 10s #maximum time to wait in the queue for a connection slot to be free
timeout connect 5s #maximum time to wait for a connection attempt to a server to succeed
timeout client 5s #minimum time for inactivity on client side
timeout server 5s #maximum inactivity time on the server side
timeout http-keep-alive 30s #maximum allowed time to wait for a new HTTP request to appear
timeout check 10s
maxconn 5000
##-----------------------------------------------------
## API Requests
##-----------------------------------------------------
## frontend to proxy HTTP callbacks coming from App servers to VPN Server
frontend api_requests
mode http
bind 10.132.2.2:80
bind 127.0.0.1:80
default_backend testbed
## backend to proxy HTTP requests from App Servers to VPN Server
backend testbed
balance roundrobin
server broker 196.XXX.YYY.136:80
Entry captured on traffic log for failed attempt over VPN :
May 30 09:15:10 localhost haproxy[22844]: [Lo:127.0.0.1/56046; Re:196.XXX.YYY.136/80] [Proxy - :0/127.0.0.1:80] [30/May/2019:09:15:10.285] api_requests testbed/broker 0/0/-1/-1/0 503 212 - - SC-- 1/1/0/0/2 0/0 "POST /request HTTP/1.1"
What could be the issue causing a Curl request to be accepted but a proxy request by HAProxy to be dropped specifically for a VPN connection? Anyone faced a similar issue before?
I've put my linux apache webserver running on GCP behind the google load balancer. Because i only want https traffic i've redirected port 80 to 443 as shown below:
<VirtualHost *:80>
ServerName spawnparty.com
ServerAlias www.spawnparty.com
DocumentRoot /var/www/html/wwwroot
Redirect permanent / https://www.spawnparty.com
</VirtualHost>
i've given the vm an external ip adress to test if the redirect works and it does.
I've then configured the load balancer. i've made it so that the frondend accepts both http and https. for the backend i made 2 services:
one that uses http and one that uses https so that if somoeone enters though http it is forwarded and then redirected to https by the code shown above.
for both backend services is made a basic health check:
for http: port: 80, timeout: 5s, check interval: 5s, unhealthy
threshold: 2 attempts
for https: port: 443, timeout: 5s, check interval: 5s, unhealthy
threshold: 2 attempts
the https one works fine and states 1 of 1 instance healthy but the http health check states 0 of 1 instance healthy
if change the health check from http to https and back again for the http back end service it works for a short period of time but after a few minutes it states 0 of 1 instance healthy again.
What must i change to keep it healthy?
TL;DR - Use the same HTTPS health check for both the backend services.
Health Checking and response codes
You will need to respond with 200 response code and close the connection normally within the configured period.
HTTP and HTTPS health checks
If traffic from the load balancer to your instances uses the HTTP or
HTTPS protocol, then HTTP or HTTPS health checks verify that the
instance is healthy and the web server is up and serving traffic.
For an HTTP(S) health check probe to be deemed successful, the
instance must return a valid HTTP response with code 200 and close the
connection normally within the configured period. If it does this a
specified number of times in a row, the health check returns a status
of HEALTHY for that instance. If an instance fails a specified number
of health check probes in a row, it is marked UNHEALTHY without any
notification being sent. UNHEALTHY instances do not receive new
connections, but existing connections are allowed to continue.
UNHEALTHY instances continue to receive health check probes. If an
instance later passes a health check by successfully responding to a
specified number of consecutive health check probes, it is marked
HEALTHY and starts receiving new connections, again without any
notification.
Since you have 2 separate backend services (one for HTTP and other for HTTPS), you will need 2 health checks (although backend services allows reusing the same health check too if needed - read on) since the load balancer considers them independent services.
As you have already confirmed, using the HTTPS health check will work with the HTTPS based service, but using the HTTP health check will not. The reason is you are actually returning a HTTP 301 response code for permanent URL redirection instead of the expected HTTP 200 response code.
Possible Solution(s)
One way to fix this is to use HTTPS health checks for both the backend services, since your underlying service is still the same. You lose the ability to health check the redirection, but that unfortunately is not supported by the Google Cloud Load Balancer. You can share the same HTTPS health check resource too for both the backend services.
The solution posted by CharlesB will also work, but I feel you're adding additional redirection rules just to satisfy health checks and are not used on your service path anyway. You will also need a separate HTTP health check resource. Using just HTTPS health checks for both the backend services I feel is much simpler and also verifies that your service is alive to handle new requests.
Redirect everything except your health check page to HTTPS. The How to force rewrite to HTTPS except for a few pages in Apache? question explains how you can do that. GCE Network load balancing mentions this requirement, saying "Even if your service does not use HTTP, you'll need to at least run a basic web server on each instance that the health check system can query."
How do I proxy requests to NTLM-protected websites, like TeamFoundation and SharePoint? I keep getting 401 authentication errors.
According to this Microsoft TechNet article, you can't.
Microsoft NTLM uses stateful HTTP, which is a violation of the HTTP/1.1 RFC. It relies on authentication (an affair which involves a handshake with a couple of initial 401 errors) and subsequent connections to be done through the exact same connection from client to server. This makes HTTP proxying nearly impossible, since each request would usually go through either a new or a random connection picked from a pool of open connections. It can be done though.
NGiNX apparently supports this through the "ntlm" option, but this is part of their commercial offering. Apache HTTPD seems to have a couple of experimental patches for this, but this requires rebuilding Apache. TinyProxy doesn't support this either. HAProxy to the rescue!
Here is an example of a running configuration which works - it's a fairly simple setup with a single backend server:
backend backend_tfs
server static teamfoundation.mycompany.com:8080 check maxconn 3
mode http
balance roundrobin
option http-keep-alive
option prefer-last-server
timeout server 30s
timeout connect 4s
frontend frontend_tfs
# You probably want something other than 127.0.0.1 here:
bind 127.0.0.1:8080 name frontend_tfs
mode http
option http-keep-alive
timeout client 30s
default_backend backend_tfs
The important options here are http-keep-alive and prefer-last-server.
One more thing for my scenerio;
If you are using ssl both sides(the iis servers and haproxy), the ssl must be same for iis and haproxy server. Otherwise ntlm doesn't work when you want to go iis from haproxy.
Maybe can help someone who has the same problem.