I have a brand new Azure Load Balancer configured in private mode and VMSS (Single Server) configured with nginx and the default site. Any time I try to use the load balancer nginx returns a 400 response but if I use the server directly I get a 200 response.
Further looking at the access logs I see this ->
xxx.xxx.xxx.xxx - - [30/Jun/2021:17:51:48 +0000] "\x00" 400 166 "-" "-"
xxx.xxx.xxx.xxx - - [30/Jun/2021:17:51:51 +0000] "GET / HTTP/1.1" 304 0 "-" "{Browser Info ...}"
When using the load balancer, the path is \x00 instead of / - I'm not sure what is going on here or where to look.
This was caused by a private link service configured for TCP proxy V2 that was configured on the Load Balancer
Related
I'm confused by the different logs, one reporting http2, the other http 1.0.
I'm not sure which config file to cite. Or if it's a normal occurrence for puma's stdout redirect to show 1.0 for http version? Thank you.
nginx
==> /var/log/nginx/access.log <==
[10/Oct/2021:05:45:15 +0000] "GET /users/Ovbzv/quickrates/o5l05/payment/YabQ0/pending HTTP/2.0" 200
puma
==> app/log/stdout.log <==
[5626] 2604:a880:800:10::637:b005 - - [10/Oct/2021:05:45:15 +0000] "GET /users/Ovbzv/quickrates/o5l05/payment/YabQ0/pending HTTP/1.0" 200 - 0.0826
You are looking at two different connections:
the connection between the client and nginx (the reverse proxy); and
the connection between nginx and Puma;
In this specific case, each of these connections is using a different HTTP version, as indicated by the logs.
This is easily possible because HTTP/2 was specifically designed with some backwards compatibility in mind, allowing HTTP/2 to be converted to HTTP/1 when in need (and the same goes to converting HTTP/1 to HTTP/2).
I want to put few services behind reverse proxy, simple services work.
The 502 issue occurs when trying to reach grafana simple json data source (POST + json payload).
Grafana itself is not behind reverse proxy.
current haproxy config:
frontend FRONT.AWS.WEB.PROXY
mode http
bind *:8080
timeout client 1m
option httplog
acl IS_RELE path_beg /release
acl IS_GRAF path_beg /grafana
use_backend BACK.AWS.WEB.ARTIFACTS if IS_RELE
use_backend BACK.AWS.WEB.RTGRAF if IS_GRAF
backend BACK.AWS.WEB.ARTIFACTS
mode http
http-request set-path /
http-response replace-value X-Application-Context (.*)(\release).*$ \1
server AWS.WEB.ARTIFACTS *:5581/ maxconn 1000 check port 5581
backend BACK.AWS.WEB.RTGRAF
mode http
#option forwardfor
#balance source
#option httpclose
#option httpchk HEAD / HTTP/1.0
http-request set-path /
http-response replace-value X-Application-Context (.*)(\grafana).*$ \1
server AWS.WEB.RTGRAF *:5582/ maxconn 1000 check port 5582
data source config in grafana:
http://192.168.56.101:8080/grafana/
This is working request without haproxy:
curl -d '{"requestId":"Q119","timezone":"utc".....lters":[]}' -H 'Content-Type: application/json' http://localhost:8080/query
good response:
[{"columns":[{"text":"sym","type":"string"}, {"text":"time","type":""}, .... .... {"text":"mode","type":"string"}, {"text":"proto","type":"string"}],"rows":[],"type":"table"}]
BUT with haproxy:
curl -d .... http://localhost:8080/grafana/query
502 Response:
<h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
but just to confirm, service itself works:
curl http://localhost:8080/grafana/?2+1
Response:
<html><head><style>a{text-decoration:none}a:link{color:024C7E}a:visited{color:024C7E}a:active{color:958600}body{font:10pt verdana;text-align:justify}</style></head><body><pre>3
Haproxy log:
127.0.0.1:42362 [16/Sep/2020:21:57:14.430] FRONT.AWS.WEB.PROXY BACK.AWS.WEB.RTGRAF/AWS.WEB.RTGRAF 0/0/0/3/3 200 274 - - ---- 1/1/0/0/0 0/0 "GET /grafana/?2+1 HTTP/1.1"
127.0.0.1:42418 [16/Sep/2020:21:57:32.038] FRONT.AWS.WEB.PROXY BACK.AWS.WEB.RTGRAF/AWS.WEB.RTGRAF 0/0/0/-1/0 502 214 - - PH-- 1/1/0/0/0 0/0 "POST /grafana/query HTTP/1.1"
grafana log:
INFO[09-16|22:23:02] Request Completed logger=context userId=1 orgId=1 uname=admin method=POST path=/api/datasources/proxy/2/query status=502 remote_addr=192.168.56.1 time_ms=6 size=107 referer="http://192.168.56.101:3000/d/aQPWEJFmz/system-status?orgId=1&refresh=10s"
Found the problem, for anyone in the future, that's for you!
....................................................caring ancient developers
Debug requests with simple nc
You'd find, wrong path is requested from first HTTP GET
So that needs rewriting:
backend BACK.AWS.WEB.ARTIFACTS
mode http
http-request set-uri %[url,regsub(^/release/,/,)]
http-response replace-value X-Application-Context (.*)(\release).*$ \1
server AWS.WEB.ARTIFACTS *:5581/ maxconn 1000 check port 5581
backend BACK.AWS.WEB.RTGRAF
mode http
http-response replace-value X-Application-Context (.*)(\grafana).*$ \1
server AWS.WEB.RTGRAF *:5582/ maxconn 1000 check port 5582
I need some help tracking down a sporadic http status 500 internal server error when using Graphite.
The server is running on Ubuntu 16.04, Graphite version 0.9.15.
10.0.0.10 - - [09/Jul/2017:08:07:14 -0500] "GET /render?from=-3minute&target=aliasByNode%28divideSeries%28hosts.Test.metric1.mean%2C+hosts.Test.metric2.mean%29%2C+1%29&format=json HTTP/1.1" 500 1237 "-" "python-requests/2.7.0 CPython/2.7.12 Linux/4.4.0-62-generic"
The same request works again without any modification.
I'm serving two websites through HAProxy and Varnish. There's a wiki site and a wordpress site. The wiki site works continuously and without problem. However the Wordpress site continuously shows a 504 error each time you reload the page.
If I spoof the wordpress site in my hosts file by using the IP of the varnish server instead of HAProxy the site comes back and starts working fine. It's only when wordpress is on haproxy that the site 504's.
I'd like to know how to turn on debug logging for HAProxy and also maybe get some help solving this problem.
This is all that I see in the logs for haproxy:
Apr 3 20:29:18 lb1.example.com haproxy[18501]: 52.21.231.226:52845 [03/Apr/2016:20:29:15.318] varnish-cluster varnish-cluster/varnish1 0/0/0/2786/2786 200 626 - - --NR 2/2/1/1/0 0/0 "HEAD / HTTP/1.1"
Apr 3 20:29:28 lb1.example.com haproxy[18501]: 61.174.10.22:18645 [03/Apr/2016:20:29:09.522] varnish-cluster varnish-cluster/varnish1 0/0/0/18206/19039 404 101736 - - --VN 0/0/0/0/0 0/0 "GET /groups/ HTTP/1.0"
Apr 3 20:29:34 lb1.example.com haproxy[18501]: 61.174.10.22:26372 [03/Apr/2016:20:29:31.045] varnish-cluster varnish-cluster/varnish1 0/0/0/3048/3048 301 549 - - --VN 0/0/0/0/0 0/0 "GET /members/pzwkathi09454/activity HTTP/1.0"
Apr 3 20:29:54 lb1.example.com haproxy[18501]: 61.174.10.22:27761 [03/Apr/2016:20:29:34.879] varnish-cluster varnish-cluster/varnish1 0/0/0/-1/20003 504 194 - - sHVN 0/0/0/0/0 0/0 "GET /activity/ HTTP/1.0"
And this is my config:
global
log 127.0.0.1 local2 debug
user root
group root
defaults
log global
retries 2
timeout connect 12000
timeout server 20000
timeout client 20000
listen varnish-cluster 0.0.0.0:80
mode http
stats enable
stats uri /haproxy?stats
stats realm Strictly\ Private
stats auth admin:secret
balance roundrobin
option http-server-close
timeout http-keep-alive 3000
option forwardfor
option httplog
cookie PHPSESSID prefix
server varnish1 xx.xx.xx.xx:80 cookie s1 check
listen mysql-master-cluster
bind 0.0.0.0:3306
mode tcp
option mysql-check user haproxy_check
balance roundrobin
server mysql-master-1 xx.xx.xx.xx:3306 check
server mysql-master-2 xx.xx.xx.xx:3306 check
I'd appreciate any advice you'd have in solving the 504 error with HAProxy!
I am using Nginx to handle hits of API. I checked the access log and found that sometimes Nginx is giving 400 error.
GET /url to hit/ HTTP/1.1" **400 172** "-" "-"
What is 172 in above log ? and how to solve this error in Nginx ?
172 corresponds to the size of server response in bytes.
Source: https://easyengine.io/tutorials/nginx/log-parsing/