An nginx/1.0.12 running as a proxy on Debian 6.0.1 starts throwing the following error after running for a short time:
connect() to upstreamip:80 failed (99: Cannot assign requested address)
while connecting to upstream, client: xxx.xxx.xxx.xxx, server: localhost,
request: "GET / HTTP/1.1", upstream: "http://upstreamip:80/",
host: "requesteddomain.com"
Not all requests produce this error, so I suspect that it has to do with the load of the server and some kind of limit it hit.
I have tried raising ulimit -n to 50k and worker_rlimit_nofile to 50k as well, but that does not seem to help. lsof -n shows a total of 1200 lines for nginx.
Is there a system limit on outgoing connections that might prevent nginx from opening more connections to its upstream server?
Seems like I just found the solution to my own question: Allocating more outgoing ports via
echo "10240 65535" > /proc/sys/net/ipv4/ip_local_port_range
solved the problem.
Each TCP connection has to have a unique quadruple source_ip:source_port:dest_ip:dest_port
source_ip is hard to change, source_port is chosen from ip_local_port_range but can't be more than 16 bits. The other thing left to adjust is dest_ip and/or dest_port. So add some IP aliases for your upstream server:
upstream foo {
server ip1:80;
server ip2:80;
server ip3:80;
}
Where ip1, ip2 and ip3 are different IP addresses for the same server.
Or it might be easier to have your upstream listen on more ports.
modify /etc/sysctl.conf:
net.ipv4.tcp_timestamps=1
net.ipv4.tcp_tw_recycle=0
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_max_tw_buckets=10000 #after done this: local ports decrease from 26000 to 6000(netstat -tuwanp | awk '{print $4}' | sort | uniq -c | wc -l)
run:
sysctl -p
Related
Environment
I have set up Proxy Protocol support on an AWS classic load balancer as shown here which redirects traffic to backend nginx (configured with ModSecurity) instances.
Everything works great and I can hit my websites from the open internet.
Now, since my nginx configuration is done in AWS User Data, I want to do some checks before the instance starts serving traffic which is achievable through AWS Lifecycle hooks.
Problem
Before enabling proxy protocol I used to check whether my nginx instance is healthy, and ModSecurity is working by checking a 403 response from this command
$ curl -ks "https://localhost/foo?username=1'%20or%20'1'%20=%20'"
After enabling Proxy Protocol, I can't do this anymore as the command fails with below error which is expected as per this link.
# curl -k https://localhost -v
* About to connect() to localhost port 443 (#0)
* Trying ::1...
* Connected to localhost (::1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* NSS error -5938 (PR_END_OF_FILE_ERROR)
* Encountered end of file
* Closing connection 0
curl: (35) Encountered end of file
# cat /var/logs/nginx/error.log
2017/10/26 07:53:08 [error] 45#45: *5348 broken header: "���4"�U�8ۭ�u��%d�z��mRN�[e��<�,�
�+̩� �0��/̨��98k�̪32g�5=�/<
" while reading PROXY protocol, client: 172.17.0.1, server: 0.0.0.0:443
What other options do I have to programmatically check nginx apart from curl? Maybe something in some other language?
You can use the --haproxy-protocol curl option, which adds the extra proxy protocol info to the request.
curl --haproxy-protocol localhost
So:
curl -ks "https://localhost/foo?username=1'%20or%20'1'%20=%20'"
Proxy Protocol append a plain text line before the streaming anything
PROXY TCP4 127.0.0.1 127.0.0.1 0 8080
Above is an example, but this happens the very first thing. Now if I have a NGINX listening on SSL and http both using proxy_protocol then it expects to see this line first and then any other thing
So if do
$ curl localhost:81
curl: (52) Empty reply from server
And in nginx logs
web_1 | 2017/10/27 06:35:15 [error] 5#5: *2 broken header: "GET / HTTP/1.1
If I do
$ printf "PROXY TCP4 127.0.0.1 127.0.0.1 0 80\r\nGET /test/abc\r\n\r\n" | nc localhost 81
You can reach API /test/abc and args_given = ,
It works. As I am able to send the proxy protocol it accepts
Now in case of SSL if I use below
printf "PROXY TCP4 127.0.0.1 127.0.0.1 0 8080\r\nGET /test/abc\r\n\r\n" | openssl s_client -connect localhost:8080
It would still error out
web_1 | 2017/10/27 06:37:27 [error] 5#5: *1 broken header: ",(�� #_5���_'���/��ߗ
That is because the client is trying to do Handshake first instead of sending proxy protocol first then handshake
So you possible solutions are
Terminate SSL on LB and then handle http on nginx with proxy_protocol and use the the nc command option I posted
Add a listen 127.0.0.1:<randomlargeport> and execute your test using the same. This is still safe as you are listening to localhost only
Add another SSL port and use listen 127.0.0.1:443 ssl and listen <private_ipv4>:443 ssl proxy_protocol
All solutions are in priority order as per my choice, you can make your own choice
Thanks Tarun for the detailed explanation. I discussed within the team and ended up doing creating another nginx virtual host on port 80 and using that to check ModSecurity as below.
curl "http://localhost/foo?username=1'%20or%20'1'%20=%20'"`
Unfortunately bash version didn't work in my case, so I wrote python3 code:
#!/usr/bin/env python3
import socket
import sys
def check_status(host, port):
'''Check app status, return True if ok'''
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.settimeout(3)
s.connect((host, port))
s.sendall(b'GET /status HTTP/1.1\r\nHost: api.example.com\r\nUser-Agent: curl7.0\r\nAccept: */*\r\n\r\n')
data = s.recv(1024)
if data.decode().endswith('OK'):
return True
else:
return False
try:
status = check_status('127.0.0.1', 80)
except:
status = False
if status:
sys.exit(0)
else:
sys.exit(1)
Is there any command which checks that a certain process is listening to a port.
I have tried check_tcp but it does not output which process is listening to a port
Its output was:
TCP OK - 0.000 second response time on port 8443|time=0.000421s;;;0.000000;10.000000
I didn't see anything on the Nagios Plugins Exchange to meet your needs, so I wrote one to be used with NRPE.
https://github.com/jlyoung/nagios_check_listening_port_linux
Output looks like this:
[root#joeyoung.io ~]# python /usr/lib/nagios/plugins/nagios_check_listening_port_linux.py -n nginx -p 80
OK. nginx found listening on port 80 for the following address(es): [0.0.0.0] | 'listening_on_expected_port'=1;;;;
[root#joeyoung.io ~]# python /usr/lib/nagios/plugins/nagios_check_listening_port_linux.py -n nginx -p 9999
CRITICAL - No process named nginx could be found listening on port 9999 | 'listening_on_expected_port'=0;;;;
Everything was running smoothly when suddenly my server stopped working.
I'm using Linode with Nginx fast-cgi
This is my log file:
upstream timed out (110: Connection timed out) while reading response header from upstream, client: 76.66.174.147, server: iskacanada.com, request: "GET / HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.iskacanada.com"
location ~ \.php$ {
include fastcgi_params;
fastcgi_read_timeout 120;
fastcgi_pass localhost:9000;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
When I want to restart mysql it says:
sudo service mysql restart
stop: Unknown instance:
start: Job failed to start
Any idea of what is going on?
After a few hours of Debugging here is how I did it:
Using Ubuntu 12.04, Nginx and php5-fmp
PLease check your log files! log files are your friends. a 504 Gateway problem means that my server is not communicating properly with the website. So In my case I had Nginx and php-fpm that was managing the requests. I had to check 2 log files:
/var/log/nginx/error.log and /var/log/php5-fpm.log
in error.log:
recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 76.66.174.147, server: xxxxxxx.com, request: "GET /wp-admin/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "www.xxxxxxx.com"
in php5-fpm.log:
unable to bind listening socket for address '127.0.0.1:9000': Address already in use (98)
So I figured out that I needed to check my php5-fpm process by typing
netstat | grep 9000
tcp 0 0 localhost.localdom:9000 localhost.localdo:58424 SYN_RECV
tcp 913 0 localhost.localdom:9000 localhost.localdo:57917 CLOSE_WAIT
tcp 857 0 localhost.localdom:9000 localhost.localdo:58032 CLOSE_WAIT
tcp 1633 0 localhost.localdom:9000 localhost.localdo:58395 CLOSE_WAIT
tcp 961 0 localhost.localdom:9000 localhost.localdo:58025 CLOSE_WAIT
tcp 857 0 localhost.localdom:9000 localhost.localdo:58040 CLOSE_WAIT
tcp 953 0 localhost.localdom:9000 localhost.localdo:58005 CLOSE_WAIT
tcp 761 0 localhost.localdom:9000 localhost.localdo:58016 CLOSE_WAIT
tcp 1137 0 localhost.localdom:9000 localhost.localdo:57960 CLOSE_WAIT
Lots of close_wait!!! that's abnormal...so I killed all the processes by typing
fuser -k 9000/tcp
I then changed my
/etc/php5/fpm/pool.d/www.conf
and changing this:
request_terminate_timeout=30s
Now the website works. I hope this solved the problem since it was intermittent.
Check if PHP is still running: sudo ps aux | grep php
If it is, restart it sudo service php5-fpm restart if not start it sudo service php5-fpm start.
If you need to restart your database only pass restart, stop or start to the service command: sudo service mysql start or sudo service mysql restart or sudo service mysql stop.
I just installed winginx and had the 504 gateway problem. The error log pointed at the upstream server "fastcgi://127.0.0.1:9000". This is where nginx proxies to php.
I opened the php-cgi.conf and found that php was listening on port 9054. Changed the port to 9000 and all is well.
Gateway is the route and/or port that nginx uses to connect to a service. For instance, Mongodb is configured to listen on port 27017 out of the box. For security reasons, I tend to change the default ports on services such as php etc. on production servers.
I've followed https://www.mercurial-scm.org/wiki/HgWebDirStepByStep to get "hg serve" running over CGI - but it's not quite working.
Here is the command I'm using to spawn the CGI:
spawn-fcgi -a 127.0.0.1 -p 9000 -f /path/to/hgweb.cgi -P /tmp/fcgi.pid 2>&1
The output suggests that the process spawned successfully, but a ps -p reveals that the process has already closed down. Sure enough, when I run the above command with -n, it spits out a load of HTML (the list of repositories) and then quits. Isn't it meant to stick around, listening on port 9000?
Telnetting to port 9000 gives "Connection refused" and this appears to be the problem nginx is having also:
2012/02/15 22:16:20 [error] 13483#0: *13 connect() failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: emily, request: "GET /hg/ HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "localhost:8001"
I'm confident my nginx config is correct, although I can post it here if you need to take a look.
Thanks for any help :)
Hi I'm trying to move my old dev environment to a new machine. However I keep getting "bad gateway errors" from nginx. From nginx's errorlog:
*19 kevent() reported that connect() failed (61: Connection refused) while connecting to upstream, client: 127.0.0.1, server: ~(?<app>[^.]+).gp2, request: "GET /favicon.ico HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "backend.gp2:5555"
Does anyone know why this is?
Thanks!
Turned out that PHP-fpm was not running
Looks like your upstream host at 127.0.0.1:9000 is not accepting connections. Is the upstream process working?
You seem to have nginx configured as a proxy, that tries to proxy its requests to localhost on port 9000, but cannot find anything listening on port 9000.
In my workstation, running php works for me. Take note that I'm using PHP 7.4 in my Mac. Pls adjust the PHP version according to what is installed in your workstation.
Working command:
sudo brew services start php#7.4
Please start your varnish:
sudo varnishd -a 127.0.0.1:80 -T 127.0.0.1:6082 -f /usr/local/etc/varnish/default.vcl -s file,/tmp,500M