i'm having a connection timed out issue with nginx reverse proxy since yesterday when we updated docker to version 1.13
here is the error log from Nginx
upstream timed out (110: Connection timed out) while connecting to upstream, client: 10.255.0.3, server: 10.1.232.220, request: "GET /Microservice/microservices HTTP/1.1", upstream: "http://10.255.0.10:8765/Microservice/microservices", host: "10.1.232.220"
The nginx is a service inside a docker swarm where also is the backend that i'm trying to call, i make a ping from the nginx container to the ip showing in the error log and there is no problem, any ideas on this issue?
Related
I'm running an Nginx in a Cloud Run instance as a Reverse-Proxy to a backend Cloud Run app (it will do more stuff in the future, but let's keep it simple for this example).
The Nginx Cloud Run requires authentication (via IAM), but the backend app doesn't. The Nginx is connected to the same VPC and has the setting (vpc_access_egress = all-traffic), and the backend app is set to Allow internal traffic only only.
events {}
http {
resolver 169.254.169.254;
server {
listen 8080;
server_name mirror_proxy;
location / {
proxy_pass https://my-backend.a.run.app:443;
proxy_cache off;
proxy_redirect off;
}
}
}
The setup works, and I send authenticated requests to the Nginx and get the responses from the backend. However I also get a lot of error messages from the Nginx per request.
2022/12/22 13:57:51 POST 200 1.76 KiB 1.151s curl 7.68.0 https://nginx.a.run.app/main
2022/12/22 13:57:50 [error] 4#4: *21 connect() to [1234:5678:4802:34::35]:443 failed
(113: No route to host) while connecting to upstream, client: 169.254.1.1,
server: mirror_proxy, request: "POST /main HTTP/1.1",
upstream: "https://[1234:5678:4802:34::35]:443/main", host: "nginx.a.run.app"
2022/12/22 13:57:50 [error] 4#4: *21 connect() to [1234:5678:4802:36::35]:443 failed
(113: No route to host) while connecting to upstream, client: 169.254.1.1,
server: mirror_proxy, request: "POST /main HTTP/1.1",
upstream: "https://[1234:5678:4802:36::35]:443/main", host: "nginx.a.run.app"
2022/12/22 13:57:50 [error] 4#4: *21 connect() to [1234:5678:4802:32::35]:443 failed
(113: No route to host) while connecting to upstream, client: 169.254.1.1,
server: mirror_proxy, request: "POST /main HTTP/1.1",
upstream: "https://[1234:5678:4802:32::35]:443/main", host: "nginx.a.run.app"
Why are there errors, when the request succeeds?
Doesn't the VPC router don't know the exact IP address of the Cloud Run yet, and Nginx has to try them out? Any idea?
GCP only uses IPv4 inside the VPC network.
Since I forced the Nginx to use the VPC network (vpc_access_egress = all-traffic), Nginx will fail when it tries to resolve an IPv6, and fall back to IPv4.
With the following setting you can force Nginx to immediately resolve the IPv4.
http {
resolver 169.254.169.254 ipv6=off;
...
}
``
I am struggling on this problem for 2-3 days now. My problem is : I get "SSL_do_handshake() failed" when doing proxy_pass from one reverse proxy to another.
I have a setup that looks like that :
gcp VM 1 containers :
- nginx reverse proxy 1
- acme companion for ssl
- frontend website (local nginx)
gcp VM 2 containers :
- nginx reverse proxy 2
- acme companion for ssl
- backend nodejs
DNS server is done by cloudflare :
- frontend.website.com : "gcp VM 1" IP adress
- backend.nodejs.com : "gcp VM 2" IP adress
To avoid CORS error, "frontend.website.com" make requests to "frontend.website.com/api".
"nginx reverse proxy 1" has this configuration to redirect to backend :
location /api {
proxy_pass https://backend.nodejs.com/api;
}
The error I get in the "nginx reverse proxy 1" logs :
nginx.1 | 2021/10/22 11:10:53 [error] 283#283: *11287 SSL_do_handshake() failed (SSL: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream, client: 2a01:e0a:4d0:4960:dc2e:8d3a:ba04:10a2, server: frontend.website.com, request: "POST /api HTTP/2.0", upstream: "https://172.67.155.25:443/api", host: "frontend.website.com", referrer: "https://frontend.website.com/"
nginx.1 | 2021/10/22 11:10:53 [warn] 283#283: *11287 upstream server temporarily disabled while SSL handshaking to upstream, client: 2a01:e0a:4d0:4960:dc2e:8d3a:ba04:10a2, server: frontend.website.com, request: "POST /api HTTP/2.0", upstream: "https://172.67.155.25:443/api", host: "frontend.website.com", referrer: "https://frontend.website.com/"
nginx.1 | 2021/10/22 11:10:53 [error] 283#283: *11287 no live upstreams while connecting to upstream, client: 2a01:e0a:4d0:4960:dc2e:8d3a:ba04:10a2, server: frontend.website.com, request: "POST /api HTTP/2.0", upstream: "https://backend.nodejs.com/api", host: "frontend.website.com", referrer: "https://frontend.website.com/"
Note : IP in error log 172.67.155.25:443 is not gcp VM 1 or 2 IP, I assume it's a Cloudflare IP?
Things that I already tried :
check SSL certs, they are okay on both sides.
proxy_pass to http instead of https, its raises other problems.
I already tried proxy_ssl_server_name on; (taken from here). Cloudflare returned a 403 forbidden with :
DNS points to prohibited IP
What happened?
You've requested a page on a website (frontend.website.com) that is on the Cloudflare network. Unfortunately, it is resolving to an IP address that is creating a conflict within Cloudflare's system.
What can I do?
If you are the owner of this website:
you should login to Cloudflare and change the DNS A records for frontend.website.com to resolve to a different IP address.
The thing is the DNS A records for frontend.website.com is good (other app are using it without problem)
I feel like there shoud be ssl handshake to "backend.nodejs.com/api" but according to the error log, it tries to do it to the cloudflare IP address instead (here 172.67.155.25:443 but different each time)
Am I missing something here ? What can the problem be ?
If you need any additionnal info do not hesitate to ask for it.
I want to set up nginx as a forward proxy - much like Squid might work.
This is my server block:
server {
listen 3128;
server_name localhost;
location / {
resolver 8.8.8.8;
proxy_pass http://$http_host$uri$is_args$args;
}
}
This is the curl command I use to test, and it works the first time, maybe even the second time.
curl -s -D - -o /dev/null -x "http://localhost:3128" http://storage.googleapis.com/my.appspot.com/test.jpeg
The corresponding nginx access log is
172.23.0.1 - - [26/Feb/2021:12:38:59 +0000] "GET http://storage.googleapis.com/my.appspot.com/test.jpeg HTTP/1.1" 200 2296040 "-" "curl/7.64.1" "-"
However - on repeated requests, I start getting these errors in my nginx logs (after say the 2nd or 3rd attempt)
2021/02/26 12:39:49 [crit] 31#31: *4 connect() to [2c0f:fb50:4002:804::2010]:80 failed (99: Address not available) while connecting to upstream, client: 172.23.0.1, server: localhost, request: "GET http://storage.googleapis.com/omgimg.appspot.com/test.jpeg HTTP/1.1", upstream: "http://[2c0f:fb50:4002:804::2010]:80/my.appspot.com/test.jpeg", host: "storage.googleapis.com"
2021/02/26 12:39:49 [warn] 31#31: *4 upstream server temporarily disabled while connecting to upstream, client: 172.23.0.1, server: localhost, request: "GET http://storage.googleapis.com/my.appspot.com/test.jpeg HTTP/1.1", upstream: "http://[2c0f:fb50:4002:804::2010]:80/my.appspot.com/test.jpeg", host: "storage.googleapis.com"
What might be causing these issues after just a handful of requests? (curl still fetches the URL fine)
The DNS resolver was resolving to both IPV4 and IPV6 addresses. The IPV6 part seems to be causing an issue with the upstream servers.
Switching it off made those errors disappear.
resolver 8.8.8.8 ipv6=off;
I'm facing a problem with one server mine. I have a configuration of a Jenkins application running under Nginx in a Ubuntu OS.
But after updating Jenkins to a new version, when I try to reach my Jenkins web application, I get this error:
"502 Bad Gateway - nginx/1.14.0 (Ubuntu)"
Apparently all my configuration are okay.
How can I solve it?
Nginx error log:
2019/04/24 16:20:02 [error] 11424#11424: *2 connect() failed (111: Connection refused) while connecting to upstream, client: yyy.yy.yyy.yy, server: xx.xx.xxx.xxx, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8080/favicon.ico", host: "xx.xx.xxx.xxx", referrer: "http://xx.xx.xxx.xxx/"
I've got big troubles with deploy my nuxt.js project on Ubuntu with Nginx;
Ubuntu 18.04 + nginx
Nuxt.js 2.3.2 (universal mode)
So maybe somebody can help me to resolve my problem)
I will to describe all things around this problem below:
My nuxt.conf build part
My nginx config
Then I run this command in ssh
sudo npm run build
sudo pm2 start npm --name "nuxt" -- start
Now server is listening localhost:3000, as I wrote in package.json
After all this I've got 500 error in console, when I open my site.
Server error
nginx error.log
2018/11/26 04:48:42 [error] 15032#15032: *5773 upstream prematurely closed connection while reading response header from upstream, client: 173.267.244.9, server: example.com, request: "GET /seller HTTP/1.1", upstream: "http://127.0.0.1:3000/seller", host: "example.com"
2018/11/26 04:48:42 [error] 15032#15032: *5773 connect() failed (111: Connection refused) while connecting to upstream, client: 173.267.244.9, server: example.com, request: "GET /seller HTTP/1.1", upstream: "http://[::1]:3000/seller", host: "example.com"
2018/11/26 04:48:43 [error] 15032#15032: *5773 no live upstreams while connecting to upstream, client: 173.267.244.9, server: example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://localhost/favicon.ico", host: "example.com", referrer: "http://example.com/seller"
So now I have no idea whats wrong with that. Help me please.
It seems that Nginx tries to connect on the IPv6 loopback [::1], and nodejs is probably just listening on IPv4.
Try setting 127.0.0.1 instead of localhost