Artifactory bad gateway error - nginx

I am trying to use artifactory as a docker registry. But pushing docker images gives a Bad Gateway error.
Following is my nginx configuration
upstream artifactory_lb {
server artifactory01.mycomapany.com:8081;
server artifactory01.mycomapany.com:8081 backup;
server myLoadBalancer.mycompany.com:8081;
}
log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr: $request upstream_response_time $upstream_response_time msec $msec request_time $request_time';
server {
listen 80;
listen 443 ssl;
client_max_body_size 2048M;
location / {
proxy_set_header Host $host:$server_port;
proxy_pass http://artifactory_lb;
proxy_read_timeout 90;
}
access_log /var/log/nginx/access.log upstreamlog;
location /basic_status {
stub_status on;
allow all;
}
}
# Server configuration
server {
listen 2222 ssl default_server;
ssl_certificate /etc/nginx/ssl/self-signed/self.crt;
ssl_certificate_key /etc/nginx/ssl/self-signed/self.key;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
server_name myloadbalancer.mycompany.com;
if ($http_x_forwarded_proto = '') {
set $http_x_forwarded_proto $scheme;
}
rewrite ^/(v1|v2)/(.*) /api/docker/docker_repo/$1/$2;
client_max_body_size 0;
chunked_transfer_encoding on;
location / {
proxy_read_timeout 900;
proxy_pass_header Server;
proxy_cookie_path ~*^/.* /;
proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for
proxy_pass http://myloadbalancer.company.com:8081/artifactory/;
}
}
The docker command I use to push images is
docker push myloadbalancer:2222/image_name
Nginx error logs show the following error 24084 connect() failed (111: Connection refused) while connecting to upstream, client: internal_ip, server: , request: "GET /artifactory/inhouse HTTP/1.0", upstream: "http:/internal_ip:8081/artifactory/repo"
What am I missing?

This can be fixed by changing the proxy pass to point to any of the upstream servers.
proxy_pass http://artifactory_lb;

Related

How to stop nginx pass through from resolving upstream to IP

I am new to nginx reverse proxy configuration. We have a requirement to setup a proxy server to route requests to a remote server that requires IP whitelisting. Two way SSL is also in place.
We have been trying to it up hit a roadblock. Following is the configuration :
server {
listen 80;
server_name myserver.com;
return 302 https://myserver.com;
}
server {
listen 443;
server_name myserver.com;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_certificate /etc/nginx/keys/my-net.crt;
ssl_certificate_key /etc/nginx/keys/my-net.key;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
location / {
resolver 8.8.8.8;
set $backend "https://remoteserver.com";
proxy_pass $backend;
proxy_ssl_server_name on;
proxy_ssl_trusted_certificate /home/ubuntu/myfile.pem;
proxy_ssl_session_reuse off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwared-For $proxy_add_x_forwarded_for;
}
}
The problem we are facing is that the proxy server is resolving the domain name to IP because of which SSLhandshake is failing. We need to stop this and hit the domain name because the certificate is on the domain name.
Error from error.log
2022/03/02 14:10:02 [error] 27012#27012: *8 SSL_do_handshake() failed (SSL: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:SSL alert number 40) while SSL handshaking to upstream, client: <>, server: <>, request: "POST <> HTTP/1.1", upstream: "https:<>", host: "<>"
2022/03/02 14:10:02 [alert] 27012#27012: *8 socket() failed (97: Address family not supported by protocol) while connecting to upstream, client: <>, server: <>, request: "POST /gateway/api/txb/v1/payments/transfer-payment HTTP/1.1", upstream: "https://[IPv6]:443/", host: "<>"
Already tried the solution here but no luck - How to stop nginx from resolving upstream to ip?
You need to off proxy_pass_request_headers after that set headers with proxy_set_heder and listen ssl listen 443 ssl;
server {
listen 80;
server_name myserver.com;
return 302 https://myserver.com;
}
server {
listen 443 ssl;
server_name myserver.com;
ssl on;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH";
ssl_certificate /etc/nginx/keys/my-net.crt;
ssl_certificate_key /etc/nginx/keys/my-net.key;
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
location / {
resolver 8.8.8.8;
set $backend "https://remoteserver.com";
proxy_pass $backend;
proxy_ssl_server_name on;
proxy_ssl_trusted_certificate /home/ubuntu/myfile.pem;
proxy_ssl_session_reuse off;
proxy_pass_request_headers off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwared-For $proxy_add_x_forwarded_for;
}
}

How do I fix this Nginx configuration to properly proxy WebSocket requests instead of returning a 301?

Nginx noob. Trying to configure Nginx to act as an SSL proxy server in front of another web server running at http://localhost:8082. That is, I want all requests to http://localhost to be redirected to https://localhost. That part is working just fine.
Problem is, the app on port 8082 also uses WebSocket connections at ws://localhost:8082/public-api/repossession-requests-socket. I'm trying to redirect any connections to ws://localhost/public-api/repossession-requests-socket to wss://localhost/public-api/repossession-requests-socket and have Nginx proxy those WebSocket requests to ws://localhost:8082/public-api/repossession-requests-socket.
Instead, the WebSocket connections are failing because Nginx is returning a 301 for both ws://localhost/public-api/repossession-requests-socket & wss://localhost/public-api/repossession-requests-socket. My configuration is below; I'm using the Docker image nginx:alpine in my tests ($PWD is mapped to /app).
How do I need to change this so that I no longer see 301s?
events {
worker_connections 1024;
}
http {
server {
listen 80;
return 301 https://$host$request_uri;
}
server {
listen 443;
server_name localhost;
ssl_certificate /app/docker/public.pem;
ssl_certificate_key /app/docker/private.pem;
ssl on;
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_ciphers HIGH:!aNULL:!eNULL:!EXPORT:!CAMELLIA:!DES:!MD5:!PSK:!RC4;
ssl_prefer_server_ciphers on;
access_log /app/access-443.log;
location / {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://localhost:8082;
proxy_read_timeout 90;
proxy_redirect http://localhost:8082 https://localhost;
}
location /public-api/repossession-requests-socket/ {
proxy_pass http://localhost:8082;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
}
}
Found the problem. The trailing slash on the end of the location stanza.
location /public-api/repossession-reqeuests-socket/ should have been location /public-api/repossession-reqeuests-socket.

SSL Pass-Through in Nginx Reverse proxy?

Is it possible to use Nginx reverse proxy with SSL Pass-through so that it can pass request to a server who require certificate authentication for client.
It means server will need to have certificate of client server and will not need certificate of Nginx reverse proxy server.
Not sure how much it can work in your situation, but newer (1.9.3+) versions of Nginx can pass (encrypted) TLS packets directly to an upstream server, using the stream block :
stream {
server {
listen 443;
proxy_pass backend.example.com:443;
}
}
If you want to target multiple upstream servers, distinguished by their hostnames, this is possible by using the nginx modules ngx_stream_ssl_preread and ngx_stream_map. The concept behind this is TLS Server Name Indication.
Dave T. outlines a solution nicely. See his answer on this network.
From the moment that we want to do ssl pass-through, the ssl termination will take place to the backend nginx server. Also i haven't seen an answer that takes care of the http connections as well.
The optimal solution will be a Nginx that is acting as a Layer 7 + Layer4 proxy at the same time. Something else that is rarely a subject of discussion is the IP Address redirection. When we use a proxy, this must be configured on the proxy, and not to the backend server like usually.
Lastly, the client ip address must be preserved, hence we must use the proxy protocol to do this correctly.
Sounds confusing? It's not much.
I came up with a solution that i currently using in production is works flawlessly.
worker_processes 1;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
variables_hash_bucket_size 1024;
variables_hash_max_size 1024;
map_hash_max_size 1024;
map_hash_bucket_size 512;
types_hash_bucket_size 512;
server_names_hash_bucket_size 512;
sendfile on;
tcp_nodelay on;
tcp_nopush on;
autoindex off;
server_tokens off;
keepalive_timeout 15;
client_max_body_size 100m;
upstream production_server {
server backend1:3080;
}
upstream staging_server {
server backend2:3080;
}
upstream ip_address {
server backend1:3080; #or backend2:3080 depending on your preference.
}
server {
server_name server1.tld;
listen 80;
listen [::]:80;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header Connection "";
#add_header X-Upstream $upstream_addr;
proxy_redirect off;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_buffers 16 16k;
proxy_buffer_size 64k;
proxy_cache_background_update on;
proxy_pass http://production_server$request_uri;
}
}
server {
server_name server2.tld;
listen 80;
listen [::]:80;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header Connection "";
#add_header X-Upstream $upstream_addr;
proxy_redirect off;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_buffers 16 16k;
proxy_buffer_size 16k;
proxy_cache_background_update on;
proxy_pass http://staging_server$request_uri;
}
}
server {
server_name 192.168.1.1; #replace with your own main ip address
listen 80;
listen [::]:80;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Host $server_name;
proxy_set_header Connection "";
#add_header X-Upstream $upstream_addr;
proxy_redirect off;
proxy_connect_timeout 300;
proxy_http_version 1.1;
proxy_buffers 16 16k;
proxy_buffer_size 16k;
proxy_cache_background_update on;
proxy_pass http://ip_address$request_uri;
}
}
}
stream {
map $ssl_preread_server_name $domain {
server1.tld production_server_https;
server2.tld staging_server_https;
192.168.1.1 ip_address_https;
default staging_server_https;
}
upstream production_server_https {
server backend1:3443;
}
upstream staging_server_https {
server backend2:3443;
}
upstream ip_address_https {
server backend1:3443;
}
server {
ssl_preread on;
proxy_protocol on;
tcp_nodelay on;
listen 443;
listen [::]:443;
proxy_pass $domain;
}
log_format proxy '$protocol $status $bytes_sent $bytes_received $session_time';
access_log /var/log/nginx/access.log proxy;
error_log /var/log/nginx/error.log debug;
}
Now the only thing is yet to be done is to enable proxy protocol to the backend servers. The example below will get you going:
server {
real_ip_header proxy_protocol;
set_real_ip_from proxy;
server_name www.server1.tld;
listen 3080;
listen 3443 ssl http2;
listen [::]:3080;
listen [::]:3443 ssl http2;
include ssl_config;
# Non-www redirect
return 301 https://server1.tld$request_uri;
}
server {
real_ip_header proxy_protocol;
set_real_ip_from 1.2.3.4; # <--- proxy ip address, or proxy container hostname for docker
server_name server1.tld;
listen 3443 ssl http2 proxy_protocol; #<--- proxy protocol to the listen directive
listen [::]:3443 ssl http2 proxy_protocol; # <--- proxy protocol to the listen directive
root /var/www/html;
charset UTF-8;
include ssl_config;
#access_log logs/host.access.log main;
location ~ /.well-known/acme-challenge {
allow all;
root /var/www/html;
default_type "text/plain";
}
location / {
index index.php;
try_files $uri $uri/ =404;
}
error_page 404 /404.php;
# place rest of the location stuff here
}
Now everything should work like a charm.

Configuring docker virtual repository in artifactory

I have a local docker repository and a remote docker repository , I created a virtual docker repository combining both. In order to access this repository from the client side, does this need to be added to the reverse proxy as well?
Here is the current reverse proxy configuration
upstream artifactory_lb {
server myserver.mycompany.com:8081 backup;
server myserver.mycompany.com:8081;
}
log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr: $request upstream_response_time $upstream_response_time msec $msec request_time $request_time';
ssl_certificate /etc/nginx/ssl/multidomain_cert_files/mycert.pem;
ssl_certificate_key /etc/nginx/ssl/multidomain_cert_files/mykey.key;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
ssl_session_cache shared:SSL:10m;
server {
listen 80;
listen 443 ssl;
client_max_body_size 2048M;
location / {
proxy_set_header Host $host;
proxy_pass http://artifactory_lb;
proxy_read_timeout 90;
}
access_log /var/log/nginx/access.log upstreamlog;
location /basic_status {
stub_status on;
allow all;
}
}
# Server configuration
server {
listen 2222 ssl;
if ($http_x_forwarded_proto = '') {
set $http_x_forwarded_proto $scheme;
}
rewrite ^/(v1|v2)/(.*) /api/docker/myrepo_images/$1/$2;
client_max_body_size 0;
chunked_transfer_encoding on;
location / {
allow all;s
proxy_read_timeout 900;
proxy_pass_header Server;
proxy_cookie_path ~*^/.* /;
proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host:$server_port;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass http://artifactory_lb/artifactory/;
}
}
Yes. Docker registries are referenced by their host name, only. This means that you'll need two virtual hosts in your reverse proxy with different hostnames (use the server_name directive for that), mapping to different Artifactory repositories.
The following example config (shortened) should do the trick:
server {
listen 2222 ssl;
server_name local-repo.my-artifactory.com;
rewrite ^/(v1|v2)/(.*) /api/docker/myrepo_images/$1/$2;
# <insert remaining configuration directives here>
}
server {
listen 2222 ssl;
server_name virtual-repo.my-artifactory.com;
rewrite ^/(v1|v2)/(.*) /api/docker/myrepo_virtual/$1/$2;
# <insert remaining configuration directives here>
}
Now you should be able to access both registries using the regular docker commands:
$ docker pull virtual-repo.my-artifactory.com:2222/foo/bar:latest
$ docker pull local-repo.my-artifactory.com:2222/foo/bar:latest
$ docker push local-repo.my-artifactory.com:2222/foo/bar:latest

Unable to push docker images to artifactory

I set up artifactory as a docker registry and am trying to push an image to it
docker push nginxLoadBalancer.mycompany.com/repo_name:image_name
This fails with the following error
The push refers to a repository [ nginxLoadBalancer.mycompany.com/repo_name] (len: 1)
unable to ping registry endpoint https://nginxLoadBalancer.mycompany.com/v0/
v2 ping attempt failed with error: Get https://nginxLoadBalancer.mycompany.com/v2/: Bad Request
v1 ping attempt failed with error: Get https://nginxLoadBalancer.mycompany.com/v1/_ping: Bad Request
This is my nginx conf
upstream artifactory_lb {
server mNginxLb.mycompany.com:8081;
server mNginxLb.mycompany.com backup;
}
log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr: $request upstream_response_time $upstream_response_time msec $msec request_time $request_time';
server {
listen 80;
listen 443 ssl;
ssl_certificate /etc/nginx/ssl/my-certs/myCert.pem;
ssl_certificate_key /etc/nginx/ssl/my-certs/myserver.key;
client_max_body_size 2048M;
location / {
proxy_set_header Host $host:$server_port;
proxy_pass http://artifactory_lb;
proxy_read_timeout 90;
}
access_log /var/log/nginx/access.log upstreamlog;
location /basic_status {
stub_status on;
allow all;
}
}
# Server configuration
server {
listen 2222 ssl;
server_name mNginxLb.mycompany.com;
if ($http_x_forwarded_proto = '') {
set $http_x_forwarded_proto $scheme;
}
rewrite ^/(v1|v2)/(.*) /api/docker/my_local_repo_key/$1/$2;
client_max_body_size 0;
chunked_transfer_encoding on;
location / {
proxy_read_timeout 900;
proxy_pass_header Server;
proxy_cookie_path ~*^/.* /;
proxy_pass http://artifactory_lb;
proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
There are no errors in the nginx error log. What might be wrong?
I verfied that the SSL verification works fine with the set up. Do I need to set up authentication before I push images?
I also verified artifactory server is listening on port 2222
Update,
I added the following to the nginx configuration
location /v1 {
proxy_pass http://myNginxLb.company.com:8080/artifactory/api/docker/docker-local/v1;
}
With this it now gives a 405 - Not allowed error when trying to push to the repository
I fixed this by removing the location /v1 configuration and also changing proxy pass to point to the upstream servers

Resources