How to convert Nginx.conf configuration to nginx-ingress-controller ingress.yml? - nginx

I have nginx.conf with following configuration.
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_connect_timeout 65;
proxy_send_timeout 65;
proxy_read_timeout 300;
proxy_buffers 4 256k;
proxy_buffer_size 128k;
proxy_busy_buffers_size 256k;
large_client_header_buffers 8 64k;
client_header_buffer_size 64k;
client_max_body_size 1024m;
server {
listen 443 ssl;
server_name server1.com;
ssl_certificate /etc/nginx/ssl/server1.com.crt;
ssl_certificate_key /etc/nginx/ssl/server1.com.key;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
proxy_ssl_session_reuse on;
proxy_ssl_protocols TLSv1.2;
rewrite_log on;
proxy_cache_bypass $http_cache_control;
proxy_ignore_headers "Set-Cookie";
add_header X-Proxy-Cache $upstream_cache_status;
location / {
location = /robots.txt {
add_header Content-Type text/plain;
return 200 "User-agent: *\nDisallow: /\n";
}
location ~ ^/(files/app) {
proxy_pass http://server1.com:31111;
}
proxy_pass https://server2.com;
}
}
I would like to shift my nginx config to kubernetes ingress-nginx ingress.yml. Is there a way to implement this config using kubernetes Ingress resources?
This is my ingress.yml which ofcourse isnt working and hitting the url https://server1.com/files/app goes into multiple redirections and throwing 502 Bad Gateway in the end.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
meta.helm.sh/release-name: my-service
meta.helm.sh/release-namespace: default
generation: 1
labels:
app.kubernetes.io/managed-by: Helm
name: my-service
namespace: default
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/my-service
spec:
rules:
- host: server1.com
http:
paths:
- backend:
serviceName: my-service
servicePort: 31111
path: /files/app
pathType: ImplementationSpecific

The solution was removing entries from httpd.conf. my service was directly bypassing the request but I had provided these entries in httpd conf as well due to which it was looping internally. As far as ingress rule is concerned. Above configuration worked fine after i removed related rules from httpd.conf.

Related

Artifactory 7.x behind nginx issue

I have a fresh installation of Artifactory 7.2.1(docker based)which is working fine, but I want to access it via nginx proxy, and that's not working.
my artifactory is running under http://192.168.211.207:8082/
Custom base URL is set to: http://192.168.211.207:8081/artifactory ->which is redirecting me to http://192.168.211.207:8082/
Now, I have an nginx server which is running on the same server, also via docker.
When I try to access:
http://192.168.211.207 -> redirects me to https://192.168.211.207/artifactory + 502 Bad Gateway
https://192.168.211.207 ->redirects me to https://192.168.211.207/ui + 502 Bad Gateway
http://192.168.211.207/artifactory -> redirects to https + 502 Bad Gateway
https://192.168.211.207/artifactory -> 502 Bad Gateway
I do not really understand what is behind port 8081 since I am not able to use it in any circumstances. The port 8082 is working, but not behind a nginx proxy.
Here is my docker-compose file:
version: '2'
services:
artifactory:
image: docker.bintray.io/jfrog/artifactory-pro:7.2.1
container_name: artifactory
ports:
- 8081:8081
- 8082:8082
volumes:
- /data/artifactory:/var/opt/jfrog/artifactory
restart: always
ulimits:
nproc: 65535
nofile:
soft: 32000
hard: 40000
nginx:
image: docker.bintray.io/jfrog/nginx-artifactory-pro:7.2.1
container_name: nginx
ports:
- 80:80
- 443:443
depends_on:
- artifactory
links:
- artifactory
volumes:
- /data/nginx:/var/opt/jfrog/nginx
environment:
- ART_BASE_URL=http://localhost:8081/artifactory
- SSL=true
# Set SKIP_AUTO_UPDATE_CONFIG=true to disable auto loading of NGINX conf
#- SKIP_AUTO_UPDATE_CONFIG=true
restart: always
ulimits:
nproc: 65535
nofile:
soft: 32000
hard: 40000
and here is my nginx config file:
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_certificate /var/opt/jfrog/nginx/ssl/example.crt;
ssl_certificate_key /var/opt/jfrog/nginx/ssl/example.key;
ssl_session_cache shared:SSL:1m;
ssl_prefer_server_ciphers on;
## server configuration
server {
listen 443 ssl;
listen 80 ;
server_name ~(?<repo>.+)\.artifactory artifactory;
if ($http_x_forwarded_proto = '') {
set $http_x_forwarded_proto $scheme;
}
## Application specific logs
## access_log /var/log/nginx/artifactory-access.log timing;
## error_log /var/log/nginx/artifactory-error.log;
if ( $repo != "" ){
rewrite ^/(v1|v2)/(.*) /artifactory/api/docker/$repo/$1/$2;
}
rewrite ^/$ /ui/ redirect;
rewrite ^/ui$ /ui/ redirect;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
proxy_busy_buffers_size 256k;
chunked_transfer_encoding on;
client_max_body_size 0;
location / {
proxy_read_timeout 2400s;
proxy_pass_header Server;
proxy_cookie_path ~*^/.* /;
proxy_pass http://localhost:8082;
proxy_set_header X-JFrog-Override-Base-Url $http_x_forwarded_proto://$host:$server_port;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
location ~ ^/artifactory/ {
proxy_pass http://localhost:8082;
}
}
}
I can't figure out what I am doing wrong here, but is possible to miss something since I am not an nginx expert.
Does someone spot the issue?
Does someone have an example config file for nginx and artifactory 7.x?
Thank you all for the answers. I have been able to get in touch with support, and after talking with a specialist they confirmed that in the version 7.x they don't support webcontext anymore, therefore in my case, the only way to run two artifactory was to create separate subdomains.
In order to be clear for future visitors of this topic, the jFrog Support confirmed me that starting with version 7.0 and newer, Artifactory does not support /webcontext feature anymore, and they don't plan to support it.
Therefore mydomain.com/artifactory-one and mydomain.com/artifactory-two is not anymore possible, you have to do it using subdomains.
mydomain.com/artifactory-one -> artifactory-one.mydomain.com
mydomain.com/artifactory-two -> artifactory-two.mydomain.com
Probably the issue is here. As you are running it in docker container nginx in container doesn't correctly process this - >proxy_pass http://localhost:8082;
Use IP instead. It worked for me
try this
location ~ ^/artifactory/ {
proxy_pass http://127.0.0.1:8081;
}
Here is my Nginx reverse proxy configuration with AWS NLB in front of the Nginx reverse proxy
ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
ssl_certificate /var/opt/jfrog/nginx/ssl/tls.crt;
ssl_certificate_key /var/opt/jfrog/nginx/ssl/tls.key;
ssl_session_cache shared:SSL:1m;
ssl_prefer_server_ciphers on;
## server configuration
server {
listen 443 ssl;
listen 80;
server_name ~(?<repo>.+)\.artifactory artifactory;
if ($http_x_forwarded_proto = '') {
set $http_x_forwarded_proto $scheme;
}
## Application specific logs
## access_log /var/log/nginx/artifactory-access.log timing;
## error_log /var/log/nginx/artifactory-error.log;
rewrite ^/$ /ui/ redirect;
rewrite ^/ui$ /ui/ redirect;
rewrite ^/artifactory/?$ / redirect;
if ( $repo != "" ) {
rewrite ^/(v1|v2)/(.*) /artifactory/api/docker/$repo/$1/$2 break;
}
chunked_transfer_encoding on;
client_max_body_size 0;
location / {
proxy_read_timeout 2400;
proxy_pass_header Server;
proxy_cookie_path ~*^/.* /;
proxy_buffer_size 128k;
proxy_buffers 40 128k;
proxy_busy_buffers_size 128k;
proxy_pass http://artifactory:8082/;
proxy_set_header X-JFrog-Override-Base-Url $http_x_forwarded_proto://$host;
proxy_set_header Host $http_host;
add_header Strict-Transport-Security always;
location /artifactory/ {
if ( $request_uri ~ ^/artifactory/(.*)$ ) {
proxy_pass http://artifactory:8081/artifactory/$1;
}
proxy_pass http://artifactory:8081/artifactory/;
}
}
}

Kibana with nginx ingress controller in Kubernetes

I am trying to get Kibana 6.2.4 in my GKE Kubernetes cluster running under www.mydomain.com/kibana without success. Though, I can run it perfectly fine with kubectl proxy and the default SERVER_BASEPATH.
Here is my Kibana deployment with the SERVER_BASEPATH removed.
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: kibana-logging
namespace: logging
labels:
k8s-app: kibana-logging
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
replicas: 1
selector:
matchLabels:
k8s-app: kibana-logging
template:
metadata:
labels:
k8s-app: kibana-logging
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
spec:
containers:
- name: kibana-logging
image: docker.elastic.co/kibana/kibana-oss:6.2.4
resources:
# need more cpu upon initialization, therefore burstable class
limits:
cpu: 1000m
requests:
cpu: 100m
env:
- name: ELASTICSEARCH_URL
value: http://elasticsearch-logging:9200
# - name: SERVER_BASEPATH
# value: /api/v1/namespaces/logging/services/kibana-logging/proxy
ports:
- containerPort: 5601
name: ui
protocol: TCP
My nginx ingress definition (nginx-ingress-controller:0.19.0):
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: logging-ingress
namespace: logging
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
rewrite ^/kibana/(.*)$ /$1 break;
spec:
tls:
- hosts:
- dev.mydomain.net
secretName: mydomain-net-tls-secret
rules:
- host: dev.mydomain.net
http:
paths:
- path: /kibana
backend:
serviceName: kibana-logging
servicePort: 5601
This results in this nginx location
location /kibana {
set $namespace "logging";
set $ingress_name "logging-ingress";
set $service_name "kibana-logging";
set $service_port "5601";
set $location_path "/kibana";
rewrite_by_lua_block {
balancer.rewrite()
}
log_by_lua_block {
balancer.log()
monitor.call()
}
port_in_redirect off;
set $proxy_upstream_name "logging-kibana-logging-5601";
# enforce ssl on server side
if ($redirect_to_https) {
return 308 https://$best_http_host$request_uri;
}
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering "off";
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout;
proxy_next_upstream_tries 3;
rewrite ^/kibana/(.*)$ /$1 break;
proxy_pass http://upstream_balancer;
proxy_redirect off;
}
However, going to /kibana results in a 404.
Stackdriver
2018-10-30 08:30:48.000 MDT
GET /kibana 404 61ms - 9.0B
Web page
{
statusCode: 404,
error: "Not Found",
message: "Not Found"
}
I feel as though I am missing some sort of setting with either SERVER_BASEPATH and/or my nginx ingress configuration.
I believe what you want is the nginx.ingress.kubernetes.io/rewrite-target: / annotation in your ingress.
This way the location {} block will look something like this:
location ~* ^/kibana\/?(?<baseuri>.*) {
...
rewrite (?i)/kibana/(.*) /$1 break;
rewrite (?i)/kibana$ / break;
...
}

kubernetes, port omitted when ingress-nginx redirects

I have a small cluster, and there are three external services. I use clusterIP as internal pod communication. Then I use ingress(nginx) as inverse proxy. The ingress connects internet to cluster.
When nginx redirects traffic, but did not sent port with domain. For example, ykt:31080/workflow redirects to ykt/workflow/login, it omits the port 31080. Then server could not find the page.
My ingress resource is configured as follows:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: my-ingress
spec:
rules:
- host: ykt
http:
paths:
- path: /business
backend:
serviceName: core-ykt
servicePort: 80
- host: ykt
http:
paths:
- path: /pre
backend:
serviceName: pre-core-ykt
servicePort: 80
- host: ykt
http:
paths:
- path: /workflow
backend:
serviceName: virtual-apply-ykt
servicePort: 80
And part of my ingress controller is configured as follows:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
spec:
type: NodePort
selector:
app: ingress-nginx
ports:
- name: http
port: 80
nodePort: 31080
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
name: ingress-nginx
spec:
replicas: 1
template:
metadata:
labels:
app: ingress-nginx
spec:
terminationGracePeriodSeconds: 60
serviceAccount: lb
containers:
- image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.15.0
name: ingress-nginx
imagePullPolicy: IfNotPresent
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/nginx-default-backend
================================================
Second edition
datalook_virtual_apply_pod.yaml file:
apiVersion: v1
kind: Pod
metadata:
name: virtual-apply-ykt
labels:
app: virtual-apply-ykt
purpose: ykt_production
spec:
containers:
- name: virtual-apply-ykt
image: app
imagePullPolicy: IfNotPresent
ports:
- containerPort: 80
volumeMounts:
- name: volume-virtual-apply-ykt
mountPath: /usr/application
env:
- name: spring.config.location
value: application.properties
volumes:
- name: volume-virtual-apply-ykt
hostPath:
path: /opt/docker/datalook-virtual-apply
type: Directory
datalook_virtual_apply_svc.yaml file is as follows, I have no deployment file for datalook_virtual_apply service.
apiVersion: v1
kind: Service
metadata:
name: virtual-apply-ykt
labels:
name: virtual-apply-ykt
spec:
selector:
app: virtual-apply-ykt
type: ClusterIP
ports:
- port: 80
name: tcp
/////////////////////////////
second edition
///////////////////////////////
The backend generated nginx configuation is as follows: instead let application listen to 80, May I make ingress-nginx default port as 31080?
daemon off;
worker_processes 1;
pid /run/nginx.pid;
worker_rlimit_nofile 1047552;
worker_shutdown_timeout 10s ;
events {
multi_accept on;
worker_connections 16384;
use epoll;
}
http {
lua_package_cpath "/usr/local/lib/lua/?.so;/usr/lib/x86_64-linux-gnu/lua/5.1/?.so;;";
lua_package_path "/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/?.lua;/usr/local/lib/lua/?.lua;;";
init_by_lua_block {
require("resty.core")
collectgarbage("collect")
local lua_resty_waf = require("resty.waf")
lua_resty_waf.init()
}
real_ip_header X-Forwarded-For;
real_ip_recursive on;
set_real_ip_from 0.0.0.0/0;
geoip_country /etc/nginx/geoip/GeoIP.dat;
geoip_city /etc/nginx/geoip/GeoLiteCity.dat;
geoip_org /etc/nginx/geoip/GeoIPASNum.dat;
geoip_proxy_recursive on;
aio threads;
aio_write on;
tcp_nopush on;
tcp_nodelay on;
log_subrequest on;
reset_timedout_connection on;
keepalive_timeout 75s;
keepalive_requests 100;
client_header_buffer_size 1k;
client_header_timeout 60s;
large_client_header_buffers 4 8k;
client_body_buffer_size 8k;
client_body_timeout 60s;
http2_max_field_size 4k;
http2_max_header_size 16k;
types_hash_max_size 2048;
server_names_hash_max_size 1024;
server_names_hash_bucket_size 32;
map_hash_bucket_size 64;
proxy_headers_hash_max_size 512;
proxy_headers_hash_bucket_size 64;
variables_hash_bucket_size 128;
variables_hash_max_size 2048;
underscores_in_headers off;
ignore_invalid_headers on;
limit_req_status 503;
include /etc/nginx/mime.types;
default_type text/html;
gzip on;
gzip_comp_level 5;
gzip_http_version 1.1;
gzip_min_length 256;
gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
gzip_proxied any;
gzip_vary on;
# Custom headers for response
server_tokens on;
# disable warnings
uninitialized_variable_warn off;
# Additional available variables:
# $namespace
# $ingress_name
# $service_name
log_format upstreaminfo '$the_real_ip - [$the_real_ip] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status $req_id';
map $request_uri $loggable {
default 1;
}
access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
error_log /var/log/nginx/error.log notice;
resolver 10.96.0.10 valid=30s;
# Retain the default nginx handling of requests without a "Connection" header
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
map $http_x_forwarded_for $the_real_ip {
default $remote_addr;
}
# trust http_x_forwarded_proto headers correctly indicate ssl offloading
map $http_x_forwarded_proto $pass_access_scheme {
default $http_x_forwarded_proto;
'' $scheme;
}
# validate $pass_access_scheme and $scheme are http to force a redirect
map "$scheme:$pass_access_scheme" $redirect_to_https {
default 0;
"http:http" 1;
"https:http" 1;
}
map $http_x_forwarded_port $pass_server_port {
default $http_x_forwarded_port;
'' $server_port;
}
map $pass_server_port $pass_port {
443 443;
default $pass_server_port;
}
# Obtain best http host
map $http_host $this_host {
default $http_host;
'' $host;
}
map $http_x_forwarded_host $best_http_host {
default $http_x_forwarded_host;
'' $this_host;
}
# Reverse proxies can detect if a client provides a X-Request-ID header, and pass it on to the backend server.
# If no such header is provided, it can provide a random value.
map $http_x_request_id $req_id {
default $http_x_request_id;
"" $request_id;
}
server_name_in_redirect off;
port_in_redirect off;
ssl_protocols TLSv1.2;
# turn on session caching to drastically improve performance
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_session_timeout 10m;
# allow configuring ssl session tickets
ssl_session_tickets on;
# slightly reduce the time-to-first-byte
ssl_buffer_size 4k;
# allow configuring custom ssl ciphers
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
ssl_prefer_server_ciphers on;
ssl_ecdh_curve auto;
proxy_ssl_session_reuse on;
upstream upstream-default-backend {
least_conn;
keepalive 32;
server 192.168.38.188:8080 max_fails=0 fail_timeout=0;
}
upstream default-gearbox-rack-api-gateway-5555 {
least_conn;
keepalive 32;
server 192.168.38.22:5555 max_fails=0 fail_timeout=0;
}
## start server _
server {
server_name _ ;
listen 80 default_server backlog=511;
listen [::]:80 default_server backlog=511;
set $proxy_upstream_name "-";
listen 443 default_server backlog=511 ssl http2;
listen [::]:443 default_server backlog=511 ssl http2;
# PEM sha: c1e1519ef05c8531e334ee947817a2ad495fe83a
ssl_certificate /ingress-controller/ssl/default-fake-certificate.pem;
ssl_certificate_key /ingress-controller/ssl/default-fake-certificate.pem;
location / {
log_by_lua_block {
}
if ($scheme = https) {
more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains";
}
access_log off;
port_in_redirect off;
set $proxy_upstream_name "upstream-default-backend";
set $namespace "";
set $ingress_name "";
set $service_name "";
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering "off";
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_next_upstream_tries 0;
proxy_pass http://upstream-default-backend;
proxy_redirect off;
}
# health checks in cloud providers require the use of port 80
location /healthz {
access_log off;
return 200;
}
# this is required to avoid error if nginx is being monitored
# with an external software (like sysdig)
location /nginx_status {
allow 127.0.0.1;
allow ::1;
deny all;
access_log off;
stub_status on;
}
}
## end server _
## start server master8g
server {
server_name master8g ;
listen 80;
listen [::]:80;
set $proxy_upstream_name "-";
location / {
log_by_lua_block {
}
port_in_redirect off;
set $proxy_upstream_name "default-gearbox-rack-api-gateway-5555";
set $namespace "default";
set $ingress_name "my-ingress";
set $service_name "gearbox-rack-api-gateway";
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Request-ID $req_id;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering "off";
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_next_upstream_tries 0;
proxy_pass http://default-gearbox-rack-api-gateway-5555;
proxy_redirect off;
}
}
## end server master8g
# default server, used for NGINX healthcheck and access to nginx stats
server {
# Use the port 18080 (random value just to avoid known ports) as default port for nginx.
# Changing this value requires a change in:
# https://github.com/kubernetes/ingress-nginx/blob/master/controllers/nginx/pkg/cmd/controller/nginx.go
listen 18080 default_server backlog=511;
listen [::]:18080 default_server backlog=511;
set $proxy_upstream_name "-";
location /healthz {
access_log off;
return 200;
}
location /is-dynamic-lb-initialized {
access_log off;
content_by_lua_block {
local configuration = require("configuration")
local backend_data = configuration.get_backends_data()
if not backend_data then
ngx.exit(ngx.HTTP_INTERNAL_SERVER_ERROR)
return
end
ngx.say("OK")
ngx.exit(ngx.HTTP_OK)
}
}
location /nginx_status {
set $proxy_upstream_name "internal";
access_log off;
stub_status on;
}
location / {
set $proxy_upstream_name "upstream-default-backend";
proxy_pass http://upstream-default-backend;
}
}
}
stream {
log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;
access_log /var/log/nginx/access.log log_stream;
error_log /var/log/nginx/error.log;
# TCP services
# UDP services
}

website not accessible after idling some time running nginx and kubernetes

My website is not accessible from the browser for a few minutes after idling or not accessing it like 30 minutes or more. I would have to reload the page for how many times to view the page and I am not sure which to debug.
the stack I am running is a Golang app behind nginx that runs on kubernetes ingress. here is part of my nginx.conf.
daemon off;
worker_processes 2;
pid /run/nginx.pid;
worker_rlimit_nofile 523264;
events {
multi_accept on;
worker_connections 16384;
use epoll;
}
http {
real_ip_header X-Forwarded-For;
set_real_ip_from 0.0.0.0/0;
real_ip_recursive on;
geoip_country /etc/nginx/GeoIP.dat;
geoip_city /etc/nginx/GeoLiteCity.dat;
geoip_proxy_recursive on;
# lua section to return proper error codes when custom pages are used
lua_package_path '.?.lua;./etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/lua-resty-http/lib/?.lua;';
init_by_lua_block {
require("error_page")
}
sendfile on;
aio threads;
tcp_nopush on;
tcp_nodelay on;
log_subrequest on;
reset_timedout_connection on;
keepalive_timeout 75s;
client_header_buffer_size 1k;
large_client_header_buffers 4 8k;
types_hash_max_size 2048;
server_names_hash_max_size 512;
server_names_hash_bucket_size 64;
map_hash_bucket_size 64;
include /etc/nginx/mime.types;
default_type text/html;
gzip on;
gzip_comp_level 5;
gzip_http_version 1.1;
gzip_min_length 256;
gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
gzip_proxied any;
server_tokens on;
log_format upstreaminfo '$remote_addr - '
'[$proxy_add_x_forwarded_for] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" '
'$request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';
map $request_uri $loggable {
default 1;
}
access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
error_log /var/log/nginx/error.log notice;
resolver 10.131.240.10 valid=30s;
# Retain the default nginx handling of requests without a "Connection" header
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
# trust http_x_forwarded_proto headers correctly indicate ssl offloading
map $http_x_forwarded_proto $pass_access_scheme {
default $http_x_forwarded_proto;
'' $scheme;
}
map $http_x_forwarded_port $pass_server_port {
default $http_x_forwarded_port;
'' $server_port;
}
# map port 442 to 443 for header X-Forwarded-Port
map $pass_server_port $pass_port {
442 443;
default $pass_server_port;
}
# Map a response error watching the header Content-Type
map $http_accept $httpAccept {
default html;
application/json json;
application/xml xml;
text/plain text;
}
map $httpAccept $httpReturnType {
default text/html;
json application/json;
xml application/xml;
text text/plain;
}
server_name_in_redirect off;
port_in_redirect off;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
# turn on session caching to drastically improve performance
ssl_session_cache builtin:1000 shared:SSL:10m;
ssl_session_timeout 10m;
# allow configuring ssl session tickets
ssl_session_tickets on;
# slightly reduce the time-to-first-byte
ssl_buffer_size 4k;
# allow configuring custom ssl ciphers
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_prefer_server_ciphers on;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
upstream default-ui-80 {
sticky hash=sha1 name=route httponly;
server 10.128.2.104:4000 max_fails=0 fail_timeout=0;
server 10.128.4.37:4000 max_fails=0 fail_timeout=0;
}
server {
server_name app.com;
listen [::]:80;
listen 442 ssl http2;
# PEM sha: a51bd3f56b3ec447945f1f92f0ad140bb8134d11
ssl_certificate /ingress-controller/ssl/default-linker-secret.pem;
ssl_certificate_key /ingress-controller/ssl/default-linker-secret.pem;
more_set_headers "Strict-Transport-Security: max-age=15724800; includeSubDomains; preload";
location / {
set $proxy_upstream_name "default-ui-80";
port_in_redirect off;
# enforce ssl on server side
if ($scheme = http) {
return 301 https://$host$request_uri;
}
client_max_body_size "1024m";
proxy_set_header Host $host;
# Pass Real IP
proxy_set_header X-Real-IP $remote_addr;
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers
proxy_connect_timeout 5s;
proxy_send_timeout 3600s;
proxy_read_timeout 3600s;
proxy_redirect off;
proxy_buffering off;
proxy_buffer_size "4k";
proxy_http_version 1.1;
proxy_pass http://default-ui-80;
}
}
}
ingress controller
apiVersion: v1
kind: ReplicationController
metadata:
name: nginx-ingress-controller
labels:
k8s-app: nginx-ingress-lb
spec:
replicas: 1
selector:
k8s-app: nginx-ingress-lb
template:
metadata:
labels:
k8s-app: nginx-ingress-lb
name: nginx-ingress-lb
spec:
terminationGracePeriodSeconds: 60
containers:
- image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.2
name: nginx-ingress-lb
imagePullPolicy: Always
readinessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
livenessProbe:
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
timeoutSeconds: 1
# use downward API
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- containerPort: 80
hostPort: 80
- containerPort: 443
hostPort: 443
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-ingress-sticky-session
- --configmap=$(POD_NAMESPACE)/nginx-settings-configmap
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-configmaps
- --v=2
ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: foo-prod
annotations:
kubernetes.io/tls-acme: "true"
kubernetes.io/ingress.class: "nginx"
ingress.kubernetes.io/affinity: "cookie"
ingress.kubernetes.io/session-cookie-name: "route"
ingress.kubernetes.io/session-cookie-hash: "sha1"
nginx.org/client-max-body-size: "1024m"
spec:
tls:
- hosts:
- foo.io
secretName: foo-secret
rules:
- host: foo.io
http:
paths:
- backend:
serviceName: foo.io
servicePort: 80
service
apiVersion: v1
kind: Service
metadata:
name: foo-prod-nginx
spec:
type: LoadBalancer
ports:
- port: 80
name: http
- port: 443
name: https
selector:
app: nginx-ingress-controller
The service.type=LoadBalancer allocates a public IP per k8s service, which is not how ingress works. You should expose your service as nodeport, and let ingress route traffic to it. example here
Also, if you are going to use nginx as ingress controller, you should use endpoint instead of service. Here is why

Artifactory bad gateway error

I am trying to use artifactory as a docker registry. But pushing docker images gives a Bad Gateway error.
Following is my nginx configuration
upstream artifactory_lb {
server artifactory01.mycomapany.com:8081;
server artifactory01.mycomapany.com:8081 backup;
server myLoadBalancer.mycompany.com:8081;
}
log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr: $request upstream_response_time $upstream_response_time msec $msec request_time $request_time';
server {
listen 80;
listen 443 ssl;
client_max_body_size 2048M;
location / {
proxy_set_header Host $host:$server_port;
proxy_pass http://artifactory_lb;
proxy_read_timeout 90;
}
access_log /var/log/nginx/access.log upstreamlog;
location /basic_status {
stub_status on;
allow all;
}
}
# Server configuration
server {
listen 2222 ssl default_server;
ssl_certificate /etc/nginx/ssl/self-signed/self.crt;
ssl_certificate_key /etc/nginx/ssl/self-signed/self.key;
ssl_protocols TLSv1.1 TLSv1.2;
ssl_ciphers 'EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH';
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:10m;
server_name myloadbalancer.mycompany.com;
if ($http_x_forwarded_proto = '') {
set $http_x_forwarded_proto $scheme;
}
rewrite ^/(v1|v2)/(.*) /api/docker/docker_repo/$1/$2;
client_max_body_size 0;
chunked_transfer_encoding on;
location / {
proxy_read_timeout 900;
proxy_pass_header Server;
proxy_cookie_path ~*^/.* /;
proxy_set_header X-Artifactory-Override-Base-Url $http_x_forwarded_proto://$host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for
proxy_pass http://myloadbalancer.company.com:8081/artifactory/;
}
}
The docker command I use to push images is
docker push myloadbalancer:2222/image_name
Nginx error logs show the following error 24084 connect() failed (111: Connection refused) while connecting to upstream, client: internal_ip, server: , request: "GET /artifactory/inhouse HTTP/1.0", upstream: "http:/internal_ip:8081/artifactory/repo"
What am I missing?
This can be fixed by changing the proxy pass to point to any of the upstream servers.
proxy_pass http://artifactory_lb;

Resources