Dear friendly developer,
I am trying to register a Gitlab Kubernetes Agent inside a Minikube with a self hosted Gitlab instance. The Gitlab instance is a dockerized Omnibus installation. It does not have any exposed ports. Instead I chose to use a nginx within the same docker network to proxy_pass requests to Gitlab.
When I deploy the agent to the cluster and the container is running, it logs theses errors:
{"level":"warn","time":"2022-02-26T00:12:59.647Z","msg":"GetConfiguration.Recv failed","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01FWSNZ31HRVTAAD5J5700BBXH"}
{"level":"error","time":"2022-02-26T00:13:28.271Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"rpc error: code = Unauthenticated desc = unauthenticated","correlation_id":"01FWSP040J2CRGF5WFHMEX1ACC"}
Visiting http://gitlab.local/api/v4/internal/kubernetes/agent_info results in
{
"message": "KAS JWT authentication invalid"
}
The agent successfully connects to Gitlab when I expose the gitlab ports directly to localhost (and change the agent's kubernetes config accordingly). That is why I am quite sure that it has to be a problem with my nginx websocket configuration.
I have triple checked that the token inside the kubernetes secret for the agent matches the base64 registration token generated by Gitlab.
This is an excerpt of my docker-compose file for gitlab:
services:
gitlab:
image: gitlab/gitlab-ee:latest
container_name: gitlab
restart: always
hostname: gitlab.local
networks:
- ci-cd
environment:
GITLAB_OMNIBUS_CONFIG: |
external_url 'http://gitlab.local'
registry_external_url 'http://gitlab.local:5050'
registry['enable'] = true
registry['env'] = {
"REGISTRY_HTTP_RELATIVEURLS" => true
}
gitlab_kas['enable'] = true
gitlab_kas['gitlab_address'] = 'http://gitlab.local'
volumes:
- $GITLAB_HOME/etc:/etc/gitlab:rw
- $GITLAB_HOME/opt:/var/opt/gitlab:rw
- $GITLAB_HOME/log:/var/log/gitlab:rw
shm_size: "512m"
ulimits:
sigpending: 62793
nproc: 131072
nofile: 60000
core: 0
sysctls:
net.core.somaxconn: 1024
The default API path that gitlab uses for the agent websocket connection is:
/-/kubernetes-agent/
This is my nginx configuration:
upstream gitlab_container {
server gitlab;
}
upstream gitlab_registry_container {
server gitlab:5050;
}
map $http_upgrade $connection_upgrade {
default upgrade;
`` close;
}
server {
listen 80;
listen [::]:80;
server_name gitlab.local;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Host $host;
proxy_pass http://gitlab_container;
proxy_ssl_session_reuse off;
proxy_redirect off;
proxy_cache_bypass $http_upgrade;
}
location /-/kubernetes-agent/ {
proxy_pass http://gitlab;
proxy_http_version 1.1;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Host $host;
proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_cache_bypass $http_upgrade;
}
}
server {
listen 5050;
listen [::]:5050;
server_name gitlab.local;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-NginX-Proxy true;
proxy_set_header Host $host;
proxy_pass http://gitlab_registry_container;
proxy_redirect off;
proxy_ssl_session_reuse off;
proxy_cache_bypass $http_upgrade;
}
}
This is the kubernetes configuration for my agent:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: gitlab-agent
namespace: gitlab-kubernetes-agent
spec:
replicas: 1
selector:
matchLabels:
app: gitlab-agent
strategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
annotations:
prometheus.io/path: /metrics
prometheus.io/port: "8080"
prometheus.io/scrape: "true"
labels:
app: gitlab-agent
spec:
hostAliases:
- ip: ${INTERNAL_HOST_IP}
hostnames:
- "gitlab.local"
containers:
- args:
- --token-file=/config/token
- --kas-address
- ws://gitlab.local/-/kubernetes-agent/
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
image: registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:stable
livenessProbe:
httpGet:
path: /liveness
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
name: agent
readinessProbe:
httpGet:
path: /readiness
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
volumeMounts:
- mountPath: /config
name: token-volume
serviceAccountName: gitlab-agent
volumes:
- name: token-volume
secret:
secretName: ${GITLAB_AGENT_TOKEN_NAME}
The handshake and the protocol upgrade seems to be working fine, as my nginx log shows
172.19.0.1 - - [26/Feb/2022:00:29:32 +0000] "GET /-/kubernetes-agent/ HTTP/1.1" 101 3450 "-" "gitlab-agent/v14.8.1/86d5bf7" "-"
I guess that somehow the registration token gets lost when passing through the reverse proxy. Sadly, I cannot find any technical documentation on how the authentication works in detail.
Any clue as to what I am missing is highly appreciated!
I had exactly same error although probably had a different reason of it. My new gitlab server, by my error, was using an alternate DNS which had an entry for gitlab's external URL poining to my OLD gitlab server.
Everything else in the network (kubernetes with agent I was trying to install) were using correct DNS and effectively were pointing to the correct new gitlab server, but the gitlab server itself wasn't. I've discovered it by looking at the tcpdump and noticing the traffic on 443 between my old and new gitlab servers (so advice is to trace the https traffic on your gitlab server). Took me 2 days :( These messages should be a bit more elaborative (if they could give IPs and ports for this connection I would figure out my error in 2 minutes).
Hope this helps next people with similar problem to pin point the issue.
Another option:
See GitLab 14.10 (April 2022)
The agent server for Kubernetes enabled by default in the Helm chart
The first step for using the agent for Kubernetes in self-managed instances is
to enable the agent server, a backend service for the agent for Kubernetes.
In GitLab 14.8 we enabled the agent server for Omnibus based installations.
The feature has matured in the past few months, so we are now making the agent server enabled by default in the GitLab Helm chart as well to simplify setup for GitLab administrators.
Besides being enabled by default, the agent server accepts various configuration options to customize it according to your needs.
See Documentation and Issue.
That might be easier than setting it up through NGiNX.
This is confirmed with See GitLab 15.1 (June 2022)
Agent server for Kubernetes enabled by default in the Helm chart
The first step for using the agent for Kubernetes in self-managed instances is to enable the agent server, a backend service for the agent for Kubernetes.
In GitLab 14.8, we enabled the agent server for Omnibus based installations.
The feature has matured in the past few months, so we are now making the agent server enabled by default in the GitLab Helm chart as well to simplify setup for GitLab administrators.
Besides being enabled by default, the agent server accepts various configuration options to customize it according to your needs.
See Documentation and Issue.
See GitLab 15.1 (June 2022)
GitLab agent for Kubernetes supports proxied connections
Many users require a proxy to connect Kubernetes clusters to GitLab. Previously, the default installation method for the GitLab agent for Kubernetes did not support proxied connections.
Now, you can use the HTTP_PROXY environment variable in the agentk Helm package to support proxied connections.
See Documentation and Issue.
I don't know exactly what I did different then the previous 10 times, but suddenly the agent connected successfully with the configuration shown above. I suppose it was any of these lines inside my nginx configuration for gitlab:
proxy_set_header Sec-WebSocket-Protocol $http_sec_websocket_protocol;
proxy_set_header Sec-WebSocket-Extensions $http_sec_websocket_extensions;
proxy_set_header Sec-WebSocket-Key $http_sec_websocket_key;
proxy_set_header Sec-WebSocket-Version $http_sec_websocket_version;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
Those I added last. But I cannot guarantee that was the breaking change. Good luck to everyone with similar issues reading this post.
Related
Express-Gateway is unable to bind to localhost or 127.0.0.1
Calling endpoints directly works as expected:
curl http://localhost:5000/ip
curl http://localhost:5010/erp
Accessing all endpoints via the ExpressGateway on port 5000 works as expected
curl http://localhost:5000/ip
curl http://localhost:5000/api/erp
The issue
The nginx reverse proxy works normally but returns a failed response when accessing the gateway
Cannot GET /api/erp
Binding host: localhost for the http in gateway.config.yml has no effect whatsoever.
Even when I change the host to another IP Address and port, the port reflects the change but the IP address of the host remains unchanged as [:::5000] in the express-gateway console.
Please, how can I resolve this?
gateway.config.yml
http:
port: 5000
admin:
port: 9876
host: localhost
apiEndpoints:
api:
host: localhost
paths: '/ip'
erp:
host: localhost
paths: ['/api/erp', '/api/erp/*']
serviceEndpoints:
httpbin:
url: 'https://httpbin.org'
erpService:
url: 'http://localhost:5010'
policies:
- basic-auth
- cors
- expression
- key-auth
- log
- oauth2
- proxy
- rate-limit
pipelines:
default:
apiEndpoints:
- api
policies:
# Uncomment `key-auth:` when instructed to in the Getting Started guide.
# - key-auth:
- proxy:
- action:
serviceEndpoint: httpbin
changeOrigin: true
erpPipeline:
apiEndpoints:
- erp
policies:
# Uncomment `key-auth:` when instructed to in the Getting Started guide.
# - key-auth:
- proxy:
- action:
serviceEndpoint: erpService
changeOrigin: true
The Reverse proxy with Nginx
server {
listen 82;
location / {
proxy_pass http://localhost:5010;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
server {
listen 81;
location / {
proxy_pass http://localhost:5000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
You need to specify the hostname to gateway.config.* file
"http": {
"hostname": "0.0.0.0",
"port": 1234
},
The host name could be localhost as well.
For express-gateway, If the hostname is not specified, the server IP address will be used as the default hostname.
Example: Your server IP address: 123.456.789.12
The following log will show up when you start the gateway
gateway http server listening on 123.456.789.12:5000
That's why nginx can't call to localhost:5000
When the hostname specified, the log should be:
info: gateway http server listening on 0.0.0.0:5000
Change localhost to your local ip in this part:
erpService:
url: 'http://localhost:5010'
Change to example:
erpService:
url: 'http://192.168.0.3:5010'
And change in your nginx config port 82 to 5010
server {
listen 5010;
location / {
proxy_pass http://localhost:5010;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
This config works for me.
You can change the localhost.
This solution works for me:
http:
port: 8080
admin:
port: 9876
host: localhost
apiEndpoints:
api:
host: "gateway.example.com" =>>main domain for gateway
paths: "/ip"
panel:
host: "gateway.example.com" =>>main domain for gateway
paths: "/panel/api/v1/*"
account:
host: "gateway.example.com" =>>main domain for gateway
paths: "/account/api/v1/*"
serviceEndpoints:
httpbin:
url: "https://httpbin.org"
panelService:
urls:
- "https://panel-api.example.com" =>> panel domain
accountService:
urls:
- "https://account-api.example.com" =>> account domain
I am trying to run a service called Grafana behind Nginx webserver,where both services are being run in a docker-compose file.
docker-compose.yml:
version: '3.1'
services:
nginx:
image: nginx
ports: ['443:443',"80:80"]
restart: always
volumes:
- ./etc/nginx.conf:/etc/nginx/nginx.conf:ro
- /home/ec2-user/certs:/etc/ssl
grafana:
image: grafana/grafana
restart: always
ports: ["3000:3000"]
nginx.conf:
events {
worker_connections 1024;
}
http {
ssl_certificate /etc/ssl/cert.pem;
ssl_certificate_key /etc/ssl/key.pem;
server {
listen 443 ssl;
server_tokens off;
location /grafana/ {
rewrite /grafana/(.*) /$1 break;
proxy_pass http://127.0.0.1:3000/;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_bind $server_addr;
}
}
}
The grafana service is running on port 3000.
My goal is to access this nginx server from outside, (lets assume its public ip address is: 1.1.1.1) on the address https://1.1.1.1/grafana. With the current configuration i get 502 Bad Gateway and the error on nginx side:
(111: Connection refused) while connecting to upstream, client: <<my-public-ip-here>>,
Your containers are running on two separate IP addresses in the docker network, usually 172.17.. by default.
By using a proxy pass like this in the nginx container:
proxy_pass http://127.0.0.1:3000/
You are essentially telling it to look for a process on port 3000 local to itself, because of the 127.0.0.1 right?
You need to point it in the direction of the Grafana container, try doing:
docker inspect <grafana ID> | grep IPAddress
Then set the proxy pass to that IP:
proxy_pass http://172.0.0.?:3000/
I've solved the same issue using something like #james suggested:
docker inspect <your inaccessible container is> | grep Gateway
Then use this IP address:
proxy_pass http://172.xx.0.1:3000/
I would like to setup an Nginx reverse proxy, which works fine, but if I set network_mode: "host" it stops working because it is unable to find the hostname of other docker containers. I have a web container and an nginx container.
I get the following error:
reverseproxy_1 | nginx: [emerg] host not found in upstream "web:80" in /etc/nginx/nginx.conf:10
My Nginx conf file is:
worker_processes 1;
events { worker_connections 1024; }
http {
sendfile on;
upstream docker-web {
server web:80;
}
server {
listen 8080;
location / {
proxy_pass http://docker-web;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
}
}
and my docker-compose.yml file is:
version: '2'
services:
redis:
image: "redis:alpine"
web:
depends_on:
- redis
build: .\app
volumes:
- .\app:/code
restart: always
reverseproxy:
image: reverseproxy
network_mode: "host"
ports:
- 8080:8080
depends_on:
- web
I need to set network_mode to host else the the X-Forwarded-For will wrong.
I managed to get it working by using a Linux host instead of Windows which meant I didn't need to use network_mode: "host". I also had to change my Python code to
request.environ.get('HTTP_X_REAL_IP', request.remote_addr)
from
request.environ['REMOTE_ADDR']
I have tried various options e.g. expose, bridge, networks options of docker-compose but can't get it to work with the nginx connection to upstream gunicorn running in separate container, I am receiving 502 Bad Gateway error from nginx. I am not sure as to what i am missing exactly. Below is my docker-compose file:
version: "3"
services:
web:
build: .
container_name: web
command: bash -c "/start_web.sh"
restart: always
depends_on:
- worker
ports:
- "80:80"
- "443:443"
worker:
build: .
container_name: worker
command: bash -c "/start_worker.sh"
restart: always
ports:
- "8000:8000"
nginx conf:
upstream worker {
server 127.0.0.1:8000;
}
server {
listen 80 default_server;
location / {
proxy_http_version 1.1;
proxy_buffering off;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Url-Scheme $scheme;
proxy_redirect off;
# Mitigate httpoxy attack
proxy_set_header Proxy "";
proxy_pass http://worker;
}
}
Gunicorn config:
import multiprocessing
import os
bind = '127.0.0.1:8000'
default_workers = multiprocessing.cpu_count() * 2 + 1
workers = os.getenv('GUNICORN_WORKERS', os.getenv('WEB_CONCURRENCY', default_workers))
worker_class = 'tornado'
# This is to fix issues with compressor package: broken offline manifest for
# custom domain. It randomly breaks, I think because of global variable inside.
preload_app = True
timeout = 200
graceful_timeout = 60
max_requests = 250
max_requests_jitter = max_requests
accesslog = '/tmp/gunicorn_access.log'
errorlog = '/tmp/gunicorn_error.log'
Circus ini files:
web.ini
[watcher:nginx]
cmd = /usr/sbin/nginx
stop_signal = QUIT
worker.ini
[watcher:gunicorn]
cmd = /usr/local/bin/gunicorn test:app -c /etc/gunicorn/app.py
working_dir = /opt/app
copy_env = True
uid = www-data
The whole code is available on github as well at repository docker_test for your ease to test it.
Gunicon config:
bind = '127.0.0.1:8000'
This will bind to loopback interface (localhost only), change it to 0.0.0.0 to bind to every available interface in the container. This will make it reachable from nginx.
Nginx config:
upstream worker {
server 127.0.0.1:8000;
}
You need to change loopback ip to DNSname/IP of worker container. I recommend creating an user-defined network, Then put all containers that are related in that network and call them by DNS names. You wont have internal DNS in default bridge network so following nginx config wont work.
upstream worker {
server worker:8000;
}
I have been following a tutorial on how to make a load balanced application using docker-compose and nginx. However, my load balancer/coordinator doesn't work - what I am trying to do is have nginx accept requests and split them between three workers, and I want nginx and the three workers to be running in separate docker containers, but I get the following error. My compilerwebservice_worker does work correctly, and I can see all three in docker ps, and I can ping them with wget on the localhost post they are listening to.
The error message
$ docker-compose up
Starting compilerwebservice_worker1_1
Starting compilerwebservice_worker3_1
Starting compilerwebservice_worker2_1
Starting compilerwebservice_nginx_1
Attaching to compilerwebservice_worker1_1, compilerwebservice_worker3_1, compilerwebservice_worker2_1, compilerwebservice_nginx_1
nginx_1 | 2016/09/06 07:17:47 [emerg] 1#1: host not found in upstream "compiler-web-service" in /etc/nginx/nginx.conf:14
nginx_1 | nginx: [emerg] host not found in upstream "compiler-web-service" in /etc/nginx/nginx.conf:14
compilerwebservice_nginx_1 exited with code 1
NGINX Config
http {
upstream compiler {
least_conn;
server worker1:4567;
server worker2:4567;
server worker3:4567;
}
server {
listen 4567;
location / {
proxy_pass http://compiler;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
}
}
My Docker-compose file
nginx:
build: ./src/main/nginx
links:
- worker2:worker2
- worker3:worker3
- worker1:worker1
ports:
- "4567:4567"
worker1:
build: .
ports:
- "4567"
worker2:
build: .
ports:
- "4567"
worker3:
build: .
ports:
- "4567"
NGINX Docker file
# Set nginx base image
FROM nginx
# Copy custom configuration file from the current directory
COPY nginx.conf /etc/nginx/nginx.conf
In the below demo there are 2 express app running with port 1111 and 2222 in localhost, on calling http://localhost:8080 it should automatically choose any one of the port 1111 or 2222. here nginx uses round robin
index.js file
const express = require('express');
const app = express();
const appId = process.env.APPID;
const PORTNUMBER = appId;
app.get('/', (req, res) => {
res.send({
message: `Welcome to ${appId} home page running on port ${appId}`
});
});
app.listen(PORTNUMBER, () => {
console.log(`APP STARTED ON PORT ${appId} for APP id ${appId}`);
})
express app docker file
FROM node:12.13.0-alpine
WORKDIR /EXPRESSAPP
COPY ./API/package.json /EXPRESSAPP
RUN npm install
COPY ./API/. /EXPRESSAPP
CMD ["npm", "start"]
nginx file
http {
upstream backend {
server 127.0.0.1:1111;
server 127.0.0.1:2222;
}
server {
listen 8080 default_server;
listen [::]:8080 default_server;
# listen [::]:8080 default_server ipv6only=on;
server_name localhost;
proxy_read_timeout 5m;
location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
proxy_pass http://backend;
}
}
}
nginx dockercompose file
FROM nginx
COPY nginx.conf /etc/nginx/nginx.conf
docker-compose.yml file
version: '3'
services:
myapp1:
restart: always
container_name: myapp1
build: ./APICONTAINER
environment:
- APPID=1111
ports:
- "1111:1111"
network_mode: host
myapp2:
restart: always
container_name: myapp2
build: ./APICONTAINER
environment:
- APPID=2222
ports:
- "2222:2222"
network_mode: host
myproxy:
container_name: myproxy
build: ./NGINXCONTAINER
ports:
- "127.0.0.1:8080:8080"
depends_on:
- myapp1
- myapp2
network_mode: host
to spin up the containers use the below command
sudo docker-compose down && sudo docker-compose up --build --force-recreate
go to below link to see the round robin nginx load balancer
http://localhost:8080
reference github link to get the full code
I needed to rebuild with docker-compose build between configuration changes. As I changed the name of the app, the error message indicated a server, whose name was the original one that I selected instead of the one I kept changing.