We have k8s-cluster with ingress-controller (nginx version: nginx/1.17.8, ingress: rke2-ingress-nginx:1.36.301) - 4 replicas (one per worker)
To improve performance and be able to reuse the session between many replicas, I try to activate the ssl_session_cache & tickets.
ingress-configmap have following data-Block:
ssl-session-cache-size: 100m
ssl-session-tickets: "true"
ssl-session-timeout: 1440m
ssl-protocols: "TLSv1.2"
ssl-session-ticket-key: "RESULT OF openssl rand 80 | openssl enc -A -base64"
When i decrease replica count to 1, i get correct handshake and i can reuse the ticket in further requests without handshake.
When i increase the replicas count to > 1, and send the session ticket in second (third, fourth...) request (round-robin), i'm getting each time a new ticket from the server incl. full handshake. So session resumption doesn't work.
Configmap auto-update is working, each replica have the same nginx.conf and all the replicas have the same /etc/nginx/tickets.key
Ingress-controller-pod logs have no issues while updating the configmap:
I0929 13:56:50.165135 8 controller.go:137] Configuration changes detected, backend reload required.
I0929 13:56:50.267572 8 controller.go:153] Backend successfully reloaded.
Restarting all ingress-pods after updating the configmap also doesn't help.
Result of the configmap in nginx.conf:
ssl_protocols TLSv1.2;
ssl_early_data off;
# turn on session caching to drastically improve performance
ssl_session_cache builtin:1000 shared:SSL:100m;
ssl_session_timeout 1440m;
# allow configuring ssl session tickets
ssl_session_tickets on;
ssl_session_ticket_key /etc/nginx/tickets.key;
Thanks!
Related
My current cerbot version running is:
certbot --version
certbot 0.27.1
My ngnix has TLS v1.0 but I do not want that running anymore:
cat /etc/letsencrypt/options-ssl-nginx.conf
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file.
ssl_session_cache shared:le_nginx_SSL:1m;
ssl_session_timeout 1440m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS";
These are managed by certbot as seen and after some reseach I see that the latest version of certbot dont have tls v1.0
now I do not know how to update the certbot version.. need help
Certbot uses a configuration file to keep track of all the settings it uses, or at very least a record of the non-default ones. See here for information on that. However, updating certbot itself might be enough as I believe recent versions have been disabling 1.0 and 1.1 by default.
Honestly, I would also remove TLS 1.1 as well, which also has known issues. 1.2 is still safe. The only reason not to would be if clients are still using 1.1, which by now is fairly low of a chance. In fact, all major browsers are dropping support for 1.0 and 1.1 entirely. Support for TLS 1.2 was out in browsers ~2013, so the chances of you needing to support it are not that great.
Also, I always refer to this site which is periodically updated with recommended security configurations. Even their intermediate configuration recommends a minimum of TLS 1.2.
When I try to start the nginx server with this configuration I get an error
nginx: [emerg] no ssl_client_certificate for ssl_client_verify
My Configuration looks like
# HTTPS server
server {
listen 4443;
server_name localhost;
ssl on;
ssl_certificate /home/user/conf/ssl/server.pem;
ssl_certificate_key /home/user/conf/ssl/server.pem;
ssl_protocols TLSv1.2;
ssl_verify_client optional;
ssl_trusted_certificate /home/user/ssl/certs/certificate_bundle.pem;
include conf.d/api_proxy.conf;
}
As per the error, I should use ssl_client_certificate directive but as per the documentation if I don't want to send the list of certificates to clients I should use ssl_trusted_certificate.
http://nginx.org/en/docs/http/ngx_http_ssl_module.html#ssl_client_certificate
Can someone help me figure out what am I missing?
The answer is in the error message itself:
nginx: [emerg] no ssl_client_certificate for ssl_client_verify
If you disable ssl_client_verify in your configuration, the error will disappear upon the next start of nginx. It appears that the semantics of ssl_trusted_certificate only pertain to its exclusive use and is subject to "logical override" when other configuration directives are in play.
I would personally prefer enabling ssl_client_verify to mean: "client certificates are verified if presented; clients are given no information regarding what client certificates or authorities are trusted by the web server". But, I can also see advantage in troubleshooting TLS handshakes when this information is available. From a security perspective, I only see metadata presented to the client via openssl s_client; there is no public key or other "signature-critical" information that a malicious client could use to, for example, attempt to clone/reconstruct the CA.
For example, running the following against a local instance of nginx with your configuration:
openssl s_client -key client.key -cert client.crt -connect localhost:443
... would show data similar in structure to the following in the response:
Acceptable client certificate CA names
/CN=user/OU=Clients/O=Company/C=Location
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Shared Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA+SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
In my opinion, the value of the above is confined in value to troubleshooting.
Your DN structures are (ideally) not "security relevant" (especially if the context is an Internet-facing web service).
I am very concerned about the difference in performance between HTTPS and HTTP/2.
I am very concerned about the consumed time about each step of the process of connecting.
Include TCP handshakes SSL Handshakes and Client sent Application Data Client received Application Data.
I made a test.
Use Nginx Server, set port 440 : ssl, and port 442 : ssl http2
Turn off ssl_session_cache and ssl_session_tickets, then every request would process TCP Handshakes and SSL Handshakes
// https url : ssl_session_cache off; ssl_session_tickets off;
https://www.example.com:440/index.html
// http/2 url : ssl_session_cache off; ssl_session_tickets off;
https://www.example.com:442/index.html
At the end, I launched the two urls above 1000 times from a android app with OKHttp 3.8.1.
Each request has been added Header Connection : Close.
At the same time, use Wireshark for capturing packets.
The statistical results are unexpected!!!
HTTP/2 consume more time than HTTPS in the step Client Receive Application Data From Server.
Why? I am confused. Am i wrong? Any ideas??? Thank u.
statistical results (Time : ms)
https.url.and.http2.url
nginx.conf
[root#iZ941gs04jwZ ~]# nginx -V
nginx version: nginx/1.12.0
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC)
built with OpenSSL 1.0.2l 25 May 2017
TLS SNI support enabled
configure arguments: --prefix=/usr/local/webserver/nginx --with-http_stub_status_module --with-http_ssl_module --with-pcre=/root/tmp/pcre-8.35 --with-http_v2_module --with-openssl=../openssl-1.0.2l
I can't get Nginx working with memcached module, the requirement is to query remote service, cache data in memcached and never fetch remote endpoint until backend invalidates the cache. I have 2 containers with memcached v1.4.35 and one with Nginx v1.11.10.
The configuration is the following:
upstream http_memcached {
server 172.17.0.6:11211;
server 172.17.0.7:11211;
}
upstream remote {
server api.example.com:443;
keepalive 16;
}
server {
listen 80;
location / {
set $memcached_key "$uri?$args";
memcached_pass http_memcached;
error_page 404 502 504 = #remote;
}
location #remote {
internal;
proxy_pass https://remote;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
I tried to set memcached upstream incorrectly but I get HTTP 499 instead and warnings:
*3 upstream server temporarily disabled while connecting to upstream
It seems with described configuration Nginx can reach memcached successfully but can't write or read from it. I can write and read to memcached with telnet successfully.
Can you help me please?
My guesses on what's going on with your configuration
1. 499 codes
HTTP 499 is nginx' custom code meaning the client terminated connection before receiving the response (http://lxr.nginx.org/source/src/http/ngx_http_request.h#0120)
We can easily reproduce it, just
nc -k -l 172.17.0.6 172.17.0.6:11211
and curl your resource - curl will hang for a while and then press Ctrl+C — you'll have this message in your access logs
2. upstream server temporarily disabled while connecting to upstream
It means nginx didn't manage to reach your memcached and just removed it from the pool of upstreams. Suffice is to shutdown both memcached servers and you'd constantly see it in your error logs (I see it every time with error_log ... info).
As you see these messages your assumption that nginx can freely communicate with memcached servers doesn't seem to be true.
Consider explicitly setting http://nginx.org/en/docs/http/ngx_http_memcached_module.html#memcached_bind
and use the -b option with telnet to make sure you're correctly testing memcached servers for availability via your telnet client
3. nginx can reach memcached successfully but can't write or read from it
Nginx can only read from memcached via its built-in module
(http://nginx.org/en/docs/http/ngx_http_memcached_module.html):
The ngx_http_memcached_module module is used to obtain responses from
a memcached server. The key is set in the $memcached_key variable. A
response should be put in memcached in advance by means external to
nginx.
4. overall architecture
It's not fully clear from your question how the overall schema is supposed to work.
nginx's upstream uses weighted round-robin by default.
That means your memcached servers will be queried once at random.
You can change it by setting memcached_next_upstream not_found so a missing key will be considered an error and all of your servers will be polled. It's probably ok for a farm of 2 servers, but unlikely is it what your want for 20 servers
the same is ordinarily the case for memcached client libraries — they'd pick a server out of a pool according to some hashing scheme => so your key would end up on only 1 server out of the pool
5. what to do
I've managed to set up a similar configuration in 10 minutes on my local box - it works as expected. To mitigate debugging I'd get rid of docker containers to avoid networking overcomplication, run 2 memcached servers on different ports in single-threaded mode with -vv option to see when requests are reaching them (memcached -p 11211 -U o -vv) and then play with tail -f and curl to see what's really happening in your case.
6. working solution
nginx config:
https and http/1.1 is not used here but it doesn't matter
upstream http_memcached {
server 127.0.0.1:11211;
server 127.0.0.1:11212;
}
upstream remote {
server 127.0.0.1:8080;
}
server {
listen 80;
server_name server.lan;
access_log /var/log/nginx/server.access.log;
error_log /var/log/nginx/server.error.log info;
location / {
set $memcached_key "$uri?$args";
memcached_next_upstream not_found;
memcached_pass http_memcached;
error_page 404 = #remote;
}
location #remote {
internal;
access_log /var/log/nginx/server.fallback.access.log;
proxy_pass http://remote;
proxy_set_header Connection "";
}
}
server.py:
this is my dummy server (python):
from random import randint
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello: {}\n'.format(randint(1, 100000))
This is how to run it (just need to install flask first)
FLASK_APP=server.py [flask][2] run -p 8080
filling in my first memcached server:
$ telnet 127.0.0.1 11211
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
set /? 0 900 5
cache
STORED
quit
Connection closed by foreign host.
checking:
note that we get a result every time although we stored data
only in the first server
$ curl http://server.lan && echo
cache
$ curl http://server.lan && echo
cache
$ curl http://server.lan && echo
cache
this one is not in the cache so we'll get a response from server.py
$ curl http://server.lan/?q=1 && echo
Hello: 32337
whole picture:
the 2 windows on the right are
memcached -p 11211 -U o -vv
and
memcached -p 11212 -U o -vv
I would like to cluster my faye websocket server. A single server works well but I want to be prepared to scale.
My first attempt is to start a few thin servers on different sockets and then add them to the upstream for my server in nginx.
The bayeux messages are distributed amongst the clusters but chrome dev tools shows about 16
websocket 101 connections with connection closed in the frame tab of the networking panel:
Connection Close Frame (Opcode 8)
Connection Close Frame (Opcode 8, mask)
And a whole bunch of /meta/connect and /meta/handshake on the server side for all of the faye instances.
An excerpt:
D, [2013-11-15T15:34:50.215631 #5344] DEBUG -- : {"channel"=>"/meta/connect", "clientId"=>"q7odwfbovudiw87dg0jke3xbrg51tui", "connectionType"=>"callback-polling", "id"=>"p"}
D, [2013-11-15T15:34:50.245012 #5344] DEBUG -- : {"channel"=>"/meta/connect", "clientId"=>"ckowb5vz9pnbh7jwomc8h0qsk8t0nus", "connectionType"=>"callback-polling", "id"=>"r"}
D, [2013-11-15T15:34:50.285460 #5344] DEBUG -- : {"channel"=>"/meta/handshake", "version"=>"1.0", "supportedConnectionTypes"=>["callback-polling"], "id"=>"u"}
D, [2013-11-15T15:34:50.312919 #5344] DEBUG -- : {"channel"=>"/meta/handshake", "version"=>"1.0", "supportedConnectionTypes"=>["callback-polling"], "id"=>"w"}
D, [2013-11-15T15:34:50.356219 #5344] DEBUG -- : {"channel"=>"/meta/handshake", "version"=>"1.0", "supportedConnectionTypes"=>["callback-polling"], "id"=>"y"}
D, [2013-11-15T15:34:50.394820 #5344] DEBUG -- : {"channel"=>"/meta/handshake", "version"=>"1.0", "supportedConnectionTypes"=>["callback-polling"], "id"=>"10"}
Starting the thin servers (each in its own terminal):
thin start -C config/thin/development.yml -S /tmp/faye.1.sock
thin start -C config/thin/development.yml -S /tmp/faye.2.sock
thin start -C config/thin/development.yml -S /tmp/faye.3.sock
My thin config:
---
environment: development
timeout: 30
log: log/thin.log
pid: tmp/pids/thin.pid
max_conns: 1024
max_persistent_conns: 100
require: []
wait: 30
# socket: /tmp/faye.sock
daemonize: false
rackup: config.ru
My NGINX config:
upstream thin_cluster {
server unix:/tmp/faye.1.sock fail_timeout=0;
server unix:/tmp/faye.2.sock fail_timeout=0;
server unix:/tmp/faye.3.sock fail_timeout=0;
}
server {
# listen 443;
server_name ~^push\.mysite\.dev(\..*\.xip\.io)?$;
charset UTF-8;
tcp_nodelay on;
# ssl on;
# ssl_certificate /var/www/heypresto/certificates/cert.pem;
# ssl_certificate_key /var/www/heypresto/certificates/key.pem;
# ssl_protocols TLSv1 SSLv3;
location / {
proxy_pass http://thin_cluster;
proxy_redirect off;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
break;
}
}
Thought it was too good to be true to put more upstreams for websockets and it would just work, but it seems so close...
EDIT: I've realised that this is probably not going to work and I should probably create N servers (push1.mysite.dev, push2.mysite.dev etc) and have the backend tell the frontend which one to connect.
Still If there are any thoughts out there...