nginx: oscp stapling not performed on revoked cert - nginx

It seems nginx only performs OCSP stapling if the certificate is found to be not revoked.
Here is a TLS handshake with OCSP stapling enabled on nginx, with the server certificate revoked:
openssl s_client -connect helloworld.example.com:11443 -status -servername helloworld.example.com | grep OCSP
OCSP response: no response sent
As can be seen above, there is no OCSP response included.
Judging from the logs below, nginx does make an OCSP request to the OCSP server, it has understood that the certificate is revoked and creates an [error] log.
2023/01/26 14:35:55 [error] 29#29: certificate status "revoked" in the OCSP response while requesting certificate status, responder: ocsp, peer: 172.18.0.2:8888, certificate: "/etc/ssl/helloworld.example.com.crt"
o
In the OCSP server logs the request from nginx is also visible:
ocsp_1 | OCSP Request Data:
(...)
ocsp_1 | OCSP Response Data:
ocsp_1 | OCSP Response Status: successful (0x0)
ocsp_1 | Response Type: Basic OCSP Response
(...)
ocsp_1 | Cert Status: revoked
Here's the same nginx configuration, but with the certificate not revoked. In that case, everything works as expected and the client gets a stapled HTTP response.
depth=1 O = (...)
verify return:1
depth=0 CN = helloworld.example.com
verify return:1
OCSP response:
OCSP Response Data:
OCSP Response Status: successful (0x0)
Response Type: Basic OCSP Response
...
nginx itself doesn't log anything. The OSCP shows the request from nginx as well.
ocsp_1 | OCSP Request Data:
(...)
ocsp_1 | OCSP Response Data:
ocsp_1 | OCSP Response Status: successful (0x0)
ocsp_1 | Response Type: Basic OCSP Response
(...)
ocsp_1 | Cert Status: good
My conclusion is that nginx does not staple revoked certificates. This article seems to be in line with this observation:
https://mailman.nginx.org/pipermail/nginx/2014-April/043126.html
I would like OCSP stapling specifically for the case of revoked certificates. How can I get that to work or is this a design-choice of nginx that cannot be changed?

Related

Parse HTTP Basic Authorization Header in NGINX

Can Nginx natively parse the Authorization header (base 64) and return the username password? Or is a custom Lua function required to do this?
Example Request
curl -v -u USERNAME:PASSWORD https://example.com
> GET / HTTP/2
> Host: example.com
> Authorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ=

proper config for reverse proxy for grafana simple json data source

I want to put few services behind reverse proxy, simple services work.
The 502 issue occurs when trying to reach grafana simple json data source (POST + json payload).
Grafana itself is not behind reverse proxy.
current haproxy config:
frontend FRONT.AWS.WEB.PROXY
mode http
bind *:8080
timeout client 1m
option httplog
acl IS_RELE path_beg /release
acl IS_GRAF path_beg /grafana
use_backend BACK.AWS.WEB.ARTIFACTS if IS_RELE
use_backend BACK.AWS.WEB.RTGRAF if IS_GRAF
backend BACK.AWS.WEB.ARTIFACTS
mode http
http-request set-path /
http-response replace-value X-Application-Context (.*)(\release).*$ \1
server AWS.WEB.ARTIFACTS *:5581/ maxconn 1000 check port 5581
backend BACK.AWS.WEB.RTGRAF
mode http
#option forwardfor
#balance source
#option httpclose
#option httpchk HEAD / HTTP/1.0
http-request set-path /
http-response replace-value X-Application-Context (.*)(\grafana).*$ \1
server AWS.WEB.RTGRAF *:5582/ maxconn 1000 check port 5582
data source config in grafana:
http://192.168.56.101:8080/grafana/
This is working request without haproxy:
curl -d '{"requestId":"Q119","timezone":"utc".....lters":[]}' -H 'Content-Type: application/json' http://localhost:8080/query
good response:
[{"columns":[{"text":"sym","type":"string"}, {"text":"time","type":""}, .... .... {"text":"mode","type":"string"}, {"text":"proto","type":"string"}],"rows":[],"type":"table"}]
BUT with haproxy:
curl -d .... http://localhost:8080/grafana/query
502 Response:
<h1>502 Bad Gateway</h1>
The server returned an invalid or incomplete response.
but just to confirm, service itself works:
curl http://localhost:8080/grafana/?2+1
Response:
<html><head><style>a{text-decoration:none}a:link{color:024C7E}a:visited{color:024C7E}a:active{color:958600}body{font:10pt verdana;text-align:justify}</style></head><body><pre>3
Haproxy log:
127.0.0.1:42362 [16/Sep/2020:21:57:14.430] FRONT.AWS.WEB.PROXY BACK.AWS.WEB.RTGRAF/AWS.WEB.RTGRAF 0/0/0/3/3 200 274 - - ---- 1/1/0/0/0 0/0 "GET /grafana/?2+1 HTTP/1.1"
127.0.0.1:42418 [16/Sep/2020:21:57:32.038] FRONT.AWS.WEB.PROXY BACK.AWS.WEB.RTGRAF/AWS.WEB.RTGRAF 0/0/0/-1/0 502 214 - - PH-- 1/1/0/0/0 0/0 "POST /grafana/query HTTP/1.1"
grafana log:
INFO[09-16|22:23:02] Request Completed logger=context userId=1 orgId=1 uname=admin method=POST path=/api/datasources/proxy/2/query status=502 remote_addr=192.168.56.1 time_ms=6 size=107 referer="http://192.168.56.101:3000/d/aQPWEJFmz/system-status?orgId=1&refresh=10s"
Found the problem, for anyone in the future, that's for you!
....................................................caring ancient developers
Debug requests with simple nc
You'd find, wrong path is requested from first HTTP GET
So that needs rewriting:
backend BACK.AWS.WEB.ARTIFACTS
mode http
http-request set-uri %[url,regsub(^/release/,/,)]
http-response replace-value X-Application-Context (.*)(\release).*$ \1
server AWS.WEB.ARTIFACTS *:5581/ maxconn 1000 check port 5581
backend BACK.AWS.WEB.RTGRAF
mode http
http-response replace-value X-Application-Context (.*)(\grafana).*$ \1
server AWS.WEB.RTGRAF *:5582/ maxconn 1000 check port 5582

nginx-ingress pods keep crashing when request comes - AKS

our nginx-controller pods keep crashing when a request comes. From the logs, it looks like it has timeout connecting to API server, any idea how to enable more detailed logs?
I1213 14:55:35.038444 7 round_trippers.go:438] GET https://11.2.9.1:443/version?timeout=32s in 46 milliseconds
I1213 14:55:35.038543 7 round_trippers.go:444] Response Headers:
I1213 14:55:35.038650 7 request.go:784] Got a Retry-After 1s response for attempt 9 to https://11.2.9.1:443/version?timeout=32s
I1213 14:55:36.038955 7 round_trippers.go:419] curl -k -v -XGET -H "Accept: application/json, */*" -H "User-Agent: nginx-ingress-controller/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer XXXXXXXXXXRiWDII8dG8v-KJ90Av6HgE" 'https://11.2.9.1:443/version?timeout=32s'
I1213 14:55:36.088346 7 round_trippers.go:438] GET https://11.2.9.1:443/version?timeout=32s in 49 milliseconds
I1213 14:55:36.088382 7 round_trippers.go:444] Response Headers:
I1213 14:55:36.088598 7 request.go:947] Response Body:
I1213 14:55:36.088730 7 main.go:212] Unexpected error discovering Kubernetes version (attempt 9): an error on the server ("") has prevented the request from succeeding
F1213 14:55:36.088826 7 main.go:235] Error while initiating a connection to the Kubernetes API server. This could mean the cluster is misconfigured (e.g. it has invalid API server certificates or Service Accounts configuration). Reason: an error on the server ("") has prevented the request from succeeding
Refer to the troubleshooting guide for more information: https://kubernetes.github.io/ingress-nginx/troubleshooting/
when kubectl into the ingress pod, this is the log
C:\Users\XXXXX>kubectl exec -it nginx-ingress-controller-85d79fd99d-tlzrz -- /bin/bash
www-data#nginx-ingress-controller-85d79fd99d-tlzrz:/etc/nginx$ curl -k -v -XGET https://11.2.9.1:443/version?timeout=32s
Note: Unnecessary use of -X or --request, GET is already inferred.
* Expire in 0 ms for 6 (transfer 0x56450f95cdd0)
* Trying 11.2.9.1...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x56450f95cdd0)
* Connected to 11.2.9.1 (11.2.9.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 11.2.9.1:443
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 11.2.9.1:443
www-data#nginx-ingress-controller-85d79fd99d-tlzrz:/etc/nginx$
it is due to network security policy enforced that does not allow the ingress node to ping API server by internal IP. Adding the env variable to the ingress controller deployment file to force it to use FQDN solves the issue.KUBERNETES_SERVICE_HOST=FQDN of the API server

Nginx SNI + OCSP stapling not working

I recently tried to setup OCSP on one of my nginx servers.
Unfortunately I couldn't get it to work and didn't find a solution so far.
The configuration looks like this:
ssl_certificate /etc/ssl/private/mysite.com/combined.pem;
ssl_certificate_key /etc/ssl/private/mysite.com/privkey.pem;
ssl_trusted_certificate /etc/ssl/private/mysite.com/fullchain.crt;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 valid=300s;
resolver_timeout 15s;
The fullchain.crt contains the servers cert, the intermediate and the root cert.
If I check these certs by hand with :
openssl ocsp -issuer intermediate.crt -CAfile fullchain.crt -cert cert.crt -url http://tm.symcd.com -no_nonce
it returns ok:
Response verify OK
cert.crt: good
This Update: Apr 7 11:26:10 2018 GMT
Next Update: Apr 14 11:26:10 2018 GMT
But checking the server with s_client from elsewhere always returns
OCSP response: no response sent
even after waiting several minutes and nginx always throws the error:
2018/04/09 12:59:06 [error] 9474#9474: OCSP_basic_verify() failed (SSL: error:27069065:OCSP routines:OCSP_basic_verify:certificate verify error:Verify error:unable to get issuer certificate) while requesting certificate status, responder: tm.symcd.com
The server uses SNI since it delivers multiple sites with different certificates.
Somebody got an idea what I am missing here?
In your openssl s_client command, you should try again by adding the SNI option (-servername) :
openssl s_client -connect <fqdn>:443 -servername <fqdn> -status -tlsextdebug -tls1_2 ...
I had the same issue et that worked for me.

Will I be able to use CURL to get HTTP/2 headers?

Right now I use curl -I to retrieve headers.
Will sites adopt a different way of serving headers with HPACK in the upcoming adoption of HTTP/2 by browsers that will render my use of the curl command ineffective?
Yes, you can use curl to see and send HTTP headers with HTTP/2 just as you do with HTTP/1.
curl supports HTTP/2 and it is implemented as a sort of translation layer. It means it shows and "pretends" that headers work 1.1 style. It shows headers as text and it sends headers in callbacks like they were done with 1.1. We made it this way to make scripts and applications get a very smooth and basically invisible transition path to HTTP/2 with curl.
Internally that is of course done by decompressing received headers before showing them, and showing them before compressing them when sending them.
I believe it depends on curl version. HTTP/2 was added in curl 7.36.x IIRC ? not all distros would have that version ?
This is with curl 7.41.0 over HTTP/2 against https://google.com
curl --http2 -I -v https://google.com
* Rebuilt URL to: https://google.com/
* Trying 173.194.123.1...
* Connected to google.com (173.194.123.1) port 443 (#0)
* ALPN, offering h2-14, http/1.1
* ALPN, server accepted to use h2-14
* Server certificate:
* subject: C=US; ST=California; L=Mountain View; O=Google Inc; CN=*.google.com
* start date: 2015-03-11 16:13:43 GMT
* expire date: 2015-06-09 00:00:00 GMT
* subjectAltName: google.com matched
* issuer: C=US; O=Google Inc; CN=Google Internet Authority G2
* SSL certificate verify ok.
* Using HTTP2
edit: correction, curl --http2 needs nghttp2 compiled for it to work https://nghttp2.org/
curl --version
curl 7.41.0 (x86_64-unknown-linux-gnu) libcurl/7.41.0 OpenSSL/1.0.2b zlib/1.2.8 nghttp2/0.7.8-DEV
Protocols: dict file ftp ftps gopher http https imap imaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets

Resources