Letsencrypt unable to verify the domain - nginx

I have the following config working:
upstream app_admin {
server 127.0.0.1:8080;
}
server {
listen 80 default_server;
listen [::]:80 default_server ipv6only=on;
server_name admin.test.com;
location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_pass http://app_admin;
proxy_redirect off;
}
# for Let's Encrypt to work properly
location ^~ /.well-known {
root /var/www/;
default_type "text/plain";
allow all;
}
}
I installed Letsencrypt and ran this command, but I get errors:
/opt/letsencrypt# ./letsencrypt-auto certonly -a webroot --webroot-path=/var/www -d admin.test.com
Performing the following challenges:
http-01 challenge for admin.test.com
Using the webroot path /var/www for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. admin.test.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://admin.test.com/.well-known/acme-challenge/zlmdu1lWhUxZdwyV_1Kf--vgzEM6ETXr_qZzwR4uf6pM: Timeout
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: admin.test.com
Type: connection
Detail: Fetching
http://admin.test.com /.well-known/acme-challenge/zlmdu1lWhUbfdwyV_1Kf--vgzEM6ETXr_qZytfR4I6pM:
Timeout
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you're using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.
I can get to my site by going to admin.test.com without issues.
EDIT1:
With this command, I got the following:
$ ./letsencrypt-auto certonly --standalone
Please enter in your domain name(s) (comma and/or space separated) (Enter 'c'
to cancel):admin.test.com
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for admin.test.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. admin.test.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Timeout
Preformatted text
Again, I CAN get to my site with: admin.test.com (port 80).
Looking at the bottom of the log
/var/log/letsencrypt/letsencrypt.log
To fix these errors, please make sure that your domain name was entered correctly and the DNS A record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
2017-06-29 03:09:34,776:INFO:certbot.auth_handler:Cleaning up challenges
2017-06-29 03:09:34,777:DEBUG:certbot.plugins.standalone:Stopping server at :::443...
2017-06-29 03:09:35,014:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "/root/.local/share/letsencrypt/bin/letsencrypt", line 11, in
sys.exit(main())
File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py", line 743, in main
return config.func(config, plugins)
File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py", line 683, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py", line 82, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/client.py", line 344, in obtain_and_enroll_certificate
certr, chain, key, _ = self.obtain_certificate(domains)
File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/client.py", line 313, in obtain_certificate
self.config.allow_subset_of_names)
File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 81, in get_authorizations
self._respond(resp, best_effort)
File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 138, in _respond
self._poll_challenges(chall_update, best_effort)
File "/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 202, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. admin.test.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Timeout
What am I doing wrong?

Related

NGINX : Using Upstream server name

I am running nginx/1.19.6 on Ubuntu.
I am struggling to get the upstream module to work without returning a 404.
My *.conf files are located in /etc/nginx/conf.d/
FILE factory_upstream.conf:
upstream factoryserver {
server factory.service.corp.com:443;
}
FILE factory_service.conf:
server
{
listen 80;
root /data/www;
proxy_cache factorycache;
proxy_cache_min_uses 1;
proxy_cache_methods GET HEAD POST;
proxy_cache_valid 200 72h;
#proxy_cache_valid any 5m;
location /factory/ {
access_log /var/log/nginx/access-factory.log main;
proxy_set_header x-apikey abcdefgh12345678;
### Works when expressed as a literal.# proxy_pass https://factory.service.corp.com/;
### 404 when using the upstream name.
proxy_pass https://factoryserver/;
}
}
I have error logging set to debug, but after reloading the configuration and attempting a call, there are no new records in the error log.
nginx -t # Scans OK
nginx -s reload # no errors
cat /var/log/nginx/error.log
...
2021/03/16 11:29:52 [notice] 26157#26157: signal process started
2021/03/16 11:38:20 [notice] 27593#27593: signal process started
The access-factory.log does record the request :
127.1.2.3 --;[16/Mar/2021:11:38:50 -0400];";GET /factory/api/manifest/get-full-manifest/PID/904227100052 HTTP/1.1" ";/factory/api/manifest/get-full-manifest/PID/904227100052" ;404; - ;/factory/api/manifest/get-full-manifest/PID/904227100052";-" ";ID="c4cfc014e3faa803c8fe136d6eae347d ";-" ";10.8.9.123:443" ";404" ";-"
To help with debugging, I cached the 404 error, "proxy_cache_valid any 5m;" commented out in the example above:
When I use the upstream name, the cache file contains the followiing:
<##$ non-printable characters $%^>
KEY: https://factoryserver/api/manifest/get-full-manifest/PID/904227100052
HTTP/1.1 404 Not Found
Server: nginx/1.17.8
Date: Tue, 16 Mar 2021 15:38:50 GMT
...
The key contains the name 'factoryserver' I don't know if that matters or not. Does it?
The server version is different than what I see when I enter the command nginx -v, which is: nginx version: nginx/1.19.6
Does the difference in version in the cache file and the command line indicate anything?
When I switch back to the literal server name in the proxy_pass, I get a 200 response with the requested data. The Key in the cache file then contains the literal upstream server name.
<##$ non-printable characters $%^>
KEY: https://factory.service.corp.com/api/manifest/get-full-manifest/PID/904227100052
HTTP/1.1 200
Server: nginx/1.17.8
Date: Tue, 16 Mar 2021 15:59:30 GMT
...
I will have multiple upstream servers, each providing different services. The configuration will be deployed to multiple factories, each with its own upstream servers.
I would like for the deployment team to only have to update the *_upstream.conf files, and keep the *_service.conf files static from deployment site to deployment site.
factory_upstream.conf
product_upstream.conf
shipping_upstream.conf
abc123_upstream.conf
Why do I get a 404 when using a named upstream server?
Based on the nginx version in the cached response not matching what you see on the command line, it seems that maybe the 404 is coming from the upstream server. I.e, your proxying is working, but the upstream server is returning a 404. To troubleshoot further, I would check the nginx logs for the upstream server and if the incoming request is what you expect.
Note that when using proxy_pass, it makes a big difference whether you have a / at the end or not. With a trailing slash, nginx treats this as the URI it should send the upstream request to and it doesn't include the URI matched by the location block (/factory/) while without, it includes the full URI as-is.
proxy_pass https://factoryserver/ results in https://factory.service.corp.com:443/
proxy_pass https://factoryserver results in https://factory.service.corp.com:443/factory/
Docs: https://docs.nginx.com/nginx/admin-guide/web-server/reverse-proxy/
So maybe when switching between using an upstream and specifying the literal server name, you're inadvertently being inconsistent with the trailing slash. It's a really easy detail to miss, especially when you don't know it's important.
Please provide more information about your server configuration where you proxy pass the requests. Only difference I see now is that you specify the port (443) in your upstream server.

BigblueButton serves only on HTTP not HTTPS

I installed on Ubuntu 16.04, 4 cores, 8Gb RAM. I ran the cerbot command and it returned a congratulatory message that it's successful.
This is my first time installing BigBlueButton. I followed the process and all seemed fine until I tried running it on HTTPS https://live.oltega.com, and it returned
This site can’t be reached
live.oltega.com refused to connect.
Try: Checking the connection
Checking the proxy and the firewall
ERR_CONNECTION_REFUSED
when I served the same on HTTP http://live.oltega.com it worked well but displays a blue screen because it can only work on HTTPS. What can I try next?
After obtaining a Let's Encrypt certificate, you should configure the BBB components like Nginx and Freeswith to use HTTPS.
Follow the instructions mentioned here.
The summary is :
1-Configure FreeSWITCH for using SSL
Edit the file /etc/bigbluebutton/nginx/sip.nginx and change the protocol and port on the proxy_pass line as shown bellow
location /ws {
proxy_pass https://203.0.113.1:7443;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_read_timeout 6h;
proxy_send_timeout 6h;
client_body_timeout 6h;
send_timeout 6h;
}
2- Configure BigBlueButton to load session via HTTPS
Edit /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties and update the property bigbluebutton.web.serverURL to use HTTPS:
#----------------------------------------------------
# This URL is where the BBB client is accessible. When a user successfully
# enters a name and password, she is redirected here to load the client.
bigbluebutton.web.serverURL=https://bigbluebutton.example.com
Next, edit the file /usr/share/red5/webapps/screenshare/WEB-INF/screenshare.properties and update the property jnlpUrl and jnlpFile to HTTPS:
streamBaseUrl=rtmp://bigbluebutton.example.com/screenshare
jnlpUrl=https://bigbluebutton.example.com/screenshare
jnlpFile=https://bigbluebutton.example.com/screenshare/screenshare.jnlp
Next, you should do the following command:
$ sudo sed -e 's|http://|https://|g' -i /var/www/bigbluebutton/client/conf/config.xml
Open /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml editing and change:
kurento:
wsUrl: ws://bbb.example.com/bbb-webrtc-sfu
to
kurento:
wsUrl: wss://bbb.example.com/bbb-webrtc-sfu
and also
note:
enabled: true
url: http://bbb.example.com/pad
to
note:
enabled: true
url: https://bbb.example.com/pad
3- Next, modify the creation of recordings so they are served via HTTPS
Edit /usr/local/bigbluebutton/core/scripts/bigbluebutton.yml and change the value for playback_protocol as follows:
playback_protocol: https
4-If you have installed the API demos, edit /var/lib/tomcat7/webapps/demo/bbb_api_conf.jsp and change the value of BigBlueButtonURL use HTTPS.
// This is the URL for the BigBlueButton server
String BigBlueButtonURL = "https://bigbluebutton.example.com/bigbluebutton/";
5-Finally, to apply all of the configuration changes made, you must restart all components of BigBlueButton:
$ sudo bbb-conf --restart

Server side event is not working Kubernetes ingress controller and erred out

We have an API which will create a cluster and wait for the status until it creates and executes a query in database.
We tried this through Ingress and are getting timed-out.
And we have set in ingress rule as below:
nginx.ingress.kubernetes.io/configuration-snippet: |
location / {
proxy_set_header Connection "";
proxy_http_version 1.1;
}
Error:
nginx.ingress.kubernetes.io/proxy-read-timeout: "600"
If we set this it's creating under /data path which is our main API path, and it's not working as expected.
Is there any direct annotation to use server side events?
If we use the above snippet in ingress rule and get this error in controller logs.
Below error Error:
exit status 1
2020/06/26 04:57:22 [emerg] 132#132: location "/" is outside location "/data/" in /tmp/nginx-cfg140739857:11409
nginx: [emerg] location "/" is outside location "/data/" in /tmp/nginx-cfg140739857:11409
nginx: configuration file /tmp/nginx-cfg140739857 test failed.
As the doc suggests, you should never use / for hosting your data from. It should be somewhere in /data/*.
Some directories in any file system should never be used for hosting data from. These include / and root. You should never use these as your document root.
Doing this leaves you open to a request outside of your expected area returning private data.
NEVER DO THIS!!!
server {
root /;
location / {
}
}

Enable Letsencrypt in Debian 9 with Nginx for Gogs

I just installed Gogs on a VPS with the help of the tuto (https://gogs.io/docs/installation/install_from_source).
I have a sub domain to reach my gogs instance: git.mydomainname.com and it works: http://git.mydomainname.com goes to my gogs instance with a reverse proxy.
I would like to have my gogs protected through SSL, so I would like to install LetsEncrypt using the following tuto (https://certbot.eff.org/#debianstretch-nginx).
I would like to say that I am new to system administration and don't necessarily understand everything I did during the Gogs install.
I am also new to Nginx (more used to Apache).
Here is the process I followed:
$ sudo certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log
How would you like to authenticate with the ACME CA?
-------------------------------------------------------------------------------
1: Place files in webroot directory (webroot)
2: Spin up a temporary webserver (standalone)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Please enter in your domain name(s) (comma and/or space separated) (Enter 'c'
to cancel):git.mydomainname.com
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for git.mydomainname.com
Select the webroot for git.mydomainname.com:
-------------------------------------------------------------------------------
1: Enter a new webroot
-------------------------------------------------------------------------------
Press 1 [enter] to confirm the selection (press 'c' to cancel): /home/git/go/src/github.com/gogits/gogs
** Invalid input **
Press 1 [enter] to confirm the selection (press 'c' to cancel): 1
Input the webroot for git.mydomainname.com: (Enter 'c' to cancel):/home/git/go/src/github.com/gogits/gogs
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. git.mydomainname.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://git.mydomainname.com/.well-known/acme-challenge/N4rMGzoq1Bwyt9MP9fUlVY3_mDnJfRYpQkdvc7WrNJs: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>"
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: git.mydomainname.com
Type: unauthorized
Detail: Invalid response from
http://git.mydomainname.com/.well-known/acme-challenge/N4rMGzoq1Bwyt9MP9fUlVY3_mDnJfRYpQkdvc7WrNJs:
"<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A record(s) for that domain
contain(s) the right IP address.
So I checked the error, the DNS A record is OK.
I also found another tuto in french (https://www.grafikart.fr/formations/serveur-linux/nginx-ssl-letsencrypt) to help me and I noticed that I had to update my nginx config for the website, I did, despite I have a reverse proxy (maybe the issue is here).
server {
listen 80;
server_name git.mydomainname.com
location ~ /\.well-known/acme-challenge {
allow all;
}
location ~ /\. {
deny all;
access_log off;
log_not_found off;
}
location / {
proxy_pass http://localhost:port_number;
}
}
Thanks for your help.
You are proxifying all of your requests to http://localhost:port_number, but this program probably doesn't know how to handle lets-encrypt request.
Instead, you should change your .well-known location to :
location ^~ /.well-known/acme-challenge/ {
allow all;
root /var/www/letsencrypt;
}
And when the certbot ask you for a webroot, you can answer /var/www/letsencrypt
Note: you can change /var/www/letsencrypt to any directory you want. It just need to be created first, and readable by your nginx's user

Nginx memcached with fallback to remote service

I can't get Nginx working with memcached module, the requirement is to query remote service, cache data in memcached and never fetch remote endpoint until backend invalidates the cache. I have 2 containers with memcached v1.4.35 and one with Nginx v1.11.10.
The configuration is the following:
upstream http_memcached {
server 172.17.0.6:11211;
server 172.17.0.7:11211;
}
upstream remote {
server api.example.com:443;
keepalive 16;
}
server {
listen 80;
location / {
set $memcached_key "$uri?$args";
memcached_pass http_memcached;
error_page 404 502 504 = #remote;
}
location #remote {
internal;
proxy_pass https://remote;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
I tried to set memcached upstream incorrectly but I get HTTP 499 instead and warnings:
*3 upstream server temporarily disabled while connecting to upstream
It seems with described configuration Nginx can reach memcached successfully but can't write or read from it. I can write and read to memcached with telnet successfully.
Can you help me please?
My guesses on what's going on with your configuration
1. 499 codes
HTTP 499 is nginx' custom code meaning the client terminated connection before receiving the response (http://lxr.nginx.org/source/src/http/ngx_http_request.h#0120)
We can easily reproduce it, just
nc -k -l 172.17.0.6 172.17.0.6:11211
and curl your resource - curl will hang for a while and then press Ctrl+C — you'll have this message in your access logs
2. upstream server temporarily disabled while connecting to upstream
It means nginx didn't manage to reach your memcached and just removed it from the pool of upstreams. Suffice is to shutdown both memcached servers and you'd constantly see it in your error logs (I see it every time with error_log ... info).
As you see these messages your assumption that nginx can freely communicate with memcached servers doesn't seem to be true.
Consider explicitly setting http://nginx.org/en/docs/http/ngx_http_memcached_module.html#memcached_bind
and use the -b option with telnet to make sure you're correctly testing memcached servers for availability via your telnet client
3. nginx can reach memcached successfully but can't write or read from it
Nginx can only read from memcached via its built-in module
(http://nginx.org/en/docs/http/ngx_http_memcached_module.html):
The ngx_http_memcached_module module is used to obtain responses from
a memcached server. The key is set in the $memcached_key variable. A
response should be put in memcached in advance by means external to
nginx.
4. overall architecture
It's not fully clear from your question how the overall schema is supposed to work.
nginx's upstream uses weighted round-robin by default.
That means your memcached servers will be queried once at random.
You can change it by setting memcached_next_upstream not_found so a missing key will be considered an error and all of your servers will be polled. It's probably ok for a farm of 2 servers, but unlikely is it what your want for 20 servers
the same is ordinarily the case for memcached client libraries — they'd pick a server out of a pool according to some hashing scheme => so your key would end up on only 1 server out of the pool
5. what to do
I've managed to set up a similar configuration in 10 minutes on my local box - it works as expected. To mitigate debugging I'd get rid of docker containers to avoid networking overcomplication, run 2 memcached servers on different ports in single-threaded mode with -vv option to see when requests are reaching them (memcached -p 11211 -U o -vv) and then play with tail -f and curl to see what's really happening in your case.
6. working solution
nginx config:
https and http/1.1 is not used here but it doesn't matter
upstream http_memcached {
server 127.0.0.1:11211;
server 127.0.0.1:11212;
}
upstream remote {
server 127.0.0.1:8080;
}
server {
listen 80;
server_name server.lan;
access_log /var/log/nginx/server.access.log;
error_log /var/log/nginx/server.error.log info;
location / {
set $memcached_key "$uri?$args";
memcached_next_upstream not_found;
memcached_pass http_memcached;
error_page 404 = #remote;
}
location #remote {
internal;
access_log /var/log/nginx/server.fallback.access.log;
proxy_pass http://remote;
proxy_set_header Connection "";
}
}
server.py:
this is my dummy server (python):
from random import randint
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello: {}\n'.format(randint(1, 100000))
This is how to run it (just need to install flask first)
FLASK_APP=server.py [flask][2] run -p 8080
filling in my first memcached server:
$ telnet 127.0.0.1 11211
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
set /? 0 900 5
cache
STORED
quit
Connection closed by foreign host.
checking:
note that we get a result every time although we stored data
only in the first server
$ curl http://server.lan && echo
cache
$ curl http://server.lan && echo
cache
$ curl http://server.lan && echo
cache
this one is not in the cache so we'll get a response from server.py
$ curl http://server.lan/?q=1 && echo
Hello: 32337
whole picture:
the 2 windows on the right are
memcached -p 11211 -U o -vv
and
memcached -p 11212 -U o -vv

Resources