Call timed out (error 1006) - bigbluebutton - nginx

I have installed bigbluebutton on my serverand it was working properly but suddenly microphone can not connect anymore and after a while being stucked in "echo test" i'll get 1006 error and i have tested:
FreeSWITCH fails to bind to IPV4Anchor link for: freeswitch fails to bind to ipv4
In rare occasions after shutdown/restart, the FreeSWITCH database can get corrupted. This will cause FreeSWITCH to have problems binding to IPV4 address (you may see error 1006 when users try to connect).
To check, look in /opt/freeswitch/var/log/freeswitch/freeswitch.log for errors related to loading the database.
2018-10-25 11:05:11.444727 [ERR] switch_core_db.c:108 SQL ERR [unsupported file format]
2018-10-25 11:05:11.444737 [ERR] switch_core_db.c:223 SQL ERR [unsupported file format]
2018-10-25 11:05:11.444759 [NOTICE] sofia.c:5949 Started Profile internal-ipv6 [sofia_reg_internal-ipv6]
2018-10-25 11:05:11.444767 [CRIT] switch_core_sqldb.c:508 Failure to connect to CORE_DB sofia_reg_external!
2018-10-25 11:05:11.444772 [CRIT] sofia.c:3049 Cannot Open SQL Database [external]!
If you see these errors, clear the FreeSWITCH database (BigBlueButton doesn’t use the database and FreeSWITCH will recreate it on startup).
$ sudo systemctl stop freeswitch
$ rm -rf /opt/freeswitch/var/lib/freeswitch/db/*
$ sudo systemctl start freeswitch
but it doesn't solve the problem.
this is output of bbb-conf --check:
BigBlueButton Server 2.2.20 (2037)
Kernel version: 4.4.0-185-generic
Distribution: Ubuntu 16.04.6 LTS (64-bit)
Memory: 4045 MB
CPU cores: 4
/usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties (bbb-web)
bigbluebutton.web.serverURL: https://online.vikipoyan.ir
defaultGuestPolicy: ALWAYS_ACCEPT
svgImagesRequired: true
/etc/nginx/sites-available/bigbluebutton (nginx)
server name: 49.12.60.238
port: 80, [::]:80
port: 443 ssl
bbb-client dir: /var/www/bigbluebutton
/var/www/bigbluebutton/client/conf/config.xml (bbb-client)
Port test (tunnel): rtmp://online.vikipoyan.ir
red5: online.vikipoyan.ir
useWebrtcIfAvailable: true
/opt/freeswitch/etc/freeswitch/vars.xml (FreeSWITCH)
local_ip_v4: 49.12.60.238
external_rtp_ip: stun:stun.freeswitch.org
external_sip_ip: stun:stun.freeswitch.org
/opt/freeswitch/etc/freeswitch/sip_profiles/external.xml (FreeSWITCH)
ext-rtp-ip: $${local_ip_v4}
ext-sip-ip: $${local_ip_v4}
ws-binding: :5066
wss-binding: :7443
/usr/local/bigbluebutton/core/scripts/bigbluebutton.yml (record and playback)
playback_host: online.vikipoyan.ir
playback_protocol: https
ffmpeg: 4.2.2-1bbb1~ubuntu16.04
/etc/bigbluebutton/nginx/sip.nginx (sip.nginx)
proxy_pass: 49.12.60.238
/usr/local/bigbluebutton/bbb-webrtc-sfu/config/default.yml (Kurento SFU)
kurento.ip: 49.12.60.238
kurento.url: ws://127.0.0.1:8888/kurento
kurento.sip_ip: 49.12.60.238
localIpAddress: 49.12.60.238
recordScreenSharing: true
recordWebcams: true
codec_video_main: VP8
codec_video_content: VP8
/usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml (HTML5 client)
build: 968
kurentoUrl: wss://online.vikipoyan.ir/bbb-webrtc-sfu
enableListenOnly: true
# Potential problems described below
# Warning: API URL IPs do not match host:
#
# IP from ifconfig: 49.12.60.238
# /var/lib/tomcat7/demo/bbb_api_conf.jsp: online.vikipoyan.ir
# Warning: The API demos are installed and accessible from:
#
# https://online.vikipoyan.ir
#
# and
#
# https://online.vikipoyan.ir/demo/demo1.jsp
#
# These API demos allow anyone to access your server without authentication
# to create/manage meetings and recordings. They are for testing purposes only.
# If you are running a production system, remove them by running:
#
# apt-get purge bbb-demo
# Warning: You have this server defined for https, but in
#
# /etc/bigbluebutton/nginx/sip.nginx
#
# did not find the use of https in definition for proxy_pass
#
# proxy_pass http://49.12.60.238:5066;
#
# Warning: You have this server defined for https, but in
#
# /etc/bigbluebutton/nginx/sip.nginx
#
# did not find the use of port 7443 in definition for proxy_pass
#
# proxy_pass http://49.12.60.238:5066;
#
I tested everything i found and the problem still exists. is there any way?
Thanks in advance.

run the command:
sudo nano /etc/bigbluebutton/nginx/sip.nginx
something like this will open:
location /ws {
proxy_pass http://49.12.60.238:4066;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_read_timeout 6h;
proxy_send_timeout 6h;
client_body_timeout 6h;
send_timeout 6h;
}
change the (proxy_pass http://...) line to the following:
proxy_pass https://49.12.60.238:7443;

Please note that I followed the posts at https://github.com/bigbluebutton/bigbluebutton/issues/2628 but didn't get the issue resolved until I ran the following command:
$ wget -qO- https://ubuntu.bigbluebutton.org/bbb-install.sh | bash -s -- -v xenial-22 -s myserver.com -c turn.myserver.com:1004a200000000004cb3f2d6f000d85d -e johndoe#myserver.com -d -g | tee bbb-install.log

Related

BigblueButton serves only on HTTP not HTTPS

I installed on Ubuntu 16.04, 4 cores, 8Gb RAM. I ran the cerbot command and it returned a congratulatory message that it's successful.
This is my first time installing BigBlueButton. I followed the process and all seemed fine until I tried running it on HTTPS https://live.oltega.com, and it returned
This site can’t be reached
live.oltega.com refused to connect.
Try: Checking the connection
Checking the proxy and the firewall
ERR_CONNECTION_REFUSED
when I served the same on HTTP http://live.oltega.com it worked well but displays a blue screen because it can only work on HTTPS. What can I try next?
After obtaining a Let's Encrypt certificate, you should configure the BBB components like Nginx and Freeswith to use HTTPS.
Follow the instructions mentioned here.
The summary is :
1-Configure FreeSWITCH for using SSL
Edit the file /etc/bigbluebutton/nginx/sip.nginx and change the protocol and port on the proxy_pass line as shown bellow
location /ws {
proxy_pass https://203.0.113.1:7443;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_read_timeout 6h;
proxy_send_timeout 6h;
client_body_timeout 6h;
send_timeout 6h;
}
2- Configure BigBlueButton to load session via HTTPS
Edit /usr/share/bbb-web/WEB-INF/classes/bigbluebutton.properties and update the property bigbluebutton.web.serverURL to use HTTPS:
#----------------------------------------------------
# This URL is where the BBB client is accessible. When a user successfully
# enters a name and password, she is redirected here to load the client.
bigbluebutton.web.serverURL=https://bigbluebutton.example.com
Next, edit the file /usr/share/red5/webapps/screenshare/WEB-INF/screenshare.properties and update the property jnlpUrl and jnlpFile to HTTPS:
streamBaseUrl=rtmp://bigbluebutton.example.com/screenshare
jnlpUrl=https://bigbluebutton.example.com/screenshare
jnlpFile=https://bigbluebutton.example.com/screenshare/screenshare.jnlp
Next, you should do the following command:
$ sudo sed -e 's|http://|https://|g' -i /var/www/bigbluebutton/client/conf/config.xml
Open /usr/share/meteor/bundle/programs/server/assets/app/config/settings.yml editing and change:
kurento:
wsUrl: ws://bbb.example.com/bbb-webrtc-sfu
to
kurento:
wsUrl: wss://bbb.example.com/bbb-webrtc-sfu
and also
note:
enabled: true
url: http://bbb.example.com/pad
to
note:
enabled: true
url: https://bbb.example.com/pad
3- Next, modify the creation of recordings so they are served via HTTPS
Edit /usr/local/bigbluebutton/core/scripts/bigbluebutton.yml and change the value for playback_protocol as follows:
playback_protocol: https
4-If you have installed the API demos, edit /var/lib/tomcat7/webapps/demo/bbb_api_conf.jsp and change the value of BigBlueButtonURL use HTTPS.
// This is the URL for the BigBlueButton server
String BigBlueButtonURL = "https://bigbluebutton.example.com/bigbluebutton/";
5-Finally, to apply all of the configuration changes made, you must restart all components of BigBlueButton:
$ sudo bbb-conf --restart

SonarQube not reachable on 127.0.0.1:9000 under Ubuntu 18.04

I want to run SonarQube on my Ubuntu 18.04 server along with nginx (a Droplet at DigitalOcean).
Mostly I've followed these instructions. I've used Postgres instead of MySQL.
Nginx should accept the request and pass it to the localhost-address used by SonarQube (127.0.0.1:9000).
Nginx is running and working. SSL is active and working. Here my codequality.example.com.conf:
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name codequality.example.com www.codequality.example.com;
root /var/www/html;
index index.html index.htm;
access_log /var/log/nginx/codequality.example.com.access.log;
error_log /var/log/nginx/codequality.example.com.error.log;
ssl_certificate /etc/letsencrypt/live/codequality.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/codequality.example.com/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
location / {
proxy_pass http://127.0.0.1:9000;
proxy_next_upstream error timeout invalid_header http_500 http_502 http_503 http_504;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto http;
}
}
The command systemctl status sonarqube gives me the following response:
● sonarqube.service - SonarQube service
Loaded: loaded (/etc/systemd/system/sonarqube.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-03-17 22:16:50 UTC; 5s ago
Process: 21796 ExecStop=/opt/sonarqube/bin/linux-x86-64/sonar.sh stop (code=exited, status=0/SUCCESS)
Process: 21855 ExecStart=/opt/sonarqube/bin/linux-x86-64/sonar.sh start (code=exited, status=0/SUCCESS)
Main PID: 21918 (wrapper)
Tasks: 43 (limit: 2361)
CGroup: /system.slice/sonarqube.service
├─21918 /opt/sonarqube/bin/linux-x86-64/./wrapper /opt/sonarqube/bin/linux-x86-64/../../conf/wrapper.co
├─21922 java -Dsonar.wrapped=true -Djava.awt.headless=true -Xms8m -Xmx32m -Djava.library.path=./lib -cl
└─21954 /usr/lib/jvm/java-11-openjdk-amd64/bin/java -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyF
lines 1-11/11 (END)
So I can assume that SonarQube-Server is running correctly. Trying to access SonarQube via https://codequality.example.com results in a 502 Error. The log-file says:
2020/03/17 22:19:44 [error] 19598#19598: *233 connect() failed (111: Connection refused) while connecting to upstream, client: 79.254.63.100, server: codequality.example.com, request: "GET / HTTP/2.0", upstream: "http://127.0.0.1:9000/", host: "codequality.example.com"
Trying to access the localhost (127.0.0.1:9000) during a ssh-session via curl http://127.0.0.1:9000 I get the error:
curl: (7) Failed to connect to 127.0.0.1 port 9000: Connection refused
This is the log from SonarQube:
2020.03.17 22:38:53 INFO app[][o.s.a.AppFileSystem] Cleaning or creating temp directory /opt/sonarqube/temp
2020.03.17 22:38:53 INFO app[][o.s.a.es.EsSettings] Elasticsearch listening on /127.0.0.1:9001
2020.03.17 22:38:54 INFO app[][o.s.a.ProcessLauncherImpl] Launch process[[key='es', ipcIndex=1, logFilenamePrefix=es]] from [/opt/sonarqube/elasticsearch]: /opt/sonarqube/elasticsearch/bin/elasticsearch
2020.03.17 22:38:54 INFO app[][o.s.a.SchedulerImpl] Waiting for Elasticsearch to be up and running
2020.03.17 22:38:55 INFO app[][o.e.p.PluginsService] no modules loaded
2020.03.17 22:38:55 INFO app[][o.e.p.PluginsService] loaded plugin [org.elasticsearch.transport.Netty4Plugin]
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
ERROR: [2] bootstrap checks failed
1: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
2020.03.17 22:39:11 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 78
2020.03.17 22:39:11 INFO app[][o.s.a.SchedulerImpl] Process[es] is stopped
2020.03.17 22:39:11 INFO app[][o.s.a.SchedulerImpl] SonarQube is stopped
<-- Wrapper Stopped
--> Wrapper Started as Daemon
Launching a JVM...
Wrapper (Version 3.2.3) http://wrapper.tanukisoftware.org
Copyright 1999-2006 Tanuki Software, Inc. All Rights Reserved.
What am I doing wrong?
the problem is not the NGINX Proxy. The logfile you shared gives some insides
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535] [2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144] 2020.03.17 22:39:11 WARN app[][o.s.a.p.AbstractManagedProcess] Process exited with exit value [es]: 78 2020.03.17 22:39:11 INFO app[][o.s.a.SchedulerImpl] Process[es] is stopped
Your SonarQube isn't running correctly because of some misconfiguration of elastic sarch.
Check this links to find out how to adjust the limit:
general information
https://www.elastic.co/guide/en/elasticsearch/reference/current/file-descriptors.html
how to chnage the settings
https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html
Make sure SonarQube is running correctly on localhost:9000 before move on to the proxy configuration.

Nginx memcached with fallback to remote service

I can't get Nginx working with memcached module, the requirement is to query remote service, cache data in memcached and never fetch remote endpoint until backend invalidates the cache. I have 2 containers with memcached v1.4.35 and one with Nginx v1.11.10.
The configuration is the following:
upstream http_memcached {
server 172.17.0.6:11211;
server 172.17.0.7:11211;
}
upstream remote {
server api.example.com:443;
keepalive 16;
}
server {
listen 80;
location / {
set $memcached_key "$uri?$args";
memcached_pass http_memcached;
error_page 404 502 504 = #remote;
}
location #remote {
internal;
proxy_pass https://remote;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
}
I tried to set memcached upstream incorrectly but I get HTTP 499 instead and warnings:
*3 upstream server temporarily disabled while connecting to upstream
It seems with described configuration Nginx can reach memcached successfully but can't write or read from it. I can write and read to memcached with telnet successfully.
Can you help me please?
My guesses on what's going on with your configuration
1. 499 codes
HTTP 499 is nginx' custom code meaning the client terminated connection before receiving the response (http://lxr.nginx.org/source/src/http/ngx_http_request.h#0120)
We can easily reproduce it, just
nc -k -l 172.17.0.6 172.17.0.6:11211
and curl your resource - curl will hang for a while and then press Ctrl+C — you'll have this message in your access logs
2. upstream server temporarily disabled while connecting to upstream
It means nginx didn't manage to reach your memcached and just removed it from the pool of upstreams. Suffice is to shutdown both memcached servers and you'd constantly see it in your error logs (I see it every time with error_log ... info).
As you see these messages your assumption that nginx can freely communicate with memcached servers doesn't seem to be true.
Consider explicitly setting http://nginx.org/en/docs/http/ngx_http_memcached_module.html#memcached_bind
and use the -b option with telnet to make sure you're correctly testing memcached servers for availability via your telnet client
3. nginx can reach memcached successfully but can't write or read from it
Nginx can only read from memcached via its built-in module
(http://nginx.org/en/docs/http/ngx_http_memcached_module.html):
The ngx_http_memcached_module module is used to obtain responses from
a memcached server. The key is set in the $memcached_key variable. A
response should be put in memcached in advance by means external to
nginx.
4. overall architecture
It's not fully clear from your question how the overall schema is supposed to work.
nginx's upstream uses weighted round-robin by default.
That means your memcached servers will be queried once at random.
You can change it by setting memcached_next_upstream not_found so a missing key will be considered an error and all of your servers will be polled. It's probably ok for a farm of 2 servers, but unlikely is it what your want for 20 servers
the same is ordinarily the case for memcached client libraries — they'd pick a server out of a pool according to some hashing scheme => so your key would end up on only 1 server out of the pool
5. what to do
I've managed to set up a similar configuration in 10 minutes on my local box - it works as expected. To mitigate debugging I'd get rid of docker containers to avoid networking overcomplication, run 2 memcached servers on different ports in single-threaded mode with -vv option to see when requests are reaching them (memcached -p 11211 -U o -vv) and then play with tail -f and curl to see what's really happening in your case.
6. working solution
nginx config:
https and http/1.1 is not used here but it doesn't matter
upstream http_memcached {
server 127.0.0.1:11211;
server 127.0.0.1:11212;
}
upstream remote {
server 127.0.0.1:8080;
}
server {
listen 80;
server_name server.lan;
access_log /var/log/nginx/server.access.log;
error_log /var/log/nginx/server.error.log info;
location / {
set $memcached_key "$uri?$args";
memcached_next_upstream not_found;
memcached_pass http_memcached;
error_page 404 = #remote;
}
location #remote {
internal;
access_log /var/log/nginx/server.fallback.access.log;
proxy_pass http://remote;
proxy_set_header Connection "";
}
}
server.py:
this is my dummy server (python):
from random import randint
from flask import Flask
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello: {}\n'.format(randint(1, 100000))
This is how to run it (just need to install flask first)
FLASK_APP=server.py [flask][2] run -p 8080
filling in my first memcached server:
$ telnet 127.0.0.1 11211
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
set /? 0 900 5
cache
STORED
quit
Connection closed by foreign host.
checking:
note that we get a result every time although we stored data
only in the first server
$ curl http://server.lan && echo
cache
$ curl http://server.lan && echo
cache
$ curl http://server.lan && echo
cache
this one is not in the cache so we'll get a response from server.py
$ curl http://server.lan/?q=1 && echo
Hello: 32337
whole picture:
the 2 windows on the right are
memcached -p 11211 -U o -vv
and
memcached -p 11212 -U o -vv

Gitlab timeouts / slow on initial page loads

I am running Gitlab on Debian using the package from the Repository. Most of the time Gitlab is running very fast, but after longer idle times Gitlab is very slow or even times out (error 502). One time I also had a timeout on a remote git access (could not reproduce the issue - timeout on the internal API).
In my setup the the Debian machine is behind another nginx proxy which also serves some other services just fine. I did the gitlab-cli checks and everything seems fine.
In the error log of my reverse proxy I only see connection timeouts:
[error] 8643#0: *4139 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.1.1.10, server: gitlab.mydomain.tld, request: "GET / HTTP/1.1", upstream: "http://{SERVER-IP}:80/", host: "gitlab.mydomain.tld"
I can see some errors in my unicorn_stderr.log
E, [2016-03-30T19:40:20.183991 #783] ERROR -- : worker=1 PID:16798 timeout (61s > 60s), killing
E, [2016-03-30T19:40:20.194969 #783] ERROR -- : reaped #<Process::Status: pid 16798 SIGKILL (signal 9)> worker=1
I, [2016-03-30T19:40:20.197554 #16871] INFO -- : worker=1 spawned pid=16871
I, [2016-03-30T19:40:20.197909 #16871] INFO -- : worker=1 ready
E, [2016-03-30T20:08:42.911429 #783] ERROR -- : worker=0 PID:16866 timeout (61s > 60s), killing
E, [2016-03-30T20:08:43.191151 #783] ERROR -- : reaped #<Process::Status: pid 16866 SIGKILL (signal 9)> worker=0
I, [2016-03-30T20:08:43.758363 #18728] INFO -- : worker=0 spawned pid=18728
I, [2016-03-30T20:08:44.108244 #18728] INFO -- : worker=0 ready
What I am a bit curious about is the fact that there are no errors in the log of the nginx delivered with gitlab.
Some more system information:
#sudo gitlab-rake gitlab:env:info
System information
System: Debian 8.3
Current User: git
Using RVM: no
Ruby Version: 2.1.8p440
Gem Version: 2.5.1
Bundler Version:1.10.6
Rake Version: 10.5.0
Sidekiq Version:4.0.1
GitLab information
Version: 8.5.0
Revision: a513e09
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: http://gitlab.mydomain.tld
HTTP Clone URL: http://gitlab.mydomain.tld/some-group/some-project.git
SSH Clone URL: git#gitlab.mydomain.tld:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 2.6.10
Repositories: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git
Edit:
My nginx config on the "external" reverse proxy looks like this:
server {
listen 443;
ssl on;
server_name gitlab.mydomain.tld;
access_log /var/log/nginx/gitlab.mydomain.tld.access.log;
error_log /var/log/nginx/gitlab.mydomain.tld.error.log;
ssl_certificate /etc/nginx/ssl/gitlab.mydomain.tld_unified.crt;
ssl_certificate_key /etc/nginx/ssl/mydomain.tld.key;
location / {
proxy_pass http://gitlab:80;
proxy_redirect default;
proxy_set_header Host $http_host;
proxy_set_header X_FORWARDED_PROTO "https";
satisfy any;
}
}
Edit2:
I took the suggested answer into account and also considered this source: https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/requirements.md
I assigned 2GB RAM to the VM now, and also added one additional unicorn worker.
Edit3:
The problem seems to be solved by adding more memory and using 3 unicorn workers.
Jan,
I have a similar setup although our box is dedicated to GITlab. Without knowing the specs of your server (GITLAB likes memory) and the load on that box I would suggest the following diagnostics:
Does your upstream nginx use identical parameters as the gitlab nginx configuration? They have tweaked a number of things including timeouts.
What kind of request result in time outs? Some operations (like generating diffs) can take some time to render.
If you run the requests via SSH do you also experience time outs?
Have you checked global logs in /var/log?
FYI: I had to enlarge my small GitLab installation to have 4GB RAM not to throw OOM errors
Now I think, I'd better go with gogs or other alternative.

Nginx error: (13: Permission denied) while connecting to upstream

I am getting this error in my nginx-error.log file:
2014/02/17 03:42:20 [crit] 5455#0: *1 connect() to unix:/tmp/uwsgi.sock failed (13: Permission denied) while connecting to upstream, client: xx.xx.x.xxx, server: localhost, request: "GET /users HTTP/1.1", upstream: "uwsgi://unix:/tmp/uwsgi.sock:", host: "EC2.amazonaws.com"
The browser also shows a 502 Bad Gateway Error. The output of a curl is the same, Bad Gateway html
I've tried to fix it by changing permissions for /tmp/uwsgi.sock to 777. That didn't work. I also added myself to the www-data group (a couple questions that looked similar suggested that). Also, no dice.
Here is my nginx.conf file:
nginx.conf
worker_processes 1;
worker_rlimit_nofile 8192;
events {
worker_connections 3000;
}
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
I am running a Flask application with Nginsx and Uwsgi, just to be thorough in my explanation. If anyone has any ideas, I would really appreciate them.
EDIT
I have been asked to provide my uwsgi config file. So, I never personally wrote my nginx or my uwsgi file. I followed the guide here which sets everything up using ansible-playbook. The nginx.conf file was generated automatically, but there was nothing in /etc/uwsgi except a README file in both apps-enabled and apps-available folders. Do I need to create my own config file for uwsgi? I was under the impression that ansible took care of all of those things.
I believe that ansible-playbook figured out my uwsgi configuration since when I run this command
uwsgi -s /tmp/uwsgi.sock -w my_app:app
it starts up and outputs this:
*** Starting uWSGI 2.0.1 (64bit) on [Mon Feb 17 20:03:08 2014] ***
compiled with version: 4.7.3 on 10 February 2014 18:26:16
os: Linux-3.11.0-15-generic #25-Ubuntu SMP Thu Jan 30 17:22:01 UTC 2014
nodename: ip-10-9-xxx-xxx
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/username/Project
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 4548
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 3
Python version: 2.7.5+ (default, Sep 19 2013, 13:52:09) [GCC 4.8.1]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x1f60260
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72760 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x1f60260 pid: 26790 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 26790, cores: 1)
The permission issue occurs because uwsgi resets the ownership and permissions of /tmp/uwsgi.sock to 755 and the user running uwsgi every time uwsgi starts.
The correct way to solve the problem is to make uwsgi change the ownership and/or permission of /tmp/uwsgi.sock such that nginx can write to this socket. Therefore, there are three possible solutions.
Run uwsgi as the www-data user so that this user owns the socket file created by it.
uwsgi -s /tmp/uwsgi.sock -w my_app:app --uid www-data --gid www-data
Change the ownership of the socket file so that www-data owns it.
uwsgi -s /tmp/uwsgi.sock -w my_app:app --chown-socket=www-data:www-data
Change the permissions of the socket file, so that www-data can write to it.
uwsgi -s /tmp/uwsgi.sock -w my_app:app --chmod-socket=666
I prefer the first approach because it does not leave uwsgi running as root.
The first two commands need to be run as root user. The third command does not need to be run as root user.
The first command leaves uwsgi running as www-data user. The second and third commands leave uwsgi running as the actual user that ran the command.
The first and second command allow only www-data user to write to the socket. The third command allows any user to write to the socket.
I prefer the first approach because it does not leave uwsgi running as root user and it does not make the socket file world-writeable .
While the accepted solution is true there might also SELinux be blocking the access. If you did set the permissions correctly and still get permission denied messages try:
sudo setenforce Permissive
If it works then SELinux was at fault - or rather was working as expected! To add the permissions needed to nginx do:
# to see what permissions are needed.
sudo grep nginx /var/log/audit/audit.log | audit2allow
# to create a nginx.pp policy file
sudo grep nginx /var/log/audit/audit.log | audit2allow -M nginx
# to apply the new policy
sudo semodule -i nginx.pp
After that reset the SELinux Policy to Enforcing with:
sudo setenforce Enforcing
Anyone who lands here from the Googles and is trying to run Flask on AWS using the default Ubuntu image after installing nginx and still can't figure out what the problem is:
Nginx runs as user "www-data" by default, but the most common Flask WSGI tutorial from Digital Ocean has you use the logged in user for the systemd service file. Change the user that nginx is running as from "www-data" (which is the default) to "ubuntu" in /etc/nginx/nginx.conf if your Flask/wsgi user is "ubuntu" and everything will start working. You can do this with one line in a script:
sudo sed -i 's/user www-data;/user ubuntu;/' /etc/nginx/nginx.conf
Trying to make Flask and uwsgi run as www-data did not work off the bat, but making nginx run as ubuntu worked just fine since all I'm running with this instance is Flask anyhow.
You have to set these permissions (chmod/chown) in uWSGI configuration.
It is the chmod-socket and the chown-socket.
http://uwsgi-docs.readthedocs.org/en/latest/Options.html#chmod-socket
http://uwsgi-docs.readthedocs.org/en/latest/Options.html#chown-socket
Nginx connect to .sock failed (13:Permission denied) - 502 bad gateway
change the name of the user on the first line in /etc/nginx/nginx.conf file.
the default user is www-data and change it to root or your username
I know it's too late, but it might helps to other. I'll suggest to follow Running flask with virtualenv, uwsgi, and nginx very simple and sweet documentation.
Must activate your environment if you run your project in virtualenv.
here is the yolo.py
from config import application
if __name__ == "__main__":
application.run(host='127.0.0.1')
And create uwsgi.sock file in /tmp/ directory and leave it blank.
As #susanpal answer said "The permission issue occurs because uwsgi resets the ownership and permissions of /tmp/uwsgi.sock to 755 and the user running uwsgi every time uwsgi starts." it is correct.
So you have to give permission to sock file whenever uwsgi starts.
so now follow the below command
uwsgi -s /tmp/uwsgi.sock -w yolo:application -H /var/www/yolo/env --chmod-socket=666
A little different command from #susanpal.
And for persist connection, simply add "&" end of command
uwsgi -s /tmp/uwsgi.sock -w yolo:app -H /var/www/yolo/env --chmod-socket=666 &
In my case changing some php permission do the trick
sudo chown user:group -R /run/php
I hope this helps someone.
You should post both nginx and uwsgi configuration file for your application (the ones in /etc/nginx/sites-enabled/ and /etc/uwsgi/ - or wherever you put them).
Typically check that you have a line similar to the following one in your nginx app configuration:
uwsgi_pass unix:///tmp/uwsgi.sock;
and the same socket name in your uwsgi config file:
socket=/tmp/uwsgi.sock

Resources