I am new to Datadog and NGiNX. I noticed when I was creating a monitor for some integrations several of the integrations were labeled as misconfigured. My guess is someone clicked the install button but did finish the remaining integration steps. I started to work with NGiNX and quickly hit a roadblock.
I verified it is running http status module
$ nginx -V 2>&1| grep -o http_stub_status_module
http_stub_status_module
The NGiNX install is under a different directory than is usual
and the configuration file is under
/<dir>/parts/nginx/conf
I created the status.conf file there.
When I reload the NGINX I get a failure. I don't understand what it means or how to proceed from here.
nginx: [error] open() "/<dir>/parts/nginx/logs/nginx.pid" failed (2: No such file or directory)
There is a logs directory with nothing in it.
ps -ef|grep nginx
user 35958 88952 0 May24 ? 00:00:43 nginx: worker process
user 35959 88952 0 May24 ? 00:00:48 nginx: worker process
root 88952 1 0 Feb21 ? 00:00:00 nginx: master process <dir>/parts/nginx/sbin/nginx -c <dir>/etc/nginx/balancer.conf -g pid <dir>/var/nginx-balancer.pid; lock_file /<dir>/var/nginx-balancer.lock; error_log <dir>/var/logs/nginx-balancer-error.log;
user 109169 63043 0 13:13 pts/0 00:00:00 grep --color=auto nginx
I think the issue is that our install doesn't seem to be following the same defaults as the instructions and I'm pretty sure I'm not doing this correctly.
If anyone has any insights that would be great!
Chris
Related
while installing letsencrypt certificate the output error is:
nginx: the configuration file /jet/etc/nginx/nginx.conf syntax is ok
nginx: [emerg] getgrnam(“nobody”) failed
nginx: configuration file /jet/etc/nginx/nginx.conf test failed
but if I run:
nginx -t
to see if there are configuration errors in the nginx.conf file,
nginx -t return the output:
nginx: the configuration file /jet/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /jet/etc/nginx/nginx.conf test is successful
Every now and then on WordPress I also get the error:
internal server 500 error
and I don’t know if this is related to the problem I described above
My web server is (include version):
The operating system my web server runs on is (include version):
Linux lemp7-optimized-g2-1-vm 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u4 (2018-08-21) x86_64
jeware lemp7 stack profile
jetware
I managed to install the certificates by inserting a line with the “user nobody nogroup;” in the nginx.conf file
After the installation I removed the line with the “user nobody nogroup;” in the nginx.conf file
sudo wget https://dl.eff.org/certbot-auto
sudo mv certbot-auto /jet/etc/letsencrypt/certbot-auto
sudo chown root /jet/etc/letsencrypt/certbot-auto
sudo chmod 0755 /jet/etc/letsencrypt/certbot-auto
sudo chmod a+x /jet/etc/letsencrypt/certbot-auto
sudo /jet/etc/letsencrypt/certbot-auto --nginx --nginx-ctl /jet/bin/nginx --nginx-server-root /jet/etc/nginx --config-dir /jet/etc/letsencrypt
this produce this output:
nginx: the configuration file /jet/etc/nginx/nginx.conf syntax is ok
nginx: [emerg] getgrnam(“nobody”) failed
nginx: configuration file /jet/etc/nginx/nginx.conf test failed
but if I run:
nginx -t
to see if there are configuration errors in the nginx.conf file,
nginx -t return the output:
nginx: the configuration file /jet/etc/nginx/nginx.conf syntax is ok
nginx: configuration file /jet/etc/nginx/nginx.conf test is successful
every now and then on WordPress I also get the error: internal server 500 error
and I don’t know if this is related to the problem I described above
I will probably also have problems with automatic certificate renewal if I do not resolve this situation
Well I have solved the problem. Fortunately I managed to find the error in the log and I understood what happened.
“certbot-auto” installs the certificates using the user: nobody. In my system there already existed by default a user “nobody” of the group “nogroup” but has no access to nginx. user nobody
So to be able to install the certificates I had to insert a line with the directive: user nobody nogroup; inside nginx.conf.
This allowed the installation of certificates. However, this procedure has changed the permissions in the directory: /jet/tmp/nginx/
where instead of: my_ssh_user:my_ssh_group I found: nobody:my_ssh_group and with this modified setting I received the internal server 500 error on the WordPress admin panel. So the problem is not related to WordPress but to the certificate installation procedure.
After installing the certificates I removed the user nobody nogroup directive; because during the restart of nginx, a warning came up saying that this directive is not supported and therefore will be ignored.
Restoring the permissions with chmod on the directory: /jet/tmp/nginx I fixed the problem and the internal server 500 error disappeared.
Here the link of the same discussion on letsencrypt. I still doubt that the error could recur when I have to renew the certificates.
A possible solution could be to assign the ssh user of the virtual instance that does not have root permissions, root permissions by adding it to the root group.
On virtual hosts, when an instance is launched, a user ssh is created.
This user, for security reasons, does not have root permissions.
Letsencrypt by default installs the certificates on the / etc / letsencrypt folder, which has root permissions (user: root group: root).
I had to install the certificates in the folder where the ssh user can have access (user: myssh_user group: myssh_group), ie su / jet / etc / letsencrypt
Could it be that the error was caused by this?
I assume I simply have to insert an entry into an nginx.conf file to resolve the error that is plaguing me (see below), but so far I haven’t had any luck figuring out the syntax. Any help would be appreciated.
I want to run nginx as a regular user while having installed it using homebrew as a user with administrative privileges. nginx is trying to write to the error.log file at /usr/local/var/log/nginx/error.log, which it cannot because my regular user lacks write privilege there.
Another wrinkle is coming from the fact that there are two nginx.conf files, a global and a local, and as far as I can tell they are both being read. They are in the default homebrew location /usr/local/etc/nginx/nginx.conf and my local project directory $BASE_DIR/nginx.conf.
Here is the error that is generated as nginx attempts to start up:
[WARN] No ENV file found
10:08:18 PM web.1 | DOCUMENT_ROOT changed to 'public/'
10:08:18 PM web.1 | Using Nginx server-level configuration include 'nginx.conf'
10:08:18 PM web.1 | 4 processes at 128MB memory limit.
10:08:18 PM web.1 | Starting php-fpm...
10:08:20 PM web.1 | Starting nginx...
10:08:20 PM web.1 | Application ready for connections on port 5000.
10:08:20 PM web.1 | nginx: [alert] could not open error log file: open() "/usr/local/var/log/nginx/error.log" failed (13: Permission denied)
10:08:20 PM web.1 | 2017/03/04 22:08:20 [emerg] 19557#0: "http" directive is duplicate in /usr/local/etc/nginx/nginx.conf:17
10:08:20 PM web.1 | Process exited unexpectedly: nginx
10:08:20 PM web.1 | Going down, terminating child processes...
[DONE] Killing all processes with signal null
10:08:20 PM web.1 Exited with exit code 1
Any help figuring out how to get nginx up and running so I can back to the development side of this project will be much appreciated.
The whole problem stems from trying to run nginx as my ordinary user self despite the fact that nginx was installed by my user self with administrative privileges. I was able to resolve both the errors shown here with the following commands executed as a user with administrative privileges:
sudo chmod a+w /usr/local/var/log/nginx/*.log
sudo chmod a+w /usr/local/var
sudo chmod a+w /usr/local/var/run
Note that the /usr/local/var directory appears to have been created by homebrew upon installing nginx and this machine is my laptop so I can’t see any reason not to open it up. You might have greater security concerns in other scenarios.
I admit that when I wrote this question I thought it was about moving the error.log file to another directory. Now I see that that is not a full solution, so instead the solution I present here is about giving ordinary users write privileges in the necessary directories.
The reason I changed my mind is that nginx can (and in this case does) generate errors before (or while) reading the nginx.conf files and needs to be able to report those errors to a log file. Modifying the nginx.conf file was never going to solve my problem. What woke me up to this issue was reading this post: How to turn off or specify the nginx error log location?
Did you try to set the log path to a custom location by editing nginx.conf?
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
Change path to somewhere use has write privileges.
I have OSX El Capitan. I installed Nginx-Full via homebrew. I am supposed to be able to start and stop services with
brew services Nginx-Full Start
I run that command and it seems to start no problem. I check the running services with
brew services list
That indicates that the Nginx-Full services is running. When i run
htop
to look at everything that is running Nginx does not show up and the server is not handling requests.
nginx is failing to launch because of an error, but brew-services is not communicating that to you.
Running it with sudo, as other users have suggested, is just masking the problem. If you just run nginx directly, you may see that there is actually a configuration or permissions issue that is causing nginx to abort. In my case, it was because it couldn't write to the error log:
nginx: [alert] could not open error log file: open() "/usr/local/var/log/nginx/error.log" failed (13: Permission denied)
2020/04/02 13:11:53 [warn] 19989#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/etc/nginx/nginx.conf:2
2020/04/02 13:11:53 [emerg] 19989#0: open() "/usr/local/var/log/nginx/error.log" failed (13: Permission denied)
The last error is causing nginx to fail to launch. You can make yourself the owner of the logs with:
sudo chown -R $(whoami) /usr/local/var/log/nginx/
This should cause subsequent config errors to be written to the error log, even if homebrew services is not reporting them in stderr/stdout for now.
I've opened an issue about this: https://github.com/Homebrew/homebrew-services/issues/215
The log path may not the same for everyone. You can check the path to log file by checking the config file /usr/local/etc/nginx/nginx.conf. You can find a line like:
error_log /Users/myusername/somepath/nginx.log;. Change the chown command above accordingly. If even this didn't solve the problem, you may have to do the same for any other log files specified in the server blocks in your nginx configuration
Try launching it with "sudo", even if the formula say
The default port has been set in /usr/local/etc/nginx/nginx.conf to 8080 so that
nginx can run without sudo.
sudo brew services Nginx-Full start
this worked for me:
sudo brew services start nginx
Running sudo nginx worked for me, it initially gave some error stating certain file in certain directory is missing, creating that file, and then another file is asked for to be created and then it runs properly.
I had similar problem, running it brew services start nginx used to show nginx running.
but brew services list used to show error.
running with sudo nginx solved my issue
I am getting this error in my nginx-error.log file:
2014/02/17 03:42:20 [crit] 5455#0: *1 connect() to unix:/tmp/uwsgi.sock failed (13: Permission denied) while connecting to upstream, client: xx.xx.x.xxx, server: localhost, request: "GET /users HTTP/1.1", upstream: "uwsgi://unix:/tmp/uwsgi.sock:", host: "EC2.amazonaws.com"
The browser also shows a 502 Bad Gateway Error. The output of a curl is the same, Bad Gateway html
I've tried to fix it by changing permissions for /tmp/uwsgi.sock to 777. That didn't work. I also added myself to the www-data group (a couple questions that looked similar suggested that). Also, no dice.
Here is my nginx.conf file:
nginx.conf
worker_processes 1;
worker_rlimit_nofile 8192;
events {
worker_connections 3000;
}
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;
#tcp_nopush on;
keepalive_timeout 65;
#gzip on;
include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;
}
I am running a Flask application with Nginsx and Uwsgi, just to be thorough in my explanation. If anyone has any ideas, I would really appreciate them.
EDIT
I have been asked to provide my uwsgi config file. So, I never personally wrote my nginx or my uwsgi file. I followed the guide here which sets everything up using ansible-playbook. The nginx.conf file was generated automatically, but there was nothing in /etc/uwsgi except a README file in both apps-enabled and apps-available folders. Do I need to create my own config file for uwsgi? I was under the impression that ansible took care of all of those things.
I believe that ansible-playbook figured out my uwsgi configuration since when I run this command
uwsgi -s /tmp/uwsgi.sock -w my_app:app
it starts up and outputs this:
*** Starting uWSGI 2.0.1 (64bit) on [Mon Feb 17 20:03:08 2014] ***
compiled with version: 4.7.3 on 10 February 2014 18:26:16
os: Linux-3.11.0-15-generic #25-Ubuntu SMP Thu Jan 30 17:22:01 UTC 2014
nodename: ip-10-9-xxx-xxx
machine: x86_64
clock source: unix
detected number of CPU cores: 1
current working directory: /home/username/Project
detected binary path: /usr/local/bin/uwsgi
!!! no internal routing support, rebuild with pcre support !!!
*** WARNING: you are running uWSGI without its master process manager ***
your processes number limit is 4548
your memory page size is 4096 bytes
detected max file descriptor number: 1024
lock engine: pthread robust mutexes
thunder lock: disabled (you can enable it with --thunder-lock)
uwsgi socket 0 bound to UNIX address /tmp/uwsgi.sock fd 3
Python version: 2.7.5+ (default, Sep 19 2013, 13:52:09) [GCC 4.8.1]
*** Python threads support is disabled. You can enable it with --enable-threads ***
Python main interpreter initialized at 0x1f60260
your server socket listen backlog is limited to 100 connections
your mercy for graceful operations on workers is 60 seconds
mapped 72760 bytes (71 KB) for 1 cores
*** Operational MODE: single process ***
WSGI app 0 (mountpoint='') ready in 3 seconds on interpreter 0x1f60260 pid: 26790 (default app)
*** uWSGI is running in multiple interpreter mode ***
spawned uWSGI worker 1 (and the only) (pid: 26790, cores: 1)
The permission issue occurs because uwsgi resets the ownership and permissions of /tmp/uwsgi.sock to 755 and the user running uwsgi every time uwsgi starts.
The correct way to solve the problem is to make uwsgi change the ownership and/or permission of /tmp/uwsgi.sock such that nginx can write to this socket. Therefore, there are three possible solutions.
Run uwsgi as the www-data user so that this user owns the socket file created by it.
uwsgi -s /tmp/uwsgi.sock -w my_app:app --uid www-data --gid www-data
Change the ownership of the socket file so that www-data owns it.
uwsgi -s /tmp/uwsgi.sock -w my_app:app --chown-socket=www-data:www-data
Change the permissions of the socket file, so that www-data can write to it.
uwsgi -s /tmp/uwsgi.sock -w my_app:app --chmod-socket=666
I prefer the first approach because it does not leave uwsgi running as root.
The first two commands need to be run as root user. The third command does not need to be run as root user.
The first command leaves uwsgi running as www-data user. The second and third commands leave uwsgi running as the actual user that ran the command.
The first and second command allow only www-data user to write to the socket. The third command allows any user to write to the socket.
I prefer the first approach because it does not leave uwsgi running as root user and it does not make the socket file world-writeable .
While the accepted solution is true there might also SELinux be blocking the access. If you did set the permissions correctly and still get permission denied messages try:
sudo setenforce Permissive
If it works then SELinux was at fault - or rather was working as expected! To add the permissions needed to nginx do:
# to see what permissions are needed.
sudo grep nginx /var/log/audit/audit.log | audit2allow
# to create a nginx.pp policy file
sudo grep nginx /var/log/audit/audit.log | audit2allow -M nginx
# to apply the new policy
sudo semodule -i nginx.pp
After that reset the SELinux Policy to Enforcing with:
sudo setenforce Enforcing
Anyone who lands here from the Googles and is trying to run Flask on AWS using the default Ubuntu image after installing nginx and still can't figure out what the problem is:
Nginx runs as user "www-data" by default, but the most common Flask WSGI tutorial from Digital Ocean has you use the logged in user for the systemd service file. Change the user that nginx is running as from "www-data" (which is the default) to "ubuntu" in /etc/nginx/nginx.conf if your Flask/wsgi user is "ubuntu" and everything will start working. You can do this with one line in a script:
sudo sed -i 's/user www-data;/user ubuntu;/' /etc/nginx/nginx.conf
Trying to make Flask and uwsgi run as www-data did not work off the bat, but making nginx run as ubuntu worked just fine since all I'm running with this instance is Flask anyhow.
You have to set these permissions (chmod/chown) in uWSGI configuration.
It is the chmod-socket and the chown-socket.
http://uwsgi-docs.readthedocs.org/en/latest/Options.html#chmod-socket
http://uwsgi-docs.readthedocs.org/en/latest/Options.html#chown-socket
Nginx connect to .sock failed (13:Permission denied) - 502 bad gateway
change the name of the user on the first line in /etc/nginx/nginx.conf file.
the default user is www-data and change it to root or your username
I know it's too late, but it might helps to other. I'll suggest to follow Running flask with virtualenv, uwsgi, and nginx very simple and sweet documentation.
Must activate your environment if you run your project in virtualenv.
here is the yolo.py
from config import application
if __name__ == "__main__":
application.run(host='127.0.0.1')
And create uwsgi.sock file in /tmp/ directory and leave it blank.
As #susanpal answer said "The permission issue occurs because uwsgi resets the ownership and permissions of /tmp/uwsgi.sock to 755 and the user running uwsgi every time uwsgi starts." it is correct.
So you have to give permission to sock file whenever uwsgi starts.
so now follow the below command
uwsgi -s /tmp/uwsgi.sock -w yolo:application -H /var/www/yolo/env --chmod-socket=666
A little different command from #susanpal.
And for persist connection, simply add "&" end of command
uwsgi -s /tmp/uwsgi.sock -w yolo:app -H /var/www/yolo/env --chmod-socket=666 &
In my case changing some php permission do the trick
sudo chown user:group -R /run/php
I hope this helps someone.
You should post both nginx and uwsgi configuration file for your application (the ones in /etc/nginx/sites-enabled/ and /etc/uwsgi/ - or wherever you put them).
Typically check that you have a line similar to the following one in your nginx app configuration:
uwsgi_pass unix:///tmp/uwsgi.sock;
and the same socket name in your uwsgi config file:
socket=/tmp/uwsgi.sock
I noticed when I test my nginx config using nginx -t, it gives me a warning:
nginx: [alert] could not open error log file: open() "/var/logs/nginx/error.log" failed (2: No such file or directory)
Which makes sense, since the log path for nginx is actually set up to be /var/log/nginx/ not /var/logs/nginx.
I scanned the entire nginx config directory and there is nothing there referencing /var/logs. I'm at a loss as to where this log location could be written?
Run this command in a terminal (note: capital V):
nginx -V
Do you find /var/logs there? Your nginx might be compiled with that default file location.
[EDIT]
I guess that some of your server blocks don't have the "error_log" directive. So nginx tries the default one for them. Note that by default the error_log is always on.
To fix this issue, you can add this line on the main block (the top level) such that all child blocks can inherit the setting:
error_log /var/log/nginx/error.log;
You can create these missed files:
cd /var/log/nginx/
sudo touch error.log
sudo touch access.log
sudo chmod 750 *.log
Then trigger nginx service:
sudo systemctl daemon-relod
sudo service nginx restart
[NOTE]:
You can also disable nginx logging:
disable nginx logging (1)
disable nginx logging (2)
You have a bad compiled default. Anything nginx notices before loading the config goes to the path for the error log defined at compile time. Recompile nginx with sane path or symlink log to logs if you can't.