proxy_pass isn't working when SELinux is enabled, why? - nginx

I'm having an application listening on port 8081 and Nginx running on port 8080. The proxy pass statement looks like:
$ cat /var/etc/opt/lj/output/services/abc.servicemanager.conf
location /api/abc.servicemanager/1.0 { proxy_pass http://localhost:8081;}
In nginx.conf, I include this file as:
include /etc/nginx/conf.d/services/*.conf;
The /etc/nginx/conf.d/service is a symlink:
# ll /etc/nginx/conf.d/
lrwxrwxrwx. 1 root root 39 Dec 10 00:19 services -> ../../../var/etc/opt/lj/output/services
This is a CentOS 7.0 SELinux Enabled system. If I setenforce 0, and make it Permissive, I don't see any issues. So the file is in right place and no issues with paths. If SELinux is enforcing, I see the following in audit log:
type=AVC msg=audit(1418348761.372:100930): avc: denied { getattr } for pid=3936 comm="nginx" path="/var/etc/opt/lj/output/services/abc.servicemanager.conf" dev="xvda1" ino=11063393 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:var_t:s0 tclass=file
I want to know how to enable Nginx to find the conf file without having to disable SELinux.

Worth noting for beginners in SELinux that if your proxied service is running on 8080, you can use the command below without compiling a policy.
$ sudo setsebool httpd_can_network_connect 1 -P

Read about audit2allow and used it to create a policy to allow access to the denied requests for Nginx.
Step 1 involves running audit2allow targeting nginxlocalconf:
$ sudo grep nginx /var/log/audit/audit.log | \
grep denied | audit2allow -m nginxlocalconf > nginxlocalconf.te
Step 2, review results:
$ cat nginxlocalconf.te
module nginxlocalconf 1.0;
require {
type httpd_t;
type var_t;
type transproxy_port_t;
class tcp_socket name_connect;
class file { read getattr open };
}
#============= httpd_t ==============
#!!!! This avc can be allowed using the boolean 'httpd_can_network_connect'
allow httpd_t transproxy_port_t:tcp_socket name_connect;
allow httpd_t var_t:file { read getattr open };
Review steps to activate:
$ sudo grep nginx /var/log/audit/audit.log | grep denied | \
audit2allow -M nginxlocalconf
******************** IMPORTANT ***********************
To make this policy package active, execute:
semodule -i nginxlocalconf.pp
Step 3, active:
$ sudo semodule -i nginxlocalconf.pp

If you have another port or custom port allow it:
Show allow port in http:
semanage port -l | grep http
This is output in my localhost:
http_cache_port_t tcp 8080, 8118, 8123, 10001-10010
http_cache_port_t udp 3130
http_port_t tcp 80, 81, 443, 488, 8008, 8009, 8443, 9000
pegasus_http_port_t tcp 5988
pegasus_https_port_t tcp 5989
And allow 8081:
semanage port -a -t http_port_t -p tcp 8081

Always prefer changing types to creating custom policies. In this case, Nginx will serve files with the httpd_sys_content_t type. Assuming your files are located in /var/www:
semanage fcontext -a -t httpd_sys_content_t /var/www/*
restorecon -R -v /var/www

Related

Octavia: Trying to delete immutable loadbalancer

I have a loadbalancer (see status below) that I want to delete. I already deleted the instances in its pool. Full disclosure: This is on a Devstack which I rebooted, and where I recreated the lb-mgmt-network routing manually. I may have overlooked a detail after the reboot. The loadbalancer worked before the reboot.
The first step to delete the loadbalancer is to delete its pool members. This fails as follows:
$ alias olb='openstack loadbalancer'
$ olb member delete website-pool 08f55..
Load Balancer 1ff... is immutable and cannot be updated. (HTTP 409)
What can I do to make it mutable?
Below, see the loadbalancer's status after recreating the o-hm0 route and restarting the amphora. Its provisioning status is ERROR, but according to the API, this should enable me to delete it:
$ olb status show kubelb
{
"loadbalancer": {
"id": "1ff7682b-3989-444d-a1a8-6c91aac69c45",
"name": "kubelb",
"operating_status": "ONLINE",
"provisioning_status": "ERROR",
"listeners": [
{
"id": "d3c3eb7f-345f-4ded-a7f8-7d97e3af0fd4",
"name": "weblistener",
"operating_status": "ONLINE",
"provisioning_status": "ACTIVE",
"pools": [
{
"id": "9b0875e0-7d16-4ebc-9e8d-d1b90d4264a6",
"name": "website-pool",
"provisioning_status": "ACTIVE",
"operating_status": "ONLINE",
"members": [
{
"id": "08f55bba-260a-4b83-ad6d-f9d6b44f0e2c",
"name": "",
"operating_status": "NO_MONITOR",
"provisioning_status": "ACTIVE",
"address": "172.16.0.21",
"protocol_port": 80
},
{
"id": "f7665e90-dad0-480e-8ef4-65e0a042b9fa",
"name": "",
"operating_status": "NO_MONITOR",
"provisioning_status": "ACTIVE",
"address": "172.16.0.22",
"protocol_port": 80
}
]
}
]
}
]
}
}
When you have a load balancer in ERROR state you have two options:
Delete the load balancer using the cascade delete option (--cascade on the cli).
Use the failover API to tell Octavia to repair the load balancer once your cloud is fixed.
In Octavia, operating status is a measured/observed status. If they don't go ONLINE it is likely that there is a network configuration issue with the lb-mgmt-net and the health heartbeat messages (UDP 5555) are not making it back to the health manager controller.
That said, devstack is not setup to work after a reboot. Specifically neutron and the network interfaces will be in an improper state. As you have found, you can manually reconfigure those and usually get things working again.
If I understand documentation and source code right, a loadbalancer in provisioning status ERROR can be deleted but not modified. Unfortunately, it can only be deleted after its pools and listeners have been deleted, which would modify the loadbalancer. Looks like a chicken and egg problem to me. I "solved" this by recreating the cloud from scratch. I guess I could also have cleaned up the database.
An analysis of the stack.sh log file revealed that a few additional steps were needed to make the Devstack cloud reboot-proof. To make Octavia ready:
Create /var/run/octavia, owned by the stack user
Ensure o-hm0 is up
Give o-hm0 the correct MAC and IP addresses, both found in the details of Neutron port octavia-health-manager-standalone-listen-port
add netfilter rules for traffic coming from o-hm0
At this point, I feel I can reboot Devstack and still have functioning load balancers. Strangely, all load balancers' operating_status (as well as their listeners' and pools' operating_status) is OFFLINE. However, that doesn't prevent them from working. I have not found out how to make that ONLINE.
In case anybody is interested, below is the script I use after rebooting Devstack. In addition, I also changed the Netplan configuration so that br-ex gets the server's IP address (further below).
restore-devstack script:
$ cat restore-devstack
source ~/devstack/openrc admin admin
if losetup -a | grep -q /opt/stack/data/stack-volumes
then echo loop devices are already set up
else
sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-default-backing-file
sudo losetup -f --show --direct-io=on /opt/stack/data/stack-volumes-lvmdriver-1-backing-file
echo restarting Cinder Volume service
sudo systemctl restart devstack#c-vol
fi
sudo lvs
openstack volume service list
echo
echo recreating /var/run/octavia
sudo mkdir /var/run/octavia
sudo chown stack /var/run/octavia
echo
echo setting up the o-hm0 interface
if ip l show o-hm0 | grep -q 'state DOWN'
then sudo ip l set o-hm0 up
else echo o-hm0 interface is not DOWN
fi
HEALTH_IP=$(openstack port show octavia-health-manager-standalone-listen-port -c fixed_ips -f yaml | grep ip_address | cut -d' ' -f3)
echo health monitor IP is $HEALTH_IP
if ip a show dev o-hm0 | grep -q $HEALTH_IP
then echo o-hm0 interface has IP address
else sudo ip a add ${HEALTH_IP}/24 dev o-hm0
fi
HEALTH_MAC=$(openstack port show octavia-health-manager-standalone-listen-port -c mac_address -f value)
echo health monitor MAC is $HEALTH_MAC
sudo ip link set dev o-hm0 address $HEALTH_MAC
echo o-hm0 MAC address set to $HEALTH_MAC
echo route to loadbalancer network:
ip r show 192.168.0.0/24
echo
echo fix netfilter for Octavia
sudo iptables -A INPUT -i o-hm0 -p udp -m udp --dport 20514 -j ACCEPT
sudo iptables -A INPUT -i o-hm0 -p udp -m udp --dport 10514 -j ACCEPT
sudo iptables -A INPUT -i o-hm0 -p udp -m udp --dport 5555 -j ACCEPT
echo fix netfilter for Magnum
sudo iptables -A INPUT -d 192.168.1.200/32 -p tcp -m tcp --dport 443 -j ACCEPT
sudo iptables -A INPUT -d 192.168.1.200/32 -p tcp -m tcp --dport 80 -j ACCEPT
sudo iptables -A INPUT -d 192.168.1.200/32 -p tcp -m tcp --dport 9511 -j ACCEPT
Netplan config:
$ cat /etc/netplan/00-installer-config.yaml
# This is the network config written by 'subiquity'
network:
ethernets:
enp1s0:
dhcp4: no
br-ex:
addresses: [192.168.1.200/24]
nameservers: { addresses: [192.168.1.16,1.1.1.1] }
gateway4: 192.168.1.1
version: 2

uwsgi nginx selinux on centos 7 audit2allow done, enforcing selinux still blocking access to the socket

I'm trying to add a flask uwsgi behind nginx with SELinux activated but no luck so far.
I've followed all suggestions to pipe denied contexts from audit.log to audit2allow generating the module and then semodule -i nginx.pp as answered in https://stackoverflow.com/a/26336047/2172543 but still, if I leave setenforce 1 nginx is being blocked to write to the socket.
I've also changed permissions to all folders in /path/to/socket.sock, changed umask of the socket to 666, did everything where there was a solution for my problem but I'm still getting 502 with setenforce 1.
Switching setenforce 0 "solves" the problem. But I want to leave SELinux activated and I have no more clues into how to further investigate the issue.
Any thoughts?
format log file
yum install setroubleshoot -y
sealert -a /var/log/audit/audit.log > /var/log/audit/audit.format.log
allow this access for now by executing
ausearch -c 'nginx' --raw | audit2allow -M my-nginx
semodule -i my-nginx.pp

Munin with Nginx and FastCGI

I'm trying to configure Munin over Nginx. To do so, I need to manually start the cgi process with this command:
spawn-fcgi -s fastcgi-graph.sock -U nginx -u nginx -g nginx /var/lib/munin/cgi-tmp/munin-cgi-graph -n
Unfortunately, I get the following error:
spawn-fcgi: exec failed: Permission denied
The permissions on the socket file is as followed:
srwxr-xr-x 1 nginx nginx 0 May 2 14:08 fastcgi-graph.sock
Can anyone point me out on what it wrong with that configuration?
Thanks.

(ubuntu) nginx: [emerg] bind() to 0.0.0.0:80 failed (13: permission denied)

I need help figuring out the root cause of this permission denied error. What permissions does nginx need? Why is it so complicated?
the socket API bind() to a port less than 1024, such as 80 as your title mentioned, need root access.
here is "Bind to ports less than 1024 without root access"
and another easier way is to run nginx as root.
If you use a port bigger than 1024 with root privilege, but still got this problem, that's may be caused by SELinux:
Check this port, say 8024, in segange port
sudo semanage port -l | grep http_port_t
If 8024 doesn't exist in the port list, add it into segange port
sudo semanage port -a -t http_port_t -p tcp 8024
###update in 2017.12.22
Sometimes your SELinux is disabled, you need to enforcing it first. Check the status of SELinux by
$ sestatus
More steps can read this wonderful article: https://www.digitalocean.com/community/tutorials/an-introduction-to-selinux-on-centos-7-part-1-basic-concepts
If see this msg after run "nginx -t", you dont have premission run as root "sudo nginx -t"
nginx needs root access. Just use
sudo nginx
next step along with your password
The best solution would be:
1) add user to sudoers ( my user is prod)
usermod -aG sudo prod
2) inside circus ( process manager ) append sudo before nginx executable, mine looks like this:
[watcher:nginx]
cmd = sudo /usr/sbin/nginx
args = -c /home/t/Projects/x_b_11/etc/nginx.conf -p /home/t/Projects/x_b_11
3) and finaly add line into file /etc/sudoers ( my user is prod). This line avoids error (sudo: no tty present and no askpass program specified). Probably need to restart session ( reboot). Enjoy.
prod ALL = NOPASSWD: /usr/sbin/nginx
Ubuntu uses AppArmor and not SELinux. The responses pointing to SELinux may not be that relevant to the OP.
For the others that Googled this: I also encountered this issue on a SELinux-enabled CentOS 7 machine. nginx would not bind port 80 and gave me error 13: permission denied despite having already run
setcap 'CAP_NET_BIND_SERVICE=+ep' /usr/sbin/nginx to allow the service to bind the port with a non-root user.
Temporarily setting SELinux to Permissive (sudo setenforce Permissive) allowed nginx to start. I then ran audit2allow -a which gave me
#============= httpd_t ==============
#!!!! This avc can be allowed using the boolean 'httpd_can_network_connect'
allow httpd_t ntop_port_t:tcp_socket name_connect;
Which meant the solution was to also run:
sudo setsebool -P httpd_can_network_connect on
After which you can set SELinux back to Enforcing (sudo setenforce Enforcing) and restart everything to verify.

Nginx Invalid PID number

I issued a nginx -s stop and after that I got this error when trying to reload it.
[error]: invalid PID number "" in "/var/run/nginx.pid"
That /var/run/nginx/pid file is empty atm.
What do I need to do to fix it?
nginx -s reload is only used to tell a running nginx process to reload its config. After a stop, you don't have a running nginx process to send a signal to. Just run nginx (possibly with a -c /path/to/config/file)
in my case I solved this by starting the service.
sudo /etc/init.d/nginx start
The command above will start the service in Debian/Ubuntu. It will issue an error if there is any problem (like Apache listening in the same port)
After that nginx -s reload will work like a charm
This will clear out the issue on ubuntu 16.04 and above
sudo service nginx stop
you may need to remove the pid file nginx.pid whose location may be defined in file /etc/nginx/nginx.conf look for line like
cat /etc/nginx/nginx.conf | grep pid # see if pid file is defined
this line may live in file /etc/nginx/nginx.conf
pid /run/nginx.pid; # in file /etc/nginx/nginx.conf
if pid file does exist then remove it now
ls -la /var/run/nginx/pid # this file may live elsewhere
ls -la /run/nginx.pid # on Ubuntu 16.04+
after the pid file has been removed lets launch nginx
sudo service nginx start
ps -eaf|grep nginx # confirm its running
sudo nginx -t && sudo nginx -s reload # confirm config is OK
# typical output
# nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
# nginx: configuration file /etc/nginx/nginx.conf test is successful
sudo service nginx stop # issue stop
ps -eaf|grep nginx # confirm it actually stopped
now sanity has been restored and you are free to launch at will
In the latest version(1.2.0) that I downloaded there is no "-s start" option, it will say
nginx: invalid option: "-s start"
You can start nginx by
sudo /etc/nginx/sbin/nginx
The server will be started and then there wont be any Invalid pid number errors.
To avoid downtime with restarting nginx,
ps aux | grep nginx
PID of nginx master process
echo PID > /var/run/nginx.pid
nginx -s reload
In my case nginx was stopped (crashed I assume). Solved the issue by:
service nginx status
nginx stop/waiting
service nginx start
nginx start/running, process 3535
Then nginx -s reload worked like a charm.
I am using nginx/1.8.0 on trusty.
This happens if the nginx process was stopped manually or was killed.
Check if the process is still running:
sudo lsof -nP -iTCP:<port> | grep LISTEN
I am on mac, and I reinstall the nginx with:
brew reinstall nginx
Then start the service using brew:
brew services start nginx
On CentOS 7 I done it with this:
sudo systemctl start nginx
#Then check all things are OK
sudo systemctl status -l nginx
For anyone who still has issues, in my case, there was an apache2 server that was running.
You can try debugging what went wrong in your nginx machine by executing this command -
systemctl status nginx
This gave me an insight that the port was already in us by apache2 server.
so you can do sudo service apache2 stop and then do sudo service nginx start.
Docker Alpine users should use
nginx
by using that nginx will be start there is no error by
nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
then reload it by
nginx -s reload

Resources