unix:///var/run/supervisor.sock refused connection during restarting supervisor - nginx

When I created a new .conf file inside /etc/supervisor/conf.d/ and tried to start this program it was showing some errors (fatal error) and restarting frequently by itself. Then I ran the command sudo service supervisor restart but now the supervisor also stopped and couldn't be restarted it. During solving my error the nginx server also got stuck also.
After spending a vast time I recovered it Alhamdulillah and writing the solution in the answer section.
Don't trust the solution entirely for your problem. Your problem may belong to another issues as well.

Sometime Supervisor can show the below horrible error when you restart the
service (by the command sudo service supervisor restart):
unix:///var/run/supervisor.sock refused connection
Try to diagnosis the problem with the command supervisord. You can also run journalctl -xe.
Problems and Solutions:
When you write a new .conf file to inside the /etc/supervisor/conf.d directory which contains some statements that are generating error.
Like, you write some statements that will run a script. That script contains some statements that runsGunicorn to deploy a python web apps. In the script you wrote a statement to bind an unix socket. But the mentioned directory where the unix socket will be created doesn't give permission to create the .sock file there. This can lead the permission error.
The demo gunicorn command is below:
SOCKFILE = /home/shamim/python_project/another_directroy/gunicorn.sock
gunicorn ${DJANGO_WSGI_MODULE}:application \
--name $NAME \
--bind=unix:$SOCKFILE
If the another_directory doesn't give the permission to create a .sock file inside it then an error can be occurred. So give it enough permission to create something here from outside. Or, Bind IP and port instead unix socket (like 127.0.0.1:ANY_PORT). Be sure first the port is not used by another application.
Sometimes the error can be occurred if any directory path is used inside .conf file but actually that directory doesn't exist at all.
Now run the command supervisord.
If the error persists after fixing the above issues and now shows a error like -
another program is already listening on a port that one of our HTTP servers is configured to use
then run the below command to fix this issue:
sudo unlink /var/run/supervisor.sock
If the command above does not work you should check run unlink the file at /tmp/supervisor.sock
Keep in mind that the nginx server can also show some errors and fail to
restart (or start) if any .conf file contains some statement where a socket
is used but actually the socket file doesn't exist or doesn't have enough permission to be executed.
Example: If you write the below code in any nginx file config:
upstream surveyapp_payment_stripe {
server unix:/home/shamim/python_project/another_directroy/gunicorn.sock fail_timeout=0 weight=5 max_fails=3;
}
If the above socket doesn't exist or not have enough permission then some error may be occurred.
Nginx can also show error if any directory path is used here but not exists at all. To run nginx at this time quickly just delete the .conf file or edit it's extension (make another another extension type other than .conf).
Hopefully this explanation will help someone in future.

Related

Unable to reload Nginx context

I am newbie to Nginx and adding upstream configuration dynamically through a java program, after that I am trying to reload the context and getting the below error. I tried to reload context on command prompt even and getting sane error over there. I opened the command prompt using admin rights.
C:\nginx-1.16.1>nginx -s reload
nginx: [error] OpenEvent("Global\ngx_reload_6512") failed (5: Access is denied)
Please share if some one have already come across this problem and found some solution.
Greatly appreciated.

Update .py file on nginx, gunicorn, supervisor

I don't know why this is so difficult, but everytime I update a file in my flask application I have to restart gunicorn so that the file updates on the server. I am mostly a front-end developer and don't play with servers enough to remember these things, and I have to spend hours google searching various phrases to find the right commands. This time I can't seam to find anything, and the file I created to save these things has conveniently disappeared.
My server:
Ubuntu 18.04
nginx
gunicorn
supervisor
I am updating a .py file. I placed the updated version on the server using ftp. I'm logged into the server, using ssh, through a git bash shell. sudo systemctl gunicorn restart give me the error Failed to restart gunicorn.service: Unit gunicorn.service not found.. Rereading and restarting supervisor does not do the trick, and neither does restarting nginx. Is there not a simple command to apply updates? I'm use to using servers on general hosting sites, and updating a file via ftp just works. I was really enjoying learning flask up until this point, but now I regret it. I keep thinking that there has to be some kind of simple trick to make such a simple thing go smoothly, but I'm at the end of my rope trying to figure this out. Any suggestions?
I finally found it.
sudo supervisorctl stop app_name
sudo supervisorctl start app_name

how to monitor a directory and include new files with tail -f in Centos(for shiny-server logs in Docker)

Due to the need to direct shiny-server logs to stdout so that "docker logs" (and monitoring utilities relying on it) can see them i'm trying to do some kind of :
tail -f <logs_directory>/*
Which works as needed when no new files are added to the directory, the problem is shiny-server dynamically creates files in this directory which we need to automatically consider.
I found other users have solved this via the xtail package, the problem is i'm using Centos and xtail is not available for centos.
The question is , is there any "clean" way of doing this via standard tail command without needing xtail ? or maybe there exists an equivalent package to xtail for centos?
You will probably find it easier to use the docker run -v option to mount a host directory into the container and collect logs there. Then you can use any tool you want that collects log files out of a directory (logstash is popular but far from the only option) to collect those log files.
This also avoids the problem of having to both run your program and a log collector inside your container; you can just run the service as the main container process, and not have to do gymnastics with tail and supervisord and whatever else to try to keep everything running.

Apache Karaf root instance already running

I am new to apache karaf.
When I start the apache karaf by executing bat file on my windows machine. It keeps giving me below error
C:\karaf\apache-karaf-4.0.5\bin>karaf.bat
There is a Root instance already running with name root and pid 1320
I checked for this pid, but there is no process with that pid.
I also checked the process in the task manager, but no process with name kafar / root is running.
I figured it out.
It seems that If we close the command window (without hitting ^D) then the pid stored does not gets cleared
Next time when it starts up, karaft reads the old pid from the properties file. Ofcourse the old process does not exist anymore, but since pid was not cleared from property file, apache karaf does not know it.
Solution :
Open KARAF_HOME\instances\instance.properties file
Set item.0.pid=0
Alternatively in development deleting KARAF_HOME\instances\instance.properties file would also work. On next startup it will create the instance.properties file

sudoers - Google Compute Engine - no access to root

I have a Google Compute Engine VM instance with a Asterisk Server running on it. I get this message when I try to run sudo:
sudo: parse error in /etc/sudoers near line 21
sudo: no valid sudoers sources found, quitting
sudo: unable to initialize policy plugin
Is there a password for root so I can try to change it there? Any suggestions on this?
It looks like you have manually edited the /etc/sudoers file so while you would normally have sudo access, due to the parse error, you won't be able to do this directly.
Here's how to fix this situation.
1. Save the current boot disk
go to to the instance view in Developers Console
find your VM instance and click on its name; you should now be looking at a URL such as
https://console.cloud.google.com/project/[PROJECT]/compute/instancesDetail/zones/[ZONE]/instances/[VM-NAME]
stop the instance
detach the boot disk from the instance
2. Fix the /etc/sudoers on the boot disk
create a new VM instance with its own boot disk; you should have sudo access here
attach the disk saved above as a separate persistent disk
mount the disk you just attached
fix the /etc/sudoers file on the disk
unmount the second disk
detach the second disk from the VM
delete the new VM instance (let it delete its boot disk, you won't need it)
3. Restore the original VM instance
re-attach the boot disk to the original VM
restart the original VM with its original boot disk, with fixed config
How to avoid this in the future
Always use the command visudo rather just any text editor directly to edit the /etc/sudoers file which will validate the contents of the file prior to saving it.
I ran into this issue as well and had the same issue Nakilon was reporting when trying the gcloud workaround.
What we ended up doing was configure a startup script that removed the broken sudoers file.
So in your metadata put something like:
#/bin/sh
rm "/etc/sudoers.d/broken-config-file"
echo "ok" > /tmp/ok.log
https://cloud.google.com/compute/docs/startupscript
As you probably figured out this requires the /etc/sudoers file to be fixed. As nobody has root access to the instance, you will not be able to do this from inside the instance.
The best way to solve this is to edit the disk from another instance. The basic steps to do this are:
Take a snapshot of your disk as a backup (!)
Shutdown your instance, taking care not to delete the boot disk.
Start a new "debugger" instance from one of the stock GCE images.
Attach the old boot disk to the new instance.
In the debugger instance, mount the disk.
In the debugger instance, fix the sudoers file on the mounted disk.
In the debugger instance, unmount the disk
Shutdown the debugger instance.
Create a new instance with the same specs as your original instance using the fixed disk as the boot disk.
The new disk will then have the fixed sudoers file.
Since i bumped into this issue too, if you have another instance or any place where you can run with gcloud privileges, you can run:
gcloud compute --project "<project id>" ssh --zone "europe-west1-b" "<servername>"
I ran this on a server which had gcloud as root, so you login to the other box as root too! Then fix your issue. (if you don't have a box, just spin a micro up with the correct gcloud privileges) saves the hassle of disk stuff etc.
As mentioned in above comments, I am getting the same error like below in gcp VM.
sudo: parse error in /etc/sudoers near line 21
sudo: no valid sudoers sources found, quitting
sudo: unable to initialize policy plugin
To solve this
I have ssh to another vm and become root then I ran gcloud ssh command to our main vm (where you are getting the sudo error.)
gcloud compute --project "<project id>" ssh --zone "europe-west1-b "<servername>"
And BOOM!, now are login as root in the VM.
Now you can access/change the /etc/sudoers file accordingly.
I found this hack better than recreating vm/disks.
Hope this helps to someone!
It is possible to connect to a VM as root from your developers console Google Cloud Shell. Make sure the VM is running, start the shell and use this command:
gcloud compute ssh root#<instance-name> --zone <zone> [--project <project-id>]
where instance-name is found in the Compute Engine VM Instances screen. project-id is optional but required if you are connecting to an instance in a different project from the project where you started the shell.
You can then fix this and other issues that may prevent you from using sudo.
I got a Permission denied error when trying to ssh to the problem instance via gcloud. Using a startup script as mentioned above by #Jorick works. Instructions for it are here. You will have to stop and restart the VM instance for the startup script to get executed. I modified the script slightly:
rm -f /etc/sudoers.d/google_sudoers >& /tmp/startup.log
After the restart, launch an SSH session from the cloud console and check that you are able to view the file contents (with sudo more /etc/sudoers.d/google_sudoers for example). If that works your problem has been solved.

Resources