I have following configuration with
worker_process 4;
But I noticed that it always hits only 1 worker.
I am testing on a local Centos VM. I am doing curl http call on specific port and added a file with 1000 curl requests and ran them from multiple terminal windows.
But see alll of them hit only 1 worker. Is there a way that I can have atleast more than 1 worker started. Can someone please share their knowledge on this.
https://blog.cloudflare.com/the-sad-state-of-linux-socket-balancing/
In the epoll-and-accept the load balancing algorithm differs: Linux seems to choose the last added process, a LIFO-like behavior. The process added to the waiting queue most recently will get the new connection. This behavior causes the busiest process, the one that only just went back to event loop, to receive the majority of the new connections. Therefore, the busiest worker is likely to get most of the load.
I have spring boot application with /health endpoint accessible deployed in AWS ECS Fargate. Sometimes the container is stopped with Task failed container health checks message. Sometimes happens once daily, sometimes once a week, maybe depends on the load. This is the healthcheck command specified in the Task Definition:
CMD-SHELL,curl -f http://localhost/actuator/health || exit 1
My question is how to troubleshoot what AWS receive when health-check is failed.
In case anyone else lands here because of failing container health checks (not the same as ELB health checks), AWS provides some basic advice:
Check that the command works from inside the container. In my case I had not installed curl in the container image, but when I tested it from outside the container it worked fine, which fooled me into thinking it was working.
Check the task logs in CloudWatch
If the checks are only failing sometimes (especially under load), you can try increasing the timeout, but also check the task metrics (memory and CPU usage). Garbage collection can cause the task to pause, and if all the vCPUs are busy handling other requests, the health check may be delayed, so you may need to allocate more memory and/or vCPUs to the task.
Thank #John Velonis,
I don't have enough reputation for commenting on your answer, so I post that in a different answer
For my case, the ecs container keeps getting UNKNOWN status from the ecs cluster. But I can access the healthcheck successfully. when I read this post, and check my base image which is node:14.19.1-alpine3.14, it doesn't have curl command.
so I have to install that in the Dockerfile
RUN apk --no-cache add curl
We are using Gunicorn with Nginx. After every time we restart gunicorn, the CPU usage took by Gunicorn keeps on increasing gradually. This increases from 0.5% to around 85% in a matter of 3-4 days. On restarting gunicorn, it comes down to 0.5%.
Please suggest what can cause this issue and how to go forward to debug and fix this.
Check workers configuration. Try use the following: cores * 2 -1
Check your application, seems that your application is blocking / freezing threads. Add timeout to all api calls, database queries, etc.
You can add an APM software to analyze your application, for example datadog.
I have an EC2 t2.medium instance i-0bf4623a779064e0a with a WordPress installation which keeps crashing (can't be accessed via http or SSH). It seems whenever CPU utilization gets to about 25% or more (which I would think isn't very much), the server crashes. I have an alert setup to restart the server whenever Network Out is <=50,000 bytes for 5 minutes and tonight it's had to restart 10 times. It has been doing this nearly everyday for weeks. Here is a screenshot of the monitoring http://i.imgur.com/zQQ4oiy.png
What can I do to stop this crashing? Can I do some sort of server config optimization? I hope I do not need a larger instance, since I am already paying quite a bit for AWS and previously using a $10/mo shared hosting which rarely went down.
My top-level question is, how can I get Puma to stop failing. But that is really made up of lots of smaller questions. I will number and bold each of them, to try to make this question answerable.
I am hosting a Rails application on an EC2 instance that is a t2.nano. This is admittedly, a very small box--but I don't expect my website to receive any traffic. I configured everything successfully with Nginx and Puma using Capistrano and Capistrano Puma. Everything was great, until one day I went to my website and saw the Nginx 504 message.
I opened the Nginx error log and saw that it could not connect to Puma:
connect() to unix:/home/deploy/myapp/shared/tmp/sockets/puma.sock failed (111: Connection refused) while connecting to upstream, client: xxx.xxx.xxx.xxx, server: localhost, request: "GET / HTTP/1.0", upstream: "http://unix:/home/deploy/myapp/shared/tmp/sockets/puma.sock:/500.html", host: "myapp.com"
Debugging this, I learned that Puma had stopped running. That is why Nginx could not connect to it. I think there are two problems here: the first, is that Puma should not stop running. The server is tiny, but there is no traffic. the second, is that when Puma does fail, it should restart gracefully. However, I am just focusing on the first issue for now. Because if Puma is constantly restarting, it seems reasonable that sometimes it kills the process in a harsh way.
To debug this, I opened htop. Sure enough, the machine was running without any memory to spare. This makes sense--I am running a database, rails app, webserver, and memcache on one tiny machine. It keeps running out of memory and killing Puma.
I looked into the Puma configuration I had set up with Capistrano. In config/deploy.rb I had these lines--
set :puma_threads, [0, 8]
set :puma_workers, 0
I read all about puma_workers and puma_threads. I also learned that Nginx has its own workers. Puma processes are very expensive. What makes Puma cool is that it is properly muli-threaded--so the independent processes are awesome. It sounds like each worker has its own set of threads--so if there are 4 workers with 8 threads, there will be 32 processes. But in my case, I want to use very little memory. 2 processes sound good to me. 1. Is my understanding of workers and threads correct?
I updated my config/deploy.rb file and deployed, with 0 puma_workers and min=0, max=2 threads.
It appears the configuration for Nginx lives here: /etc/nginx/nginx.conf. And the configuration for Puma lives here: /home/deploy/myapp/shared/puma.rb. I would have expected my updates in config/deploy.rb to have had Capistano edit the config files. No luck--my min, max threads was still set to 0,8. 2. Is it correct to try and update these values through config/deploy.rb when using Capistano?
Also--I opened the nginx.conf and saw worker_processes 4;. 3. Was this set to four when I installed Nginx or did Capistano set this default?
I opened htop and sure enough I had lots of Puma processes. Therefore, I edited my config files manually and restarted Puma and Nginx.
I changed the number of Nginx workers from 4 to 1. Looking in htop, this worked. I now only had 1 Nginx worker. However, the Nginx workers were never very expensive (compared to the Puma threads). So I don't think this matters much.
However, there were still more than 2 Puma threads--there were 6. On a lark, I changed the minimum number of threads from 0 to 1--thinking 0 isn't a possible number so maybe it's setting a default. This increased the number of Puma processes to 9. I also tried changing the number of puma_workers to 1, for the same reason, and the number of processes increased. 4. What does it mean to have 0 threads and/or workers?
I then tried to kill one of the puma processes manually (sudo kill xxxxx), and then all of the Puma processes died.
5. What do I have to do to have just 2 puma processes?
As you can see, my understanding of Puma is not great and the lines between what Puma vs Nginx vs Capistano touches is not clear. Any help is greatly appreciated. I haven't been able to find great resources regarding this issue.
This is what I've learned--
if Puma stops working, make sure you have enough memory to handle to number of workers and threads that you specified. each Puma process is pretty expensive.
if you set 0 workers, Puma will not run in cluster mode. it is recommended to run MRI using cluster mode.
threads are set per cluster. if you have 2 works and 0,8 threads that means you will have two works and each will have between 1 and 8 threads.
Puma uses processes in addition to the threads. Puma has a PID for the parent process. if you are using cluster mode, it has a PID to manager the clusters. if you are using cluster mode, it also has a PID for each cluster. then, there are a fixed number of PIDs to run other tasks (per cluster). without cluster mode, there are 5 fixed PIDs. with cluster mode, there are 7 fixed PIDs.
this is all to say--if you see more processes than you expect, this is why. also--when you add a new worker you add a significant amount of expensive processes. make sure you have the space.
i have a small app, and things seem to be working nicely with 1 worker and min=1, max=4 threads. having a max of 8 threads looks to be what kept killing puma for me.
To answer my original questions--
Yes, the explanation above of workers and threads is correct.
capistrano-puma appears to only set puma config with the first deploy.
I think the nginx config is created when nginx is installed.
0 workers means you are running puma without cluster mode. It is impossible to have 0 threads. I believe 0,8 is the same as 1,8.
Puma needs to run processes in addition to the threads you request. It is impossible to have puma running with only 2 or 3 PIDs. These processes run addition tasks.
A suspect for Puma hangs
The thing with Puma is that it's the only mainstream project that encourages the use of threading in MRI Ruby (well, anyway, Heroku encourages).
This is why we sometimes see statements from people working on Puma about how people think that Puma has various kinds of issues, while the problem is elsewhere, and it is, and it affects only Puma :P
"We" have discovered and fixed in the past some very freaky and nasty Ruby GC issues on heavy duty use of threads in Ruby MRI with some freaky corner cases (remember http://blog.skylight.io/hunting-for-leaks-in-ruby/) and who is to say this is not the last of such freaky issues that people attribute to Puma?
Try disabling threading for a while, see how it goes, and let us know, maybe the rabbit lies there, again
Docs explaining threads vs clustered mode vs workers
Thread pool docs: https://github.com/puma/puma#thread-pool
Clustered mode docs: https://github.com/puma/puma#clustered-mode
puma.rb options: https://github.com/puma/puma/blob/master/examples/config.rb
Under Thread pool the docs explain how to set up the number of worker threads. Remember, Puma is/was primarily a JRuby thing and MRI support & forking was added only later as an afterthought, the ordering of configuration entries in the docs (how to set up threading before how to set up forking) is a consequence of this.
The docs state:
Puma utilizes a dynamic thread pool which you can modify. You can set the minimum and maximum number of threads that are available in the pool with the -t (or --threads) flag:
Puma 2 offers clustered mode, allowing you to use forked processes to handle multiple incoming requests concurrently, in addition to threads already provided.
Meaning, Puma will always thread, it's what it does, if you tell it to do 0/1 thread, it will do 1 thread so it can serve requests.
Additionally, if you set the number of workers (processes) to > 1, Puma will run in "Clustered mode" which means it will fork and each fork will thread,
i.e. -w 3 -t4:4 will result with 3 processes running 4 threads each, allowing you to concurrently server 12 requests.
Puma docs don't specify which and how many processes Puma will use for it's internals, but just an educated guess is that at the very minimum it needs to run all of the workers + 1 master process to manage them, deliver data to them, start them, stop them, channel their logs etc.