My server setup works for a Rails 4.0 app, but fails on a 4.2 app. I get this error:
An error occurred.
Sorry, the page you are looking for is currently unavailable.
Please try again later.
If you are the system administrator of this resource then you should check the error log for details.
NGINX config:
server {
listen 80;
server_name localhost;
passenger_enabled on;
rails_env production;
root /home/deploy/myapp/current/public;
}
NGINX error.log:
2014/10/13 16:17:06 [error] 9261#0: *9 upstream prematurely closed connection while reading response header from upstream, client: ***.***.***.***, server: localhost, request: "GET / H$
Rails production.log:
W, [2014-10-13T16:11:57.305892 #10891] WARN -- : Warning. Error encountered while saving cache a4b17298d22d34199795f642dc5b96ec8d58cc6c/orders.css.scssc: can't dump anonymous class #<$
W, [2014-10-13T16:11:57.314170 #10891] WARN -- : Warning. Error encountered while saving cache a4b17298d22d34199795f642dc5b96ec8d58cc6c/pages.css.scssc: can't dump anonymous class #<C$
W, [2014-10-13T16:11:57.319744 #10891] WARN -- : Warning. Error encountered while saving cache a4b17298d22d34199795f642dc5b96ec8d58cc6c/registrations.css.scssc: can't dump anonymous c$
If I manually put in an index.html file in the public directory, I can see that. But it fails when I want to go to the root path of the app. Any ideas?
OK so this is a bit embarrassing. To further troubleshoot my application, I started in production mode on my local machine, and when I loaded the app I got the following error on the web page:
Missing `secret_key_base` for 'production' environment, set this value in `config/secrets.yml`
So that was it. I guess I missed this new security feature by jumping straight from Rails 4.0 to 4.2 Not sure why it didn't show in the logs, but at least I found it eventually.
Related
Linux installation of Phusion Passenger(R) 6.0.14 on nginx/1.14.1. I'm not able to load a site (on a simplified nginx.conf with the conf.d/passenger.conf and conf.d/site1.conf includes. That error:
2022/08/03 19:24:34 [alert] 55601#0: *3 Error opening '/home/user3/sites/Passengerfile.json' for reading: Permission denied (errno=13);
This error means that the Nginx worker process (PID 55601, running as UID 992) does not have permission to access this file.
Please read this page to learn how to fix this problem: https://www.phusionpassenger.com/library/admin/nginx/troubleshooting/?a=upon-accessing-the-web-app-nginx-reports-a-permission-denied-error;
Extra info, client: 192.168.1.4, server: domain1.com, request: "GET / HTTP/1.1", host: "server_f.local"
I don't even know what to ask, other than how can I get this to work? That file doesn't exist. I've restarted nginx many times, and this is the only feedback that I get. I've checked over that page, which tells me to look at the error log it's reported in already. The host and server are correct, and I am on my LAN.
I have Lapis installed and working on Debian 9. I have two beginner issues I need to solve running the simple init setup that serves static content from /static/ and the Lua app from all other requests:
I put an index.html file in a directory called static which is in the same directory I start Lapis from, but attempting to access via IP/static/index.html results in a 403 access error.
I cannot set the server to use port 80 because the address is already in use by nginx (confirmed with netstat). There is a "Welcome to OpenResty!" page served there, with the Lapis app only being served on other set ports.
I'd appreciate any advice that would solve these issues. Also any pointers towards documentation that fully describes the relationship between nginx, OpenResty and Lapis would be very helpful.
Thanks to help from LordRyan on the moonscript IRC the issues are solved.
The 403 issue was a permission error caused by the Lapis app directory being at root level.
Inability to use port 80 was caused by an nginx instance running in the background which was discovered using...
ps aux | grep nginx root
... shutting it down solved the issue and gave the Lapis app control.
I am running Gitlab on Debian using the package from the Repository. Most of the time Gitlab is running very fast, but after longer idle times Gitlab is very slow or even times out (error 502). One time I also had a timeout on a remote git access (could not reproduce the issue - timeout on the internal API).
In my setup the the Debian machine is behind another nginx proxy which also serves some other services just fine. I did the gitlab-cli checks and everything seems fine.
In the error log of my reverse proxy I only see connection timeouts:
[error] 8643#0: *4139 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.1.1.10, server: gitlab.mydomain.tld, request: "GET / HTTP/1.1", upstream: "http://{SERVER-IP}:80/", host: "gitlab.mydomain.tld"
I can see some errors in my unicorn_stderr.log
E, [2016-03-30T19:40:20.183991 #783] ERROR -- : worker=1 PID:16798 timeout (61s > 60s), killing
E, [2016-03-30T19:40:20.194969 #783] ERROR -- : reaped #<Process::Status: pid 16798 SIGKILL (signal 9)> worker=1
I, [2016-03-30T19:40:20.197554 #16871] INFO -- : worker=1 spawned pid=16871
I, [2016-03-30T19:40:20.197909 #16871] INFO -- : worker=1 ready
E, [2016-03-30T20:08:42.911429 #783] ERROR -- : worker=0 PID:16866 timeout (61s > 60s), killing
E, [2016-03-30T20:08:43.191151 #783] ERROR -- : reaped #<Process::Status: pid 16866 SIGKILL (signal 9)> worker=0
I, [2016-03-30T20:08:43.758363 #18728] INFO -- : worker=0 spawned pid=18728
I, [2016-03-30T20:08:44.108244 #18728] INFO -- : worker=0 ready
What I am a bit curious about is the fact that there are no errors in the log of the nginx delivered with gitlab.
Some more system information:
#sudo gitlab-rake gitlab:env:info
System information
System: Debian 8.3
Current User: git
Using RVM: no
Ruby Version: 2.1.8p440
Gem Version: 2.5.1
Bundler Version:1.10.6
Rake Version: 10.5.0
Sidekiq Version:4.0.1
GitLab information
Version: 8.5.0
Revision: a513e09
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: http://gitlab.mydomain.tld
HTTP Clone URL: http://gitlab.mydomain.tld/some-group/some-project.git
SSH Clone URL: git#gitlab.mydomain.tld:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 2.6.10
Repositories: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git
Edit:
My nginx config on the "external" reverse proxy looks like this:
server {
listen 443;
ssl on;
server_name gitlab.mydomain.tld;
access_log /var/log/nginx/gitlab.mydomain.tld.access.log;
error_log /var/log/nginx/gitlab.mydomain.tld.error.log;
ssl_certificate /etc/nginx/ssl/gitlab.mydomain.tld_unified.crt;
ssl_certificate_key /etc/nginx/ssl/mydomain.tld.key;
location / {
proxy_pass http://gitlab:80;
proxy_redirect default;
proxy_set_header Host $http_host;
proxy_set_header X_FORWARDED_PROTO "https";
satisfy any;
}
}
Edit2:
I took the suggested answer into account and also considered this source: https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/requirements.md
I assigned 2GB RAM to the VM now, and also added one additional unicorn worker.
Edit3:
The problem seems to be solved by adding more memory and using 3 unicorn workers.
Jan,
I have a similar setup although our box is dedicated to GITlab. Without knowing the specs of your server (GITLAB likes memory) and the load on that box I would suggest the following diagnostics:
Does your upstream nginx use identical parameters as the gitlab nginx configuration? They have tweaked a number of things including timeouts.
What kind of request result in time outs? Some operations (like generating diffs) can take some time to render.
If you run the requests via SSH do you also experience time outs?
Have you checked global logs in /var/log?
FYI: I had to enlarge my small GitLab installation to have 4GB RAM not to throw OOM errors
Now I think, I'd better go with gogs or other alternative.
On Ubuntu with REE 1.8.7, with nginx 0.8.53 manually compiled with the
Passenger 3 module doc installed as per the docs, I get an error in
the nginx error log when nginx starts:
Unable to start the Phusion Passenger watchdog because it encountered
the following error during startup: Unable to start the Phusion
Passenger logging agent: an error occurred while sending startup
arguments: Broken pipe (32) (-1: Unknown error 18446744073709551615)
I assume this means nginx never starts properly. When I try and view
my site after this, I get a 403.
2010/10/22 00:56:47 [error] 13874#0: *1813942 directory index of "/var/
rails/myapp/current/public/" is forbidden, client: 125.197.115.16,
server: new.myapp.fm, request: "GET / HTTP/1.1", host: "new.myapp.fm"
I expect the first is probably causing the second. I haven't had any
luck googling this. Can anyone provide insight or help me
troubleshoot?
I got this after upgrading to Passenger 3.0.1 as I had not changed the version of Passenger specified by passenger_root in /opt/nginx/conf/nginx.conf. You might need to change the line that looks like this to specify the correct version:
passenger_root /usr/lib/ruby/gems/1.8/gems/passenger-3.0.1;
You get told to do this when you install nginx. Hope that helps.
I ran into this problem too, and it was because passenger couldn't find ruby:
tail -f /var/log/nginx/error.log
2011/12/20 00:54:45 [alert] 30300#0: Unable to start the Phusion Passenger watchdog because it encountered the following error during startup: Unable to start the Phusion Passenger helper agent: it seems to have crashed during startup for an unknown reason, with exit code 1 (-1: Unknown error)
* Passenger ERROR (ext/common/ApplicationPool/../SpawnManager.h:220):
Could not start the spawn server: ruby: No such file or directory (2)
Even though ruby was properly listed within the system PATH, it didn't seem to work right until I added:
passenger_ruby = /usr/local/bin/ruby;
(the location of my ruby executable) into the http section of my nginx config.
The problem was that PassengerLoggingAgent wasn't installed when I did 'gem install passenger'. I installed it as per this discussion (by grabbing it from the standalone zip) and everything is fine now: http://groups.google.com/group/phusion-passenger/browse_thread/thread/323dc1c7b0e5ee11
I use nginx along with fastcgi. I see a lot of the following errors in the error logs
readv() failed (104: Connection reset
by peer) while reading upstream and
recv() failed (104: Connection reset
by peer) while reading response header
from upstream
I don't see any problem using the application. Are these errors serious or how to get rid of them.
I was using php-fpm in the background and slow scripts were getting killed after a said timeout because it was configured that way. Thus, scripts taking longer than a specified time would get killed and nginx would report a recv or readv error as the connection is closed from the php-fpm engine/process.
Update:
Since nginx version 1.15.3 you can fix this by setting the keepalive_requests option of your upstream to the same number as your php-fpm's pm.max_requests:
upstream name {
...
keepalive_requests number;
...
}
Original answer:
If you are using nginx to connect to php-fpm, one possible cause can also be having nginx' fastcgi_keep_conn parameter set to on (especially if you have a low pm.max_requests setting in php-fpm):
http|server|location {
...
fastcgi_keep_conn on;
...
}
This may cause the described error every time a child process of php-fpm restarts (due to pm.max_requests being reached) while nginx is still connected to it. To test this, set pm.max_requests to a really low number (like 1) and see if you get even more of the above errors.
The fix is quite simple - just deactivate fastcgi_keep_conn:
fastcgi_keep_conn off;
Or remove the parameter completely (since the default value is off). This does mean your nginx will reconnect to php-fpm on every request, but the performance impact is negligible if you have both nginx and php-fpm on the same machine and connect via unix socket.
Regarding this error:
readv() failed (104: Connection reset by peer) while reading upstream and recv() failed (104: Connection reset by peer) while reading response header from upstream
there was 1 more case where I could still see this.
Quick set up overview:
CentOS 5.5
PHP with PHP-FPM 5.3.8 (compiled from scratch with some 3rd party
modules)
Nginx 1.0.5
After looking at the PHP-FPM error logs as well and enabling catch_workers_output = yes in the php-fpm pool config, I found the root cause in this case was actually the amfext module (PHP module for Flash).
There's a known bug and fix for this module that can be corrected by altering the amf.c file.
After fixing this PHP extension issue, the error above was no longer an issue.
This is a very vague error as it can mean a few things. The key is to look at all possible logs and figure it out.
In my case, which is probably somewhat unique, I had a working nginx + php / fastcgi config. I wanted to compile a new updated version of PHP with PHP-FPM and I did so. The reason was that I was working on a live server that couldn't afford downtime. So I had to upgrade and move to PHP-FPM as seamlessly as possible.
Therefore I had 2 instances of PHP.
1 directly talking with fastcgi (PHP 5.3.4) - using TCP / 127.0.0.1:9000 (PHP 5.3.4)
1 configured with PHP-FPM - using Unix socket - unix:/dir/to/socket-fpm
(PHP 5.3.8)
Once I started up PHP-FPM (PHP 5.3.8) on an nginx vhost using a socket connection instead of TCP I started getting this upstream error on any fastcgi page taking longer than x minutes whether they were using FPM or not. Typically it was pages doing large SELECTS in mysql that took ~2 min to load. Bad I know, but this is because of back end DB design.
What I did to fix it was add this in my vhost configuration:
fastcgi_read_timeout 5m;
Now this can be added in the nginx global fastcgi settings as well. It depends on your set up. http://wiki.nginx.org/HttpFcgiModule
Answer # 2.
Interestingly enough fastcgi_read_timeout 5m; fixed one vhost for me.
However I was still getting the error in another vhost, just by running phpinfo();
What fixed this for me was by copying over a default production php.ini file and adding the config I needed into it.
What I had was an old copy of my php.ini from the previous PHP install.
Once I put the default php.ini from 'shared' and just added in the extensions and config I needed, this solved my problem and no longer did I have nginx errors readv() and recv() failed.
I hope 1 of these 2 fixes helps someone.
Also it can be a very simple problem - there is an infinity cicle somewhere in your code, or an infinity trying to connect an external host on your page.
Some times this problem happen because of huge of requests. By default the pm.max_requests in php5-fpm maybe is 100 or below.
To solve it increase its value depend on the your site's requests, For example 500.
And after the you have to restart the service
sudo service php5-fpm restart
Others have mentioned the fastcgi_read_timeout parameter, which is located in the nginx.conf file:
http {
...
fastcgi_read_timeout 600s;
...
}
In addition to that, I also had to change the setting request_terminate_timeout in the file: /etc/php5/fpm/pool.d/www.conf
request_terminate_timeout = 0
Source of information (there are also a few other recommendations for changing php.ini parameters, which may be relevant in some cases): https://ma.ttias.be/nginx-and-php-fpm-upstream-timed-out-failed-110-connection-timed-out-or-reset-by-peer-while-reading/