I am using wordpress. I deploy Varnish Using Docker. This Is My default.vcl. What's Wrong With This Config?? Sometimes, Get Random 503 Error. I Exclude Wordpress Search Page Using RegEx. Also Get Random 503 Error On wordpress Search Page Too!!
varnishlog
https://www.dropbox.com/s/ruczg2i3h/log.txt
I am Using NGINX Backened..
Help Appreciated
Thanks
Your log output contains the following lines:
- Error out of workspace (bo)
- LostHeader Date:
- BerespHeader Server: Varnish
- VCL_call BACKEND_ERROR
- LostHeader Content-Type:
- LostHeader Retry-After:
Apparently you ran out of workspace memory because the size of the response.
The following parameters in your Docker config might cause that:
-p http_resp_hdr_len=65536 \
-p http_resp_size=98304 \
While increasing the size of individual headers and the total response size, the total memory consumption exceeds the workspace_backend value, which defaults to 64k.
Here's the documentation for http_resp_size:
$ varnishadm param.show http_resp_size
http_resp_size
Value is: 32k [bytes] (default)
Minimum is: 0.25k
Maximum number of bytes of HTTP backend response we will deal
with. This is a limit on all bytes up to the double blank line
which ends the HTTP response.
The memory for the response is allocated from the backend
workspace (param: workspace_backend) and this parameter limits
how much of that the response is allowed to take up.
As you can see, it affects workspace_backend. So here's the documentation for that:
$ varnishadm param.show workspace_backend
workspace_backend
Value is: 64k [bytes] (default)
Minimum is: 1k
Bytes of HTTP protocol workspace for backend HTTP req/resp. If
larger than 4k, use a multiple of 4k for VM efficiency.
NB: This parameter may take quite some time to take (full)
effect.
The solution is to increase the workspace_backend through a -p runtime parameter.
Nginx i am trying to use fast_cgi_cache, but it always returns empty reply from server during make some cache files.
If there aren't cache file then return always empty reply from server, and after a few seconds, then make some cache file(in tmp directory) and works corretly.
There are ways to remove empty reply with cache?
I solved this prbolem.
just added use_temp_path=off to nginx fastcgi cache path.
NGINX first writes files that are destined for the cache to a temporary storage
area, and the use_temp_path=off directive instructs NGINX to write them to the
same directories where they will be cached. We recommend that you set this
parameter to off to avoid unnecessary copying of data between file systems.
use_temp_path was introduced in NGINX version 1.7.10 and NGINX Plus R6.
https://www.nginx.com/blog/nginx-caching-guide/
I have a Mac running OS X Server 5.2 and due to company policy I need to force the server to use TLS 1.2 only and turn off TLS 1 and TLS 1.2.
I have edit the conf files in the /Library/Server/Web/Config/apache2/sites to SSLProtocol -all +TLSv1.2 but it still allows connection with tls1 and tls1.2
Does anybody know how to enforce TLS 1.2 connections only?
Thanks!
If that is an option, update your Server app. Starting with Server version 5.3, support for TLS 1 and TLS 1.1 has been disabled by default, so this solution would be simple and effective.
If you cannot update for some reason, you will have to remove these protocols in /Library/Server/Web/config/proxy/apache_serviceproxy.conf, and possibly in /library/server/web/config/apache2/httpd.conf. Also note that I have found that you have to restart macOS in order to restart the proxy server (it is not sufficient to issue sudo serveradmin stop/start web). This seems to be unnecessary - see my update below!
Update
It looks like there is a bug in Apache that can prevent the above setting from working as expected. It seems that, if there ciphers active that are not allowed any more in TLS 1.2, the older TLS versions remain available despite the setting in SSLProtocol. These older cipher suites need to be deactivated as well.
Therefore, use this recommended configuration:
SSLProtocol -all +TLSv1.2
SSLCipherSuite ALL:+HIGH:!ADH:!EXP:!SSLv2:!SSLv3:!MEDIUM:!LOW:!NULL:!aNULL
SSLHonorCipherOrder on
Besides that, it looks like it's enough to update /Library/Server/Web/config/proxy/apache_serviceproxy.conf, as SSL/TLS is handled by the proxy process only. To restart the proxy use
$ sudo launchctl unload -w /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.apple.serviceproxy.plist
$ sudo launchctl load -w /Applications/Server.app/Contents/ServerRoot/System/Library/LaunchDaemons/com.apple.serviceproxy.plist
A server restart is not required.
There might be multiple occurrences in the configuration. I replaced them all, which may or may not be necessary.
I use nginx along with fastcgi. I see a lot of the following errors in the error logs
readv() failed (104: Connection reset
by peer) while reading upstream and
recv() failed (104: Connection reset
by peer) while reading response header
from upstream
I don't see any problem using the application. Are these errors serious or how to get rid of them.
I was using php-fpm in the background and slow scripts were getting killed after a said timeout because it was configured that way. Thus, scripts taking longer than a specified time would get killed and nginx would report a recv or readv error as the connection is closed from the php-fpm engine/process.
Update:
Since nginx version 1.15.3 you can fix this by setting the keepalive_requests option of your upstream to the same number as your php-fpm's pm.max_requests:
upstream name {
...
keepalive_requests number;
...
}
Original answer:
If you are using nginx to connect to php-fpm, one possible cause can also be having nginx' fastcgi_keep_conn parameter set to on (especially if you have a low pm.max_requests setting in php-fpm):
http|server|location {
...
fastcgi_keep_conn on;
...
}
This may cause the described error every time a child process of php-fpm restarts (due to pm.max_requests being reached) while nginx is still connected to it. To test this, set pm.max_requests to a really low number (like 1) and see if you get even more of the above errors.
The fix is quite simple - just deactivate fastcgi_keep_conn:
fastcgi_keep_conn off;
Or remove the parameter completely (since the default value is off). This does mean your nginx will reconnect to php-fpm on every request, but the performance impact is negligible if you have both nginx and php-fpm on the same machine and connect via unix socket.
Regarding this error:
readv() failed (104: Connection reset by peer) while reading upstream and recv() failed (104: Connection reset by peer) while reading response header from upstream
there was 1 more case where I could still see this.
Quick set up overview:
CentOS 5.5
PHP with PHP-FPM 5.3.8 (compiled from scratch with some 3rd party
modules)
Nginx 1.0.5
After looking at the PHP-FPM error logs as well and enabling catch_workers_output = yes in the php-fpm pool config, I found the root cause in this case was actually the amfext module (PHP module for Flash).
There's a known bug and fix for this module that can be corrected by altering the amf.c file.
After fixing this PHP extension issue, the error above was no longer an issue.
This is a very vague error as it can mean a few things. The key is to look at all possible logs and figure it out.
In my case, which is probably somewhat unique, I had a working nginx + php / fastcgi config. I wanted to compile a new updated version of PHP with PHP-FPM and I did so. The reason was that I was working on a live server that couldn't afford downtime. So I had to upgrade and move to PHP-FPM as seamlessly as possible.
Therefore I had 2 instances of PHP.
1 directly talking with fastcgi (PHP 5.3.4) - using TCP / 127.0.0.1:9000 (PHP 5.3.4)
1 configured with PHP-FPM - using Unix socket - unix:/dir/to/socket-fpm
(PHP 5.3.8)
Once I started up PHP-FPM (PHP 5.3.8) on an nginx vhost using a socket connection instead of TCP I started getting this upstream error on any fastcgi page taking longer than x minutes whether they were using FPM or not. Typically it was pages doing large SELECTS in mysql that took ~2 min to load. Bad I know, but this is because of back end DB design.
What I did to fix it was add this in my vhost configuration:
fastcgi_read_timeout 5m;
Now this can be added in the nginx global fastcgi settings as well. It depends on your set up. http://wiki.nginx.org/HttpFcgiModule
Answer # 2.
Interestingly enough fastcgi_read_timeout 5m; fixed one vhost for me.
However I was still getting the error in another vhost, just by running phpinfo();
What fixed this for me was by copying over a default production php.ini file and adding the config I needed into it.
What I had was an old copy of my php.ini from the previous PHP install.
Once I put the default php.ini from 'shared' and just added in the extensions and config I needed, this solved my problem and no longer did I have nginx errors readv() and recv() failed.
I hope 1 of these 2 fixes helps someone.
Also it can be a very simple problem - there is an infinity cicle somewhere in your code, or an infinity trying to connect an external host on your page.
Some times this problem happen because of huge of requests. By default the pm.max_requests in php5-fpm maybe is 100 or below.
To solve it increase its value depend on the your site's requests, For example 500.
And after the you have to restart the service
sudo service php5-fpm restart
Others have mentioned the fastcgi_read_timeout parameter, which is located in the nginx.conf file:
http {
...
fastcgi_read_timeout 600s;
...
}
In addition to that, I also had to change the setting request_terminate_timeout in the file: /etc/php5/fpm/pool.d/www.conf
request_terminate_timeout = 0
Source of information (there are also a few other recommendations for changing php.ini parameters, which may be relevant in some cases): https://ma.ttias.be/nginx-and-php-fpm-upstream-timed-out-failed-110-connection-timed-out-or-reset-by-peer-while-reading/
There's an option to hide the version so it will display only nginx, but is there a way to hide that too so it will not show anything or change the header?
If you are using nginx to proxy a back-end application and want the back-end to advertise its own Server: header without nginx overwriting it, then you can go inside of your server {…} stanza and set:
proxy_pass_header Server;
That will convince nginx to leave that header alone and not rewrite the value set by the back-end.
The last update was a while ago, so here is what worked for me on Ubuntu:
sudo apt-get update
sudo apt-get install nginx-extras
Then add the following two lines to the http section of nginx.conf, which is usually located at /etc/nginx/nginx.conf:
sudo nano /etc/nginx/nginx.conf
server_tokens off; # removed pound sign
more_set_headers 'Server: Eff_You_Script_Kiddies!';
Also, don't forget to restart nginx with sudo service nginx restart.
Like Apache, this is a quick edit to the source and recompile. From Calomel.org:
The Server: string is the header which
is sent back to the client to tell
them what type of http server you are
running and possibly what version.
This string is used by places like
Alexia and Netcraft to collect
statistics about how many and of what
type of web server are live on the
Internet. To support the author and
statistics for Nginx we recommend
keeping this string as is. But, for
security you may not want people to
know what you are running and you can
change this in the source code. Edit
the source file
src/http/ngx_http_header_filter_module.c
at look at lines 48 and 49. You can
change the String to anything you
want.
## vi src/http/ngx_http_header_filter_module.c (lines 48 and 49)
static char ngx_http_server_string[] = "Server: MyDomain.com" CRLF;
static char ngx_http_server_full_string[] = "Server: MyDomain.com" CRLF;
March 2011 edit: Props to Flavius below for pointing out a new option, replacing Nginx's standard HttpHeadersModule with the forked HttpHeadersMoreModule. Recompiling the standard module is still the quick fix, and makes sense if you want to use the standard module and won't be changing the server string often. But if you want more than that, the HttpHeadersMoreModule is a strong project and lets you do all sorts of runtime black magic with your HTTP headers.
It’s very simple: Add these lines to server section:
server_tokens off;
more_set_headers 'Server: My Very Own Server';
Simple, edit /etc/nginx/nginx.conf and remove comment from
#server_tokens off;
Search for http section.
Install Nginx Extras
sudo apt-get update
sudo apt-get install nginx-extras
Server details can be removed from response by adding following two lines in the nginx.conf (under http section)
more_clear_headers Server;
server_tokens off;
There is a special module: http://wiki.nginx.org/NginxHttpHeadersMoreModule
This module allows you to add, set, or clear any output or input header that you specify.
This is an enhanced version of the standard headers module because it provides more utilities like resetting or clearing "builtin headers" like Content-Type, Content-Length, and Server.
It also allows you to specify an optional HTTP status code criteria using the -s option and an optional content type criteria using the -t option while modifying the output headers with the more_set_headers and more_clear_headers directives...
If you're okay with just changing the header to another string five letters or fewer, you can simply patch the binary.
sed -i 's/nginx\r/thing\r/' `which nginx`
Which, as a solution, has a few notable advantages. Namely, that you can allow your nginx versioning to be handled by the package manager (so, no compiling from source) even if nginx-extras isn't available for your distro, and you don't need to worry about any of the additional code of something like nginx-extras being vulnerable.
Of course, you'll also want to set the option server_tokens off, to hide the version number, or patch that format string as well.
I say "five letters or fewer" because of course you can always replace:
nginx\r\0
with
bob\r\0\r\0
leaving the last two bytes unchanged.
If you actually want more than five characters, you'll want to leave server_tokens on, and replace the (slightly longer) format string, although again there's an upper limit on that length imposed by the length of the format string - 1 (for the carriage return).
...If none of the above makes sense to you, or you've never patched a binary before, you may want to stay away from this approach, though.
According to nginx documentation it supports custom values or even the exclusion:
Syntax: server_tokens on | off | build | string;
but sadly only with a commercial subscription:
Additionally, as part of our commercial subscription, starting from
version 1.9.13 the signature on error pages and the “Server” response
header field value can be set explicitly using the string with
variables. An empty string disables the emission of the “Server”
field.
After I read Parthian Shot's answer, I dig into /usr/sbin/nginx binary file. Then I found out that the file contains these three lines.
Server: nginx/1.12.2
Server: nginx/1.12.2
Server: nginx
Basically first two of them are meant for server_tokens on; directive (Server version included).
Then I change the search criteria to match those lines within the binary file.
sed -i 's/Server: nginx/Server: thing/' `which nginx`
After I dig farther I found out that the error message produced by nginx is also included in this file.
<hr><center>nginx</center>
There are three of them, one without the version, two of them included the version. So I run the following command to replace nginx string within the error message.
sed -i 's/center>nginx/center>thing/' `which nginx`
The only way is to modify the file src/http/ngx_http_header_filter_module.c . I changed nginx on line 48 to a different string.
What you can do in the nginx config file is to set server_tokens to off. This will prevent nginx from printing the version number.
To check things out, try curl -I http://vurbu.com/ | grep Server
It should return
Server: Hai
I know the post is kinda old, but I have found a solution easy that works on Debian based distribution without compiling nginx from source.
First install nginx-extras package
sudo apt install nginx-extras
Then load the nginx http headers more module by editing nginx.conf and adding the following line inside the server block
load_module modules/ngx_http_headers_more_filter_module.so;
Once it's done you'll have access to both more_set_headers and more_clear_headers directives.
Expanding on Parthian Shot's answer, you can actually replace the whole header and not only the value as long as the total length is the same:
sed -i 's/Server: nginx/My-Header: hi/' `which nginx`
Nginx-extra package is deprecated now.
The following therefore did now work for me as i tried installing various packages
more_set_headers 'Server: My Very Own Server';
You can just do the following and no server or version information will be sent back
server_tokens '';
if you just want to remove the version number this works
server_tokens off;
Are you asking about the Server header value in the response? You can try changing that with an add_header directive, but I'm not sure if it'll work. http://wiki.codemongers.com/NginxHttpHeadersModule