I have used nginx built-in functionality clientbodyinfileonly to upload my files. And in order to know its performance. I used Apache ab to test it.
when I set concurrency level to 100,
ab -n 10000 -c 100 -p xxxx.rar -T application/x-www-form-urlencoded http://??????
the result was
Requests per second: 1477.95 [#/sec] (mean)
When the concurrency level is set to 120, I got 502 errors.
connect() to unix:/???/uwsgi.sock failed (11: Resource temporarily unavailable) while connecting to upstream
The nginx config as following shown [ visit Nginx direct file upload without passing them through backend for more details ]
location /upload {
auth_request /upload/authenticate;
client_body_temp_path /tmp/;
client_body_in_file_only on;
client_body_buffer_size 16M;
client_max_body_size 80M;
uwsgi_pass_request_headers on;
uwsgi_param X-FILE $request_body_file;
uwsgi_pass_request_body off;
uwsgi_pass unix://tmp/uwsgi.sock;
include uwsgi_params;
}
location = /upload/authenticate {
uwsgi_pass unix://tmp/uwsgi.sock;
uwsgi_pass_request_body off;
uwsgi_param CONTENT-LENGTH "";
uwsgi_param X-Original-URI $request_uri;
include uwsgi_params;
}
The way I took to around it was providing much more back-end servers together with upstream directive. And now the 502 went away. However, the TPS is still less than 1600 no matter how many uWSGi instances I added [ in practice, I used at most 8 instance, each with 2 processes ].
I also looked at the cpu, memory usage [ top, free, vmstat] during loading test. They are not too high.
Why the performance does not become better even if I used more uWSGI instances?? How could I know where to optimize?
The machine used is AWS EC2 with 2 CPUs ( 2.5GHz), 4G memory. And the file size used to upload is 30K.
Related
I want to set up an NGINX server which provides the following functionality:
When a request is made NGINX to get the page at /path/to/page, it fetches the page at /path/to/page.
If the upstream server is down or NGINX can't connect to it for some reason, NGINX returns a cached version of the page if it has one.
If the cached file is over 6 hours old, don't use it, just return a 502.
If the upstream server is available, never use the cache.
I have an NGINX config here which I think should work based on my understanding of the docs, but it doesn't and I can't see why. The problem is with point (4), this NGINX server returns the cached version of the file even if the upstream server is online.
daemon off;
error_log /dev/stdout info;
events {
}
http {
proxy_cache_path
"/home/jack/Code/NGINX Caching/Codebase/cache" # Cache path
keys_zone=cache:10m # Name of cacahe, max size for keys 10 megabytes
levels=1:2 # Don't store all cached files in a single directory
max_size=500m # Max size of cache
inactive=6h; # Cached file deleted if not used within six hours
proxy_cache_valid 6h;
proxy_cache_key "$request_method$request_uri";
access_log /dev/stdout;
server {
listen 8080;
location ~ ^/(.+)$ {
proxy_pass http://0.0.0.0:8000/$1;
proxy_cache cache;
proxy_cache_valid 6h;
proxy_buffering on;
proxy_cache_use_stale error timeout;
}
}
}
Replace the proxy_cache_path with a path to a directory on your machine, and run another webserver on your machine on port 8000. When I modify a file served by the server on port 8000, NGINX doesn't see the change until I erase the cache. The issue is with NGINX and not my client (Firefox), even if I turn off caching in the browser, NGINX returns a 200 with the old file contents.
Can you please check if these two directives might help you:
proxy_cache_revalidate:
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_revalidate
and
proxy_cache_use_stale: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_use_stale
There is a video from nginx.conf '17 online describing all the cool things you could achive with caching: https://www.youtube.com/watch?v=xZrOjmAkFC8. maybe this is also of interest for you.
So, it seems I misunderstood the NGINX proxy cache directives. The docs are quite confusing on this subject, so I'll lay it out point by point.
This official help page gives a decent overview of the various directives, however it makes no mention of something which as it turns out is a very important conceptual building block in understanding how NGINX caching works: the notion of a cached file being stale.
NGINX's default behavior is to always use the cache if it's there, rather than querying the upstream server. With this config, a minimal config to do caching, NGINX will query the upstream server the first time a page is accessed, and then use the cached version forever after that:
events {
}
http {
proxy_cache_path
/path/to/cache
keys_zone=my_cache:10m;
proxy_cache_key "$request_method$request_uri";
server {
listen 8080;
location ~ ^/(.+)$ {
proxy_pass http://0.0.0.0:8000/$1;
proxy_cache cache;
}
}
}
You can use the proxy_cache_valid directive to tell NGINX when a cached file should be considered "stale". For example, if we set proxy_cache_valid 5m, then 5 minutes after a cache file is created NGINX will stop serving it and querying the upstream server again on the next request. If the upstream is down, NGINX will return a 502. However, during those five minutes, NGINX will still use the cache even if the upstream server is available, so this is still not what we want.
NGINX has another directive, proxy_cache_use_stale, gives NGINX conditions under which it may use cached files even if they're stale. We can combine these together to get a server which caches pages, makes them stale immediately (or almost immediately), and then only uses them if the upstream is down:
events {
}
http {
proxy_cache_path
/path/to/cache
keys_zone=my_cache:10m;
proxy_cache_key "$request_method$request_uri";
server {
listen 8080;
location ~ ^/(.+)$ {
proxy_pass http://0.0.0.0:8000/$1;
proxy_cache cache;
proxy_cache_valid 1s;
proxy_cache_use_stale error timeout;
}
}
}
This config has almost the behavior we want, except that if the upstream server goes down for an extended period of time, NGINX will continue to use the cache indefinitely. As far as I know there is no way to tell NGINX to totally invalidate/clear a cached file after a given amount of time. Normally that's what proxy_cache_valid is for, but we're already using that for a different purpose, to make files stale after 1 second so they're only used when the upstream is down. We would need some next level after "stale" that means the file is completely invalidated, but I don't think that exists in NGINX.
So the simplest solution is to just clear the cache manually. It's sufficient to just delete all files in the cache directory (or its subdirectories) which were last modified more than 6 hours ago, or whatever you want the expiry time to be. On a Linux system, you can run this script every 5 minutes, for example:
find /path/to/cache -type f -mmin +360 -delete
I'm trying to save packed (gzip) html in Memcached and use in from nginx:
load html from memcached by memcached module
unpack by nginx gunzip module if packed
process ssi insertions by ssi module
return result to user
mostly, configuration works, except ssi step:
location / {
ssi on;
set $memcached_key "$uri?$args";
memcached_pass memcached.up;
memcached_gzip_flag 2; # net.spy.memcached use second byte for compression flag
default_type text/html;
charset utf-8;
gunzip on;
proxy_set_header Accept-Encoding "gzip";
error_page 404 405 400 500 502 503 504 = #fallback;
}
Looks like, nginx do ssi processing before unpacking by gunzip module.
In result HTML I see unresolved ssi instructions:
<!--# include virtual="/remote/body?argument=value" -->
No errors in the nginx log.
Have tried ssi_types * -- no effect
Any idea how to fix it?
nginx 1.10.3 (Ubuntu)
UPDATE
Have tried with one more upstream. Same result =(
In the log, I see, ssi filter applied after upstream request, but without detected includes.
upstream memcached {
server localhost:11211;
keepalive 100;
}
upstream unmemcached {
server localhost:21211;
keepalive 100;
}
server {
server_name dev.me;
ssi_silent_errors off;
error_log /var/log/nginx/error1.log debug; log_subrequest on;
location / {
ssi on;
ssi_types *;
proxy_pass http://unmemcached;
proxy_max_temp_file_size 0;
proxy_http_version 1.1;
proxy_set_header Connection "";
}
location #fallback {
ssi on;
proxy_pass http://proxy.site;
proxy_max_temp_file_size 0;
proxy_http_version 1.1;
proxy_set_header Connection "";
error_page 400 500 502 503 504 /offline.html;
}
}
server {
access_log on;
listen 21211;
server_name unmemcached;
error_log /var/log/nginx/error2.log debug; log_subrequest on;
location / {
set $memcached_key "$uri?$args";
memcached_pass memcached;
memcached_gzip_flag 2;
default_type text/html;
charset utf-8;
gunzip on;
proxy_set_header Accept-Encoding "gzip";
error_page 404 405 400 500 502 503 504 = #fallback;
}
location #fallback {
#ssi on;
proxy_pass http://proxy.site;
proxy_max_temp_file_size 0;
proxy_http_version 1.1;
proxy_set_header Connection "";
error_page 400 500 502 503 504 /offline.html;
}
}
I want to avoid solution with dynamic nginx modules if possible
There are basically two issues to consider — whether the order of the filter modules is appropriate, and whether gunzip works for your situation.
0. The order of gunzip/ssi/gzip.
A simple search for "nginx order of filter modules" reveals that the order is determined at the compile time based on the content of the auto/modules shell script:
http://www.evanmiller.org/nginx-modules-guide.html
Multiple filters can hook into each location, so that (for example) a response can be compressed and then chunked. The order of their execution is determined at compile-time. Filters have the classic "CHAIN OF RESPONSIBILITY" design pattern: one filter is called, does its work, and then calls the next filter, until the final filter is called, and Nginx finishes up the response.
https://allthingstechnical-rv.blogspot.de/2014/07/order-of-execution-of-nginx-filter.html
The order of filters is derived from the order of execution of nginx modules. The order of execution of nginx modules is implemented within the file auto/modules in the nginx source code.
A quick glance at auto/modules reveals that ssi is between gzip and gunzip, however, it's not immediately clear which way the modules get executed (top to bottom or bottom to top), so, the default might either be reasonable, or, you may need to switch the two (which wouldn't necessarily be supported, IMHO).
One hint here is the location of the http_not_modified filter, which is given as an example of the If-Modified-Since handling on EMiller's guide above; I would imagine that it has to go last, after all the other ones, and, if so, then, indeed, it seems that the order of gunzip/ssi/gzip is exactly the opposite of what you need.
1. Does gunzip work?
As per http://nginx.org/r/gunzip, the following text is present in the documentation for the filter:
Enables or disables decompression of gzipped responses for clients that lack gzip support.
It is not entirely clear whether that the above statement should be construed as the description of the module (e.g., the clients lacking gzip support is why you might want to use this module), or whether it's the description of the behaviour (e.g., whether the module determines by itself whether or not gzip would be supported by the client). The source code at src/http/modules/ngx_http_gunzip_filter_module.c appears to imply that it simply checks whether the Content-Encoding of the reply as-is is gzip, and proceed if so. However, the next sentence in the docs (after the above quoted one) does appear to indicate that it has some more interaction with the gzip module, so, perhaps something else is involved as well.
My guess here is that if you're testing with a browser, then the browser DOES support gzip, hence, it would be reasonable for gunzip to not engage, hence, the SSI module would never have anything valid to process. This is why I suggest you determine whether the gunzip works properly and/or differently between doing simple plain-text requests through curl versus those made by the browser with the Accept-Encoding that includes gzip.
Solution.
Depending on the outcome of the investigation as above, I would try to determine the order of the modules, and, if incorrect, there's a choice whether recompiling or double-proxying would be the solution.
Subsequently, if the problem is still not fixed, I would ensure that gunzip filter would unconditionally do the decompression of the data from memcached; I would imagine you may have to ignore or reset the Accept-Encoding headers or some such.
First of all, I already searched for 502 error in Stackoverflow. There are a lot a threads, but the difference this time is that the error appears without a pattern and it's not in Ubuntu.
Everything works perfectly, but about once a week my site shows: 502 Bad Gateway.
After this first error, every connection starts showing this message. Restarting MySQL + PHP-FPM + Nginx + Varnish doesn't work.
I have to clone this instance, and make another one, to get my site up again (It is hosted in Amazon EC2).
In Nginx log it shows these line again and again:
[error] 16773#0: *7034 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 127.0.0.1
There are nothing in MySQL or Varnish log. But in PHP-FPM it shows theses type of line:
WARNING: [pool www] child 18978, script '/var/www/mysite.com/index.php' (request: "GET /index.php") executing too slow (10.303579 sec), logging
WARNING: [pool www] child 18978, script '/var/www/mysite.com/index.php' (request: "GET /index.php") execution timed out (16.971086 sec), terminating
Inside PHP-FPM slowlog it was showing:
[pool www] pid 20401
script_filename = /var/www/mysite.com/index.php
w3_require_once() /var/www/mysite.com/wp-content/plugins/w3-total-cache/inc/define.php:1478
(Inside the file "define.php" at line number 1478, it has this line of code: require_once $path;)
I thought the problem was with W3 Total Cache plugin. So I removed W3 Total Cache.
About 5 days later it happened again with this error in PHP-FPM slow log:
script_filename = /var/www/mysite.com/index.php
wpcf7_load_modules() /var/www/mysite.com/wp-content/plugins/contact-form-7/includes/functions.php:283
(Inside the file "functions.php" at line number 283, it has this line of code: include_once $file;)
The other day, the first error occurred in another part:
script_filename = /var/www/mysite.com/wp-cron.php
curl_exec() /var/www/mysite.com/wp-includes/class-http.php:1510
And again a different part of code:
[pool www] pid 20509
script_filename = /var/www/mysite.com/index.php
mysql_query() /var/www/mysite.com/wp-includes/wp-db.php:1655
CPU, RAM ... everything is stable when this error occurs (less then 20% usage).
I tried everything, but nothing worked:
Moved to a better server (CPU and RAM)
Decreased timeout from Nginx, PHP-FPM, MySQL (my page loads quickly, so I decrease timeout to kill any outlier process)
Changed the number of PHP-FPM spare servers
Changed a lot of configuration from Nginx and PHP-FPM
I know that there is a bug with PHP-FPM and Ubuntu that could cause this error. But I don't think there is a bug with Amazon instances (Red Hat). (And I don't want to migrate from PHP-FPM to Socks because I've read that Socks don't works well under heavy load)
This was happening about every week since 5 months ago. I'm desperate.
I got to the point that I even put Nginx and PHP-FPM in Linux's crontab, to restart theses services every day. But it didn't work too.
Anyone has any suggestion where I can solve this problem? Anything will help!!
Server:
Amazon c3.large (2 core and 3.75GB RAM)
Linux Amazon Red Hat 4.8.2 64bits
PHP-FPM:
listen = 127.0.0.1:9000
listen.allowed_clients = 127.0.0.1
listen.mode = 0664
pm = ondemand
pm.max_children = 480
pm.start_servers = 140
pm.min_spare_servers =140
pm.max_spare_servers = 250
pm.max_requests = 50
request_terminate_timeout = 15s
request_slowlog_timeout = 10s
php_admin_flag[log_errors] = on
Nginx:
worker_processes 2;
events {
worker_connections 2048;
multi_accept on;
use epoll;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
access_log off;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
types_hash_max_size 2048;
server_tokens off;
client_max_body_size 8m;
reset_timedout_connection on;
index index.php index.html index.htm;
keepalive_timeout 1;
proxy_connect_timeout 30s;
proxy_send_timeout 30s;
proxy_read_timeout 30s;
fastcgi_send_timeout 30s;
fastcgi_read_timeout 30s;
listen 127.0.0.1:8080;
location ~ .php$ {
try_files $uri =404;
include fastcgi_params;
fastcgi_index index.php;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
fastcgi_keep_conn on;
fastcgi_pass 127.0.0.1:9000;
fastcgi_param HTTP_HOST $host;
}
}
I would start by tuning some configuration parameters.
PHP-FPM
I think that your pm values are somewhat off, a bit higher than I've normally seen configured on server around your specs... but you say that memory consumption it's normal so that's kind of weird.
Anyway... for pm.max_children = 480, considering that by default WordPress increases the memory limit to 40MB, you would end up using up to 18 gigs of memory, so you definitely would like to lower that.
Check the fourth part on this post for more info about that: http://www.if-not-true-then-false.com/2011/nginx-and-php-fpm-configuration-and-optimizing-tips-and-tricks/
If you're using... let's say 512MB for nginx, MySQL, Varnish and other services, you would have about 3328 MB for php-fpm... divided by 40 MBs per process, pm.max_children should be about 80... but even 80 it's very high.
It's probable that you can also lower the values of pm.start_servers, pm.min_spare_servers and pm.max_spare_servers. I prefer to keep them low and only increase them it's necessary
For pm.max_requests you should keep the default of 500 to avoid server respawns. I think it's only advisable to lower it if you suspect memory leaks.
Nginx
Change keepalive_timeout to 60 to make better use of keep alive.
Other than that, I think everything looks normal.
I had this issue with Ubuntu, but request_terminate_timeout on PHP-FPM and fastcgi_send_timeout + fastcgi_read_timeout were enough to get rid of it.
I hope you can fix it!
I am trying to build a docker-registry server from source (not as a container) on Ubuntu 14.04.1. I was able to get most of the way there using the instructions found on digitalocean.
I am able to curl http://localhost:5000 and https://user:password#localhost:8000 with no problems
When I try to open a web browser to see hopefully more than just that, that is when the issues seem to happen.
Here is my docker-registry file in /etc/nginx/sites-available/:
# For versions of Nginx > 1.3.9 that include chunked transfer encoding support
# Replace with appropriate values where necessary
upstream docker-registry {
server 192.168.x.x:5000;
}
server {
listen 8000;
server_name docker-registry;
ssl on;
ssl_certificate /etc/nginx/ssl/docker-registry.crt;
ssl_certificate_key /etc/nginx/ssl/docker-registry.key;
proxy_set_header Host $http_host; # required for Docker client sake
X-Real-IP $remote_addr; # pass on real client IP
client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads
# required to avoid HTTP 411: see Issue #1486 (https://github.com/dotcloud/docker/issues/1486)
chunked_transfer_encoding on;
location / {
# let Nginx know about our auth file
auth_basic "Restricted";
auth_basic_user_file docker-registry.htpasswd;
proxy_pass http://docker-registry;
}
location /_ping {
auth_basic off;
proxy_pass http://docker-registry;
}
location /v1/_ping {
auth_basic off;
proxy_pass http://docker-registry;
}
}
I have my docker registry stored locally in /var/docker-registry and ensured that it was readable by the www-data user. Why can I not see my images on the web browser?
If I tag an image and push it to my repository it works, I can see it in the web browser:
https://192.168.x.x:8000/v1/repositories/ubuntu-test/tags/latest
I see the following:
"5ba9dab47459d81c0037ca3836a368a4f8ce5050505ce89720e1fb8839ea048a"
When I try to get to:
https://192.168.x.x:8000/v1
Or:
https://192.168.x.x:8000/v1/repositories
Or:
https://192.168.x.x:8000/v1/images
I get a "not found" error
How would I be able to see everything in my /var/docker-registry folder (which is where these are stored....and yes, they are owned by the www-data user) through the web interface?
This is by design. Not only is there no reason one would implement the entire url path, but there are severe security implications with implementing it.
I'm assuming you don't have much experience with web programming. There is no directory '/v1/repositories'... etc. Instead, there is a program (in this case either Python or Ruby) that is listening for the url path and has logic built-in to determine what to do.
i.e. if url = /v1/_ping: return 'ok'
I'm running nginx/ruby-on-rails and I have a simple multipart form to upload files.
Everything works fine until I decide to restrict the maximum size of files I want uploaded.
To do that, I set the nginx client_max_body_size to 1m (1MB) and expect a HTTP 413 (Request Entity Too Large) status in response when that rule breaks.
The problem is that when I upload a 1.2 MB file, instead of displaying the HTTP 413 error page, the browser hangs a bit and then dies with a "Connection was reset while the page was loading" message.
I've tried just about every option there is that nginx offers, nothing seems to work. Does anyone have any ideas about this?
Here's my nginx.conf:
worker_processes 1;
timer_resolution 1000ms;
events {
worker_connections 1024;
}
http {
passenger_root /the_passenger_root;
passenger_ruby /the_ruby;
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 80;
server_name www.x.com;
client_max_body_size 1M;
passenger_use_global_queue on;
root /the_root;
passenger_enabled on;
error_page 404 /404.html;
error_page 413 /413.html;
}
}
Thanks.
**Edit**
Environment/UA: Windows XP/Firefox 3.6.13
nginx "fails fast" when the client informs it that it's going to send a body larger than the client_max_body_size by sending a 413 response and closing the connection.
Most clients don't read responses until the entire request body is sent. Because nginx closes the connection, the client sends data to the closed socket, causing a TCP RST.
If your HTTP client supports it, the best way to handle this is to send an Expect: 100-Continue header. Nginx supports this correctly as of 1.2.7, and will reply with a 413 Request Entity Too Large response rather than 100 Continue if Content-Length exceeds the maximum body size.
Does your upload die at the very end? 99% before crashing? Client body and buffers are key because nginx must buffer incoming data. The body configs (data of the request body) specify how nginx handles the bulk flow of binary data from multi-part-form clients into your app's logic.
The clean setting frees up memory and consumption limits by instructing nginx to store incoming buffer in a file and then clean this file later from disk by deleting it.
Set body_in_file_only to clean and adjust buffers for the client_max_body_size. The original question's config already had sendfile on, increase timeouts too. I use the settings below to fix this, appropriate across your local config, server, & http contexts.
client_body_in_file_only clean;
client_body_buffer_size 32K;
client_max_body_size 300M;
sendfile on;
send_timeout 300s;
From the documentation:
It is necessary to keep in mind that the browsers do not know how to correctly show this error.
I suspect this is what's happening, if you inspect the HTTP to-and-fro using tools such as Firebug or Live HTTP Headers (both Firefox extensions) you'll be able to see what's really going on.