I'm using nginx as a cache for googleapis.com. Currently all responses are cached for 5m:
proxy_cache_path /var/cache/nginx/xxx_cache keys_zone=xxx_cache:10m;
server {
location ~ /blog/ {
proxy_pass https://www.googleapis.com/blogger/v3/blogs/;
proxy_cache xxx_cache;
proxy_cache_lock on;
proxy_cache_valid 5m;
}
I'd like to change this interval depending on the request_uri. Defining a $proxy_cache_valid variable via the map directive, and using it for proxy_cache_valid fails with invalid time value "$proxy_cache_valid" in ....
map $request_uri $proxy_cache_valid {
default 5m;
~^/blog/[0-9]+/posts/[0-9]+ 1h;
}
proxy_cache_path /var/cache/nginx/xxx_cache keys_zone=xxx_cache:10m;
server {
location ~ /blog/ {
proxy_pass https://www.googleapis.com/blogger/v3/blogs/;
proxy_cache xxx_cache;
proxy_cache_lock on;
proxy_cache_valid $proxy_cache_valid;
}
How can I realise this in nginx (nginx version: nginx/1.16.1)?
Related
proxy_cache_path /tmp/nginx_team_alert_cache keys_zone=team_alerts:10m levels=1:2 max_size=1g use_temp_path=off;
server{
...
location /api/timeentry/timeentry/team_alerts/ {
proxy_cache team_alerts;
proxy_ignore_headers Cache-Control Set-Cookie;
proxy_hide_header "Set-Cookie";
proxy_cache_valid 200 5s;
proxy_cache_key $scheme$host$request_method$request_uri;
proxy_buffering on;
add_header X-Cached $upstream_cache_status;
include uwsgi_params;
uwsgi_pass unix:/tmp/app.sock;
}
}
I have been searching on stackoverflow etc and added all the recommended options but still not caching.
I just realised I am uwsgi_pass so the proxy_* directives won't work, simply replace proxy_* to uwsgi_* worked
http {
...
server {
...
location /good {
proxy_cache mycache;
proxy_cache_key $arg_cachekey;
proxy_cache_valid 200 1h;
proxy_cache_lock on;
proxy_cache_lock_timeout 20m;
proxy_cache_lock_age 20m;
...
(upstream returning 200 with the content)
}
location /bad {
proxy_cache mycache;
proxy_cache_key $arg_cachekey;
proxy_cache_lock on;
proxy_cache_lock_timeout 20m;
proxy_cache_lock_age 20m;
...
(upstream returning 404)
}
}
}
The cache is empty. Requesting:
GET /good?cachekey=123
after a short time while /good upstream is responding with the content, requesting:
GET /bad?cachekey=123
Should request on /bad location wait until /good retrieves cache and /bad respond with 200? If no, how to achieve it?
I have a problem with using nginx as a load balancer. I could configure it to work as a load balancer but I don't how to make it cache static contents from the proxied servers in the backend such as html,css,js, etc... This means I want nginx to
weather to cache or not based on the content of the response from the backend servers if it changed to bypass cache and send requests to the backend and if not to serve from cache. I tried and seached a lot in the internet to make it using many directives such as proxy_cache_bypass and proxy_no_cache but I couldn't. Is there any means to do this if anyone has experience in such topic. these are the configurations:
upstream backend {
server www.webserver1.com:443 max_fails=3 fail_timeout=15s;
server www.webserver2.com:443 max_fails=3 fail_timeout=15s;
}
server {
listen 443 ssl;
rewrite_log on;
error_log /var/log/nginx/lb.error.log;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Proxy-Cache $upstream_cache_status;
ssl_certificate /etc/nginx/client.crt;
ssl_certificate_key /etc/nginx/client.key;
ssl on;
location / {
proxy_cache backcache;
#proxy_cache_methods GET HEAD POST;
#proxy_cache_bypass $cookie_nocache $arg_nocache;
#proxy_no_cache $cookie_nocache $arg_nocache;
proxy_cache_min_uses 1;
#proxy_cache_revalidate on;
#proxy_cache_valid 200 4m;
proxy_cache_lock on;
proxy_cache_background_update on;
add_header X-Proxy-Cache $upstream_cache_status;
proxy_pass https://backend;
}
}
server {
listen 80 ;
if ($http_x_forwarded_proto != 'https') {
rewrite ^(.*) https://$host$1 redirect;
}
}
these are the contents of a config. file under /etc/nginx/conf.d/ which is included in the main config. file which is /etc/nginx/nginx.conf and also those 2 lines are in the main config. file :
proxy_cache_path /var/lib/nginx/cache keys_zone=backcache:20m max_size=100m;
proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args$cookie_user";
Your backend servers could be the root cause of that problem, if those servers were improperly configured. For example sending Cache-Control headers on requests to static files.
According to that docs by default, NGINX respects the Cache-Control headers from origin servers. It does not cache responses with Cache-Control set to Private, No-Cache, or No-Store or with Set-Cookie in the response header.
You can permanently change this behavior by adding those directives:
proxy_ignore_headers Cache-Control;
proxy_cache_valid any 30m;
So the config will look like:
location / {
proxy_cache backcache;
proxy_cache_revalidate on;
proxy_cache_min_uses 3;
proxy_cache_valid 200 302 10m;;
proxy_cache_lock on;
proxy_cache_background_update on;
proxy_ignore_headers Cache-Control;
proxy_cache_valid any 30m;
add_header X-Proxy-Cache $upstream_cache_status;
proxy_pass https://backend;
}
Hope it will help you to figure out.
I'm running in to a really weird issue. I only want to enable Proxy Cache for "new-site.com". However, when doing so, Nginx is proxy caching all of my websites.
I've went through all my vhost / config files and made sure that all "http" and "server" blocks were opened and closed correctly. It's my understanding that Proxy_Cache is only enabled for a site when you include (for example) "proxy_cache new-site;" in your websites "server" block.
In my "http" block, I load all of my websites .conf files, but none of them include any proxy_cache directives.
What am I doing wrong?
Here is a snippet of my config file :
http {
...
...
# nginx cache
proxy_cache_path /www/new-site.com/httpdocs/cache levels=1:2
keys_zone=new-site:10m
max_size=50m
inactive=1440m;
proxy_temp_path /www/new-site.com/httpdocs/cache/tmp 1 2;
# virtual hosting
include /etc/nginx/vhosts/*.conf;
}
Then here is my "new-site.com" vhost conf file:
server {
listen xxx.xxx.xxx.xxx:80;
server_name new-site.com;
root /www/new-site.com/httpdocs;
index index.php;
...
...
proxy_cache new-site;
location / {
try_files $uri #backend;
}
location ~* \.php {
include /usr/local/etc/nginx/proxypass.conf;
proxy_ignore_headers Expires Cache-Control;
proxy_cache_use_stale error timeout invalid_header updating http_502;
proxy_cache_bypass $cookie_session $http_secret_header;
proxy_no_cache $cookie_session;
add_header X-Cache $upstream_cache_status;
proxy_cache_valid 200 302 5m;
proxy_cache_valid 404 1m;
proxy_pass http://127.0.0.1:80;
}
location #backend {
include /usr/local/etc/nginx/proxypass.conf;
proxy_ignore_headers Expires Cache-Control;
proxy_cache_use_stale error timeout invalid_header updating http_502;
proxy_cache_bypass $cookie_session $http_secret_header;
proxy_no_cache $cookie_session;
add_header X-Cache $upstream_cache_status;
proxy_cache_valid 200 302 5m;
proxy_cache_valid 404 1m;
proxy_pass http://127.0.0.1:80;
}
location ~* \.(jpg|jpeg|gif|png|bmp|ico|pdf|flv|swf|css|js)$ {
....
}
}
Once I moved the line "proxy_cache new-site;" in to a "location" block, that resolved the issue for me.
Not sure why I have this issue when it sits outside a block though.
I'm adding some https pages to my rails site. In order to test it locally, i'm running my site under one mongrel_rails instance (on 3000) and nginx.
I've managed to get my nginx config to the point where i can actually go to the https pages, and they load. Except, the javascript and css files all fail to load: looking in the Network tab in chrome web tools, i can see that it is trying to load them via an https url. Eg, one of the non-working file urls is
https://cmw-local.co.uk/stylesheets/cmw-logged-out.css?1383759216
I have these set up (or at least think i do) in my nginx config to redirect to the http versions of the static files. This seems to be working for graphics, but not for css and js files.
If i click on this in the Network tab, it takes me to the above url, which redirects to the http version. So, the redirect seems to be working in some sense, but not when they're loaded by an https page. Like i say, i thought i had this covered in the second try_files directive in my config below, but maybe not.
Can anyone see what i'm doing wrong? thanks, Max
Here's my nginx config - sorry it's a bit lengthy! I think the error is likely to be in the first (ssl) server block:
NOTE: the urls in here (elearning.dev, cmw-dev.co.uk, etc) are all just local host names, ie they're all just aliases for 127.0.0.1.
server {
listen 443 ssl;
keepalive_timeout 70;
ssl_certificate /home/max/work/charanga/elearn_container/elearn/config/nginx/certs/max-local-server.crt;
ssl_certificate_key /home/max/work/charanga/elearn_container/elearn/config/nginx/certs/max-local-server.key;
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 10m;
ssl_protocols SSLv3 TLSv1;
ssl_ciphers RC4:HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
server_name elearning.dev cmw-dev.co.uk cmw-dev.com cmw-nginx.co.uk cmw-local.co.uk;
root /home/max/work/charanga/elearn_container/elearn;
# ensure that we serve css, js, other statics when requested
# as SSL, but if the files don't exist (i.e. any non /basket controller)
# then redirect to the non-https version
location / {
try_files $uri #non-ssl-redirect;
}
# securely serve everything under /basket (/basket/checkout etc)
# we need general too, because of the email/username checking
location ~ ^/(basket|general|cmw/account/check_username_availability) {
# make sure cached copies are revalidated once they're stale
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
# this serves Rails static files that exist without running
# other rewrite tests
try_files $uri #rails-ssl;
expires 1h;
}
location #non-ssl-redirect {
return 301 http://$host$request_uri;
}
location #rails-ssl {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_read_timeout 180;
proxy_next_upstream off;
proxy_pass http://127.0.0.1:3000;
expires 0d;
}
}
#upstream elrs {
# server 127.0.0.1:3000;
#}
server {
listen 80;
server_name elearning.dev cmw-dev.co.uk cmw-dev.com cmw-nginx.co.uk cmw-local.co.uk;
root /home/max/work/charanga/elearn_container/elearn;
access_log /home/max/work/charanga/elearn_container/elearn/log/access.log;
error_log /home/max/work/charanga/elearn_container/elearn/log/error.log debug;
client_max_body_size 50M;
index index.html index.htm;
# gzip html, css & javascript, but don't gzip javascript for pre-SP2 MSIE6 (i.e. those *without* SV1 in their user-agent string)
gzip on;
gzip_http_version 1.1;
gzip_vary on;
gzip_comp_level 6;
gzip_proxied any;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript; #text/html
# make sure gzip does not lose large gzipped js or css files
# see http://blog.leetsoft.com/2007/7/25/nginx-gzip-ssl
gzip_buffers 16 8k;
# Disable gzip for certain browsers.
#gzip_disable "MSIE [1-6].(?!.*SV1)";
gzip_disable "MSIE [1-6]";
# blank gif like it's 1995
location = /images/blank.gif {
empty_gif;
}
# don't serve files beginning with dots
location ~ /\. { access_log off; log_not_found off; deny all; }
# we don't care if these are missing
location = /robots.txt { log_not_found off; }
location = /favicon.ico { log_not_found off; }
location ~ affiliate.xml { log_not_found off; }
location ~ copyright.xml { log_not_found off; }
# convert urls with multiple slashes to a single /
if ($request ~ /+ ) {
rewrite ^(/)+(.*) /$2 break;
}
# X-Accel-Redirect
# Don't tie up mongrels with serving the lesson zips or exes, let Nginx do it instead
location /zips {
internal;
root /var/www/apps/e_learning_resource/shared/assets;
}
location /tmp {
internal;
root /;
}
location /mnt{
root /;
}
# resource library thumbnails should be served as usual
location ~ ^/resource_library/.*/*thumbnail.jpg$ {
if (!-f $request_filename) {
rewrite ^(.*)$ /images/no-thumb.png
break;
}
expires 1m;
}
# don't make Rails generate the dynamic routes to the dcr and swf, we'll do it here
location ~ "lesson viewer.dcr" {
rewrite ^(.*)$ "/assets/players/lesson viewer.dcr" break;
}
# we need this rule so we don't serve the older lessonviewer when the rule below is matched
location = /assets/players/virgin_lesson_viewer/_cha5513/lessonViewer.swf {
rewrite ^(.*)$ /assets/players/virgin_lesson_viewer/_cha5513/lessonViewer.swf break;
}
location ~ v6lessonViewer.swf {
rewrite ^(.*)$ /assets/players/v6lessonViewer.swf break;
}
location ~ lessonViewer.swf {
rewrite ^(.*)$ /assets/players/lessonViewer.swf break;
}
location ~ lgn111.dat {
empty_gif;
}
# try to get autocomplete school names from memcache first, then
# fallback to rails when we can't
location /schools/autocomplete {
set $memcached_key $uri?q=$arg_q;
memcached_pass 127.0.0.1:11211;
default_type text/html;
error_page 404 =200 #rails; # 404 not really! Hand off to rails
}
location / {
# make sure cached copies are revalidated once they're stale
add_header Cache-Control "public, must-revalidate, proxy-revalidate";
# this serves Rails static files that exist without running other rewrite tests
try_files $uri #rails;
expires 1h;
}
location #rails {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_read_timeout 180;
proxy_next_upstream off;
proxy_pass http://127.0.0.1:3000;
expires 0d;
}
}
EDIT: It just occurred to me that this might be better on superuser or serverfault, or perhaps both. I'm not sure what the cross-site posting rules are.