What does Nginx take into account when caching a request? - nginx

What attributes(headers, cookies) besides url does Nginx take into account when caching a request?
I was trying to find out this information in nginx/log/access.log file. I have found only information about what type of request had been made(GET) and its status(404, 200, etc.).
Here my nginx.conf file:
#user nobody;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
# MY TEST SERVER
http {
# CACHE
proxy_cache_path "C:\data\cache" levels=1:2 keys_zone=all:32m;
server {
listen 80;
location / {
proxy_pass http://127.0.0.1:81/;
proxy_cache all;
proxy_cache_valid 404 502 503 1s;
proxy_cache_valid any 10s;
}
server {
listen 81;
location / {
root "C:\data\www";
}
location /images/ {
root "C:\data";
}
}
}

What does Nginx take into account when caching a request?
The full URL
The relevant config for what Nginx uses when caching a request is proxy_cache_key
Syntax: proxy_cache_key string;
Default: proxy_cache_key $scheme$proxy_host$request_uri;
Context: http, server, location
Defines a key for caching, for example
proxy_cache_key "$host$request_uri $cookie_user";
By default, the
directive’s value is close to the string
proxy_cache_key $scheme$proxy_host$uri$is_args$args;
So by default Nginx does not take into account any headers when caching a request, only the full url (with get Args)
How to customize the cache key
To customize the cache key only requires using the proxy_cache_key directive.
To cache based on a cookie value, there's already an example in the documentation - but I suggest:
proxy_cache_key "$host$request_uri:u:$cookie_user";
Why add more text to the key? Consider what happens if you use multiple cookies like so:
proxy_cache_key "$host$request_uri:u:$cookie_user:g:$cookie_group";
It avoids the possibility of a request with only $cookie_group colliding with the cached content for something with the same value only in $cookie_user. It also makes it easier to understand what the files on disk contain, if you have the need to look.
If you want to use an arbitrary header - just include the variable for that header e.g.:
proxy_cache_key "$host$request_uri:n:$http_name";

Related

NGINX resolving a non configured domain, why?

I have one server running on: http://localhost:8080
I'm configuring a sample NGINX server.
I copied from internet the following configuration:
# user nobody;
worker_processes 1;
error_log logs/error.log;
error_log logs/error.log notice;
error_log logs/error.log info;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
# gzip on;
server
{
listen 80;
server_name mydomain01.com www.mydomain01.com;
location /
{
proxy_pass http://localhost:8080;
include "../proxy_params.conf";
}
}
}
On the hosts file I have just the following entries:
127.0.0.1 mydomain01.com
127.0.0.1 www.mydomain01.com;
127.0.0.1 mydomain02.com
127.0.0.1 www.mydomain02.com;
When I go to: http://mydomain01.com I get the same content as on: http://localhost:8080
My question is:
Why when I go to: http://mydomain02.com I also get the same content as on: http://localhost:8080?
I think I should not get that content because this last domain is not on the NGINX configuration.
Do I have an error on the configuration above?
Thanks!
nginx always contains a default server which will handle requests for server names that do not match a server_name directive. If you do not define a default_server, nginx will use the first server block with a matching location. See this document for details.

Nginx supervisord configuration

I have a supervisord server running on localhost:9001.
I am trying to serve it at localhost/supervisord.
The nginx config is like this:
worker_processes 1;
error_log /var/log/nginx/error.log;
pid /tmp/nginx.pid;
#daemon off;
events {
worker_connections 1024;
}
http {
# MIME / Charset
default_type application/octet-stream;
charset utf-8;
# Logging
access_log /var/log/nginx/access.log;
# Other params
server_tokens off;
tcp_nopush on;
tcp_nodelay off;
sendfile on;
upstream supervisord {
server localhost:9001;
}
server {
listen 80;
client_max_body_size 4G;
keepalive_timeout 5;
location ^~ /stylesheets {
alias /Users/ocervell/.virtualenvs/ndc-v3.3/lib/python2.7/site-packages/supervisor/ui/stylesheets;
access_log off;
}
location ^~ /images {
alias /Users/ocervell/.virtualenvs/ndc-v3.3/lib/python2.7/site-packages/supervisor/ui/images;
access_log off;
}
location /supervisord {
# Set client IP / Proxy IP
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
# Set host header
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://supervisord/;
}
}
}
Before adding ^~ /images and ^~ /stylesheets locations the page was returning a 502 Bad Gateway.
With the above config I am able to access localhost/supervisord but the CSS is missing on the page.
I see the css / images are loaded correctly in the browser:
But I see an error message in the browser console and it seems to be the culprit:
The mimetype in the browser for localhost/stylesheets/supervisor.css it shows as octet-stream instead of text/css.
The mimetype in the browser for localhost:9001/stylesheets/supervisor.css it shows as the correct text/css.
How can I fix this error ?
I thought about dynamically rewriting the mimetype for static files, but I am not an expert in nginx and have no idea how to do that from nginx config.
It's really interesting such an obvious function like putting web interfaces behind transparent reverse proxy is not that straightforward to configure as it should be.
Anyway this is what I do to get reverse proxy working with applications where root can't be modified, like supervisor:
location /supervisor {
proxy_pass http://127.0.0.1:9001/;
}
location / {
if ($http_referer ~ "^.*/supervisor"){
return 301 /supervisor/$request_uri;
}
}
Application-side requests will hit the main end-point but then NginX will re-direct them to the /supervisor EP
This works in most cases but not always.The following supervisor's web functions will fail:
getting action confirmation - you can start / stop services but the result page will fail to load; just go to the /supervisor EP to check the result
live tail does not work; however there is a log display with manual refresh which works under the link with the program name.
Anyway this partial support is good engough for me and you may find it also useful.
I was able to get it working simply with this:
upstream supervisor {
server 127.0.0.1:9001;
}
server {
# ...
location /supervisor/ {
proxy_pass http://supervisor/;
}
}
Even worked in the browser with and without ending slash in url (ie both http://example.com/supervisor and http://example.com/supervisor/ worked).
This was a must for me!

nginx rewrite not working

I'm trying to set up a simple nginx server to act as a proxy between my front end ui and my back end api. The setup is fairly simple. The UI makes all api requests to /api/endpoint and the proxy server passes the request to the api. The proxy also needs to rewrite the request so that instead of going to http://api.location.net/api/endpoint, it goes to http://api.location.net/endpoint. The UI resides on http://api.location.net. This part isn't working (i get a 500 error) and I'm pretty sure it has to do with how I'm writing my rewrite rule. Here's my nginx config.
daemon off;
error_log off;
worker_processes 2;
worker_rlimit_nofile 100000;
events {
worker_connections 50000;
accept_mutex off;
}
http {
include /etc/nginx/mime.types;
access_log off;
sendfile on;
server {
listen 80 default_server;
server_name localhost _;
location / {
alias /srv/site/;
}
location /api/ {
rewrite ^/api ""; # I think this is the problem
proxy_pass http://api.location.net;
proxy_pass_request_headers on;
proxy_pass_header X-ResponseData;
proxy_redirect off;
}
}
}
Any help would be greatly appreciated, nginx is still fairly new for me and the documentation on nginx rewrite doesn't seem to have what I need.
If I understood you right, this should help
location /api/ {
proxy_pass http://api.location.net/;
proxy_pass_request_headers on;
proxy_pass_header X-ResponseData;
proxy_redirect off;
}
Note the URI part at proxy_pass directive
If the proxy_pass directive is specified with a URI, then when a
request is passed to the server, the part of a normalized request URI
matching the location is replaced by a URI specified in the directive:
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass

Nginx loosing header sometime

I am using nginx as reverse proxy server for android application(get/post requests only). Some of the data contained in the headers. In some cases nginx loses "id" or "fail_id" header.
config:
user user;
worker_processes 4;
error_log /var/log/nginx/error.log;
events {
worker_connections 100000;
use epoll;
}
http {
upstream myproject {
server 192.168.88.246:2053;
}
server {
listen 2054;
ssl on;
ssl_certificate /home/user/android/cert/cert.pem;
ssl_certificate_key /home/user/android/cert/key.pem;
proxy_read_timeout 600;
proxy_send_timeout 600;
location / {
proxy_pass http://myproject;
proxy_pass_request_headers on;
}
}
}
Could i set original headers of request?
Updated:
A more detailed study found that nginx miss "fail_id" header. All other headers are working.
Problem solved!
Nginx default config misses headers with underscore.
This directive solved the problem:
underscores_in_headers on;
Thank You For the underscore directive. I have used underscores_in_headers on; directive and the header value with underscore passed to my node app.
Now I am able to access header value (api_key) using postman request and angular request from the web.
But now when The request is raise from the android app and I have set the api_key in the android request header, I am unable to access the api_key.
My config is:
server {
listen 80;
server_name uat.api.myserver.com;
underscores_in_headers on;
location / {
proxy_pass http://localhost:9102;
}
}

Tainted canvas when accessing HLS live stream on Galaxy Ace 2

I'm using avconv (ffmpeg) and nginx to stream frames from a camera over HLS and RTMP. Since my phone doesn't support flash it uses HTML5 video tags and HLS in order to stream the video. One feature that I'm trying to support is to record the live stream and save that to a file. However, I am unable to record the stream due to a cross-domain issue.
The live stream is coming from my machine on port 8080 (I'm referencing it using my internal IP, 10.150.x.x:8080/hls/mystream.m3u8) and the server is run on my machine through port 8000 (also referenced through internal IP). Because they are on different ports it is still viewed as cross domain.
In my nginx.conf I have added Access-Control-Allow-Origin: *
and I've also tried adding Access-Control-Allow-Methods GET, PUT, POST, DELETE, OPTIONS
and Access-Control-Allow-Headers Content-Type, Authorization, X-Requested-With
When I examine the headers using curl -I http://10.150.x.x:8080/hls/mystream.m3u8 and through firefox and chrome from my desktop I can see the appropriate headers. But when I look at the headers using the chrome dev tools for my phone I get "CAUTION: Provisional headers shown."
I attempt to capture the frames using canvas.toDataURL() and it is this function that is giving the security error.
Why is it that even though I have Access-Control-Allow-Origin: * in my nginx.conf I still get a cross domain issue?
nginx.conf:
#user nobody;
worker_processes 1;
error_log logs/error.log debug;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 8080;
server_name 10.150.x.x;
#server_name bsid.ca;
add_header 'Access-Control-Allow-Origin' "*";
#add_header 'Access-Control-Allow-Methods' "GET, PUT, POST, DELETE, OPTIONS";
#add_header 'Access-Control-Allow-Headers' "Content-Type, Authorization, X-Requested-With";
location /hls {
types {
application/vnd.apple.mpegurl m3u8;
video/mp2t ts;
}
root /path/to/nginxVideo;
}
# sample handlers
#location /on_play {
# if ($arg_pageUrl ~* 127.0.0.1) {
# return 201;
# }
# return 202;
#}
#location /on_publish {
# return 201;
#}
#location /vod {
# alias /var/myvideos;
#}
# rtmp stat
location /stat {
rtmp_stat all;
rtmp_stat_stylesheet stat.xsl;
}
location /stat.xsl {
# you can move stat.xsl to a different location
root /usr/build/nginx-rtmp-module;
}
# rtmp control
location /control {
rtmp_control all;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
rtmp {
server {
listen 1935;
ping 30s;
notify_method get;
application myapp {
live on;
# sample play/publish handlers
#on_play http://127.0.0.1:8080/on_play;
#on_publish http://127.0.0.1:8080/on_publish;
# sample recorder
#recorder rec1 {
# record all;
# record_interval 30s;
# record_path /tmp;
# record_unique on;
#}
# sample HLS
hls on;
hls_path /home/richard/Media/nginxVideo/hls;
hls_base_url http://10.150.x.x:8080/hls/;
hls_sync 2ms;
}
# Video on demand
#application vod {
# play /var/Videos;
#}
# Video on demand over HTTP
#application vod_http {
# play http://127.0.0.1:8080/vod/;
#}
}
}
Full error:
Uncaught SecurityError: Failed to execute 'toDataURL' on 'HTMLCanvasElement': Tainted canvases may not be exported.
UPDATE
After a lengthy discussion with Ray Nicholus it was determined that the issue was that I was setting the crossorigin attribute on my video element after the stream had begun. By setting it earlier I was able to access the frames without the need of a proxy.
Dev tools will not reveal most specifics about the request if it believes the response has not properly acknowledged the cross-origin request. All I can think of is that you are setting the crossorigin attribute after the bytes have started to stream in (at which point the video is already tainted), either that or your server is not properly acknowledging the request. If the request is lacking an Origin header, the former is likely the case.

Resources