I need to configure Nginx to log the $request_body only for certain location blocks.
Inspired by this post, I expected the following to work:
http {
log_format l2met "request_body = $xxxx";
access_log logs/nginx/access.log l2met;
error_log logs/nginx/error.log;
server {
location /path/to/log {
set $xxxx $request_body;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
location / {
set $xxxx "NOT_LOGGED";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
}
}
However, the request body just gets printed as -
I can print the $request_body all the time with this:
http {
log_format l2met "request_body = $request_body";
access_log logs/nginx/access.log l2met;
error_log logs/nginx/error.log;
server {
location /path/to/log {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
}
}
I can conditionally set the logging with this:
http {
log_format l2met "xxxx = $xxxx";
access_log logs/nginx/access.log l2met;
error_log logs/nginx/error.log;
server {
location /path/to/log {
set $xxxx "NEED TO LOG REQUEST BODY HERE";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
location / {
set $xxxx "NOT_LOGGED";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_redirect off;
proxy_pass http://app_server;
}
}
}
For some reason, it seems that as soon as I reference $request_body from inside a location block, the log output changes to -.
I've tried adding client_body_in_single_buffer to the http block, but that doesn't seem to help.
How do I make this work?
I'm using the version of nginx that was installed by default with apt-get:
$ nginx -v
nginx version: nginx/1.1.19
You could declare any number of log_formats and use ones that you need.
http {
log_format l2met 'measure#nginx.service=$request_time request_id=$http_x_request_id';
log_format capturebody "request_body = $request_body";
access_log logs/nginx/access.log l2met;
server {
...
location / {
...
}
location /other/ {
# this is the location where you need to log $request_body
access_log logs/nginx/access.log capturebody;
...
}
...
}
}
Related
I have nginx .conf file as below; I'm wondering if it can be write simpler:
default.conf
upstream docsapp {
server app:8000;
}
server {
listen 80;
location / {
alias /usr/share/nginx/html/;
}
location /admin/ {
proxy_pass http://docsapp/admin/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /api/ {
proxy_pass http://docsapp/api/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /api-token-auth/ {
proxy_pass http://docsapp/api-token-auth/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /docs/ {
proxy_pass http://docsapp/docs/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /media/ {
add_header Access-Control-Allow-Origin *;
alias /usr/local/src/app/media/;
internal;
}
location /static/ {
alias /usr/local/src/app/static/;
}
client_max_body_size 8M;
}
Location \ is Vue.js app, remaining is Django REST Framework. I've searched a web for a while but none of found solutions worke for me.
Another problem occurred during attempt to split this file into two:
vue.conf
server {
listen 80;
location / {
alias /usr/share/nginx/html/;
}
}
and
drf.conf
upstream docsapp {
server app:8000;
}
server {
listen 80;
location /admin/ {
proxy_pass http://docsapp/admin/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /api/ {
proxy_pass http://docsapp/api/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /api-token-auth/ {
proxy_pass http://docsapp/api-token-auth/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /docs/ {
proxy_pass http://docsapp/docs/;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
location /media/ {
add_header Access-Control-Allow-Origin *;
alias /usr/local/src/app/media/;
internal;
}
location /static/ {
alias /usr/local/src/app/static/;
}
client_max_body_size 8M;
}
It also didn't work. Any help would be appreciated.
I think since you do not change request URI when you proxy requests to the docsapp upstream, you can replace four location blocks where proxy_pass directive used with the following one:
location ~ ^/(?:admin|api|api-token-auth|docs)/ {
proxy_pass http://docsapp;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
You can't split your server block in two, only one of them will work acting as default server (see the documentation).
I have below configuration. I only want to authorize / because the only one that has a UI. Other URLs are already apis like /report/, /group/, /delete/ and so on.
upstream gofastdfs{
server localhost:8081;
keepalive 32;
}
server {
listen 8080;
server_name localhost;
gzip on;
gzip_types '*';
location / {
auth_basic "Caution";
auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
keepalive_timeout 620;
proxy_redirect off;
proxy_buffering off;
proxy_pass http://gofastdfs;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
But my configuration authorizes every URL, I only want to limit to /
location / matches any URI that is not handled by some other location block - i.e. it is the default location.
location = / only matches the single URI /.
See this document for details.
You can split your configuration into two location blocks using one with authentication and the other without. Some statements will need to be duplicated between both location blocks, but most can be moved into the outer context.
For example:
keepalive_timeout 620;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host:$server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
location = / {
auth_basic "Caution";
auth_basic_user_file /etc/nginx/conf.d/.htpasswd;
proxy_pass http://gofastdfs;
}
location / {
proxy_pass http://gofastdfs;
}
I have two application and a nginx server on top of that. I want to proxy all GET requests coming on nginx to one app running on http://127.0.0.1:9101/ and proxy all other request methods to http://10.41.115.241:8000/
I have tried couple of options but none worked
I have tried using limit_exempt
location /api/v1/executions {
error_page 502 = #apiError;
rewrite ^/api/(.*) /$1 break;
proxy_pass http://127.0.0.1:9101/;
limit_except PUT POST DELETE {
proxy_pass http://10.41.115.241:8000/;
}
proxy_read_timeout 90;
proxy_connect_timeout 90;
proxy_redirect off;
proxy_set_header Host $host;
#proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header Connection '';
chunked_transfer_encoding off;
proxy_buffering off;
proxy_cache off;
#proxy_set_header Host $host;
}
I have also tried if condition
location /api/v1/executions {
error_page 502 = #apiError;
rewrite ^/api/(.*) /$1 break;
proxy_pass http://10.41.115.241:8000/;
if ($request_method = GET) {
proxy_pass http://127.0.0.1:9101/;
}
proxy_read_timeout 90;
proxy_connect_timeout 90;
proxy_redirect off;
proxy_set_header Host $host;
#proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header Connection '';
chunked_transfer_encoding off;
proxy_buffering off;
proxy_cache off;
#proxy_set_header Host $host;
}
but in both ways I got this error
"proxy_pass" cannot have URI part in location given by regular expression, or inside named location, or inside "if" statement, or inside "limit_except" block in /path/to/config
Always try to use map instead of if for conditional logic in NGINX. Unless it's absolutely impossible with map.. In your particular case, it's easy.
Create a variable $backend which will hold your desired proxy_pass value depending on the request methods:
http {
map $request_method $backend {
default http://10.41.115.241:8000/;
GET http://127.0.0.1:9101/;
}
...
}
Then use it you in your config:
location /api/v1/executions {
error_page 502 = #apiError;
rewrite ^/api/(.*) /$1 break;
proxy_pass $backend;
proxy_read_timeout 90;
proxy_connect_timeout 90;
proxy_redirect off;
proxy_set_header Host $host;
#proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
#proxy_set_header Connection '';
chunked_transfer_encoding off;
proxy_buffering off;
proxy_cache off;
#proxy_set_header Host $host;
}
I have a NGINX server as front-end cache server and I'd like to disable cache on specific urls.
Here is the configuration on NGINX:
proxy_cache_path /tmp/nginx levels=1:2 keys_zone=my_zone:10m inactive=120m max_size=1000m;
proxy_cache_key "$scheme$request_method$host$request_uri$is_args$args";
server {
listen 10.0.0.45:80 default_server;
server_name proxy2.jjd;
include /etc/nginx/default.d/*.conf;
location / {
client_max_body_size 20m;
proxy_cache my_zone;
proxy_cache_bypass $http_cache_control;
proxy_no_cache $http_pragma $http_authorization $cookie_nocache $arg_nocache;
add_header X-Proxy-Cache-NGINX $upstream_cache_status;
add_header X-Real-IP $remote_addr;
add_header Cache-Control "public";
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8080;
proxy_read_timeout 90;
proxy_connect_timeout 90;
proxy_redirect off;
}
}
Add the following location to avoid an url:
location ^~ /your-url/ {
add_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_set_header X-Forwarded-Port 443;
proxy_set_header Host $host;
proxy_pass http://127.0.0.1:8080;
proxy_read_timeout 90;
proxy_connect_timeout 90;
proxy_redirect off;
}
It just assigns this location to the proxy and doesn't enable caching for it.
As I get it, you just need a nested location with a single string proxy_cache off; inside to disable caching for nested URLs. Like this:
location / {
proxy_cache my_zone;
proxy_cache_bypass $http_cache_control;
// other stuff related to proxying or other processing
location /do/not/cache/this/url/ {
proxy_cache off;
}
}
you can just specify location do proxy_pass only for disable cache
location /will/not/cache {
proxy_pass http://127.0.0.1:8080;
..set_header ..
}
I've got an Elasticsearch cluster plus Logstash and Kibana, and I only want to expose a read-only window into the indexes, with the exception of the index kibana-int so that dashboards can be saved.
I've found a suitable ES proxy config, and I've modified it to use limit_except to disallow write/modify to other indexes, but much of the config is needlessly duplicated. Is there a cleaner way to define this?
upstream elasticsearch {
server es-01.iad.company.com:9200;
server es-02.iad.company.com:9200;
}
server {
listen 9200;
server_name elasticsearch.proxy;
client_max_body_size 50m;
location / {
limit_except GET POST HEAD OPTIONS {
deny all;
}
proxy_pass http://elasticsearch;
proxy_redirect off;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_pass_header Access-Control-Allow-Origin;
proxy_pass_header Access-Control-Allow-Methods;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
}
location /kibana-int/ {
proxy_pass http://elasticsearch;
proxy_redirect off;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_pass_header Access-Control-Allow-Origin;
proxy_pass_header Access-Control-Allow-Methods;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
}
}
There are several ways:
Solution 1
You could put repeating config into file and include it.
Your config:
upstream elasticsearch {
server es-01.iad.company.com:9200;
server es-02.iad.company.com:9200;
}
server {
listen 9200;
server_name elasticsearch.proxy;
client_max_body_size 50m;
location / {
limit_except GET POST HEAD OPTIONS {
deny all;
}
include proxy.inc;
}
location /kibana-int/ {
include proxy.inc;
}
}
proxy.inc:
proxy_pass http://elasticsearch;
proxy_redirect off;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
Solution 2
Other way is use nginx's directive inheritance.
upstream elasticsearch {
server es-01.iad.company.com:9200;
server es-02.iad.company.com:9200;
}
server {
listen 9200;
server_name elasticsearch.proxy;
client_max_body_size 50m;
proxy_redirect off;
proxy_set_header Connection "";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_hide_header Access-Control-Allow-Headers;
add_header Access-Control-Allow-Headers 'X-Requested-With, Content-Type';
add_header Access-Control-Allow-Credentials true;
location / {
limit_except GET POST HEAD OPTIONS {
deny all;
}
proxy_pass http://elasticsearch;
}
location /kibana-int/ {
proxy_pass http://elasticsearch;
}
}
BTW, your proxy_pass_header directives are needless. Nginx proxies almost all headers by default.