"proxy_pass" cannot have URI part in location given by regular expression - nginx

I've developed a URL shortening web application.
It consists of two separate docker containers: one containing the backend REST api and another containing the frontend static website.
These two containers are linked to an nginx container.
The configuration for this nginx container is below:
worker_processes 1;
events { worker_connections 1024; }
http {
upstream api {
server short-url:8080;
}
upstream frontend {
server short-url-frontend:8081;
}
gzip on;
gzip_vary on;
gzip_min_length 860;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml application/javascript application/x-javascript application/xml;
gzip_disable "MSIE [1-6]\.";
server {
listen 80;
root /user/share/nginx/html;
location /urlshortener/v1 {
proxy_pass http://api/urlshortener/v1;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $server_name;
}
location ~ ^/([A-Za-z0-9]+) {
rewrite ^/([A-Za-z0-9]+) /$1
proxy_pass http://api/urlshortener/v1;
}
location / {
proxy_pass http://frontend;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-for $remote_addr;
}
}
}
If a url ends with /urlshortening/v1, I'm proxying to the backend.
If a url starts with /, I'm proxying to the frontend.
Shortened urls e.g. /3xTy or /a0q need to be proxied to the backend so that the user can be navigated to the original url. In order to do this, I've defined a location with a regular expression.
location ~ ^/([A-Za-z0-9]+) {
rewrite ^/([A-Za-z0-9]+) /$1
proxy_pass http://api/urlshortener/v1;
}
This block of code gives me the following error:
2018/11/17 16:47:03 [emerg] 1#1: "proxy_pass" cannot have URI part in location given by regular expression, or inside named location, or inside "if" statement, or inside "limit_except" block in /etc/nginx/nginx.conf:36
I've gone through several examples and reviewed a number of answers and I believe that the configuration I have should work. Can someone please explain why I'm getting this error?

If you use a URI with a proxy_pass statement within a regular expression location, you need to build the entire URI using one or more variables. See this document for details.
So the alternatives are to (1), capture the URI from the location expression and add it to the proxy_pass statement. For example:
location ~ ^/([A-Za-z0-9]+) {
proxy_pass http://api/urlshortener/v1/$1;
}
Or (2), use proxy_pass without a URI part, and construct the desired URI using a rewrite...break. For example:
location ~ ^/([A-Za-z0-9]+) {
rewrite ^/([A-Za-z0-9]+) /urlshortener/v1/$1 break;
proxy_pass http://api;
}
See this document for details.

I was getting same error and the correct answer could not be helping in my situation. here was my problematic configs:
# proxying wordpress static files
location ~* (((product|blog|product-category)\/.*)|(\.(ogg|ogv|svg|svgz|eot|otf|woff|woff2|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)))$ {
proxy_pass https://mainhost.cdkeysell.ir/;
...
}
so after comparing it with my other wesite with similar configs without such error I found this could be simply solved by removing the latter / at end of proxy_pass config:
# proxying wordpress static files
location ~* (((product|blog|product-category)\/.*)|(\.(ogg|ogv|svg|svgz|eot|otf|woff|woff2|mp4|ttf|css|rss|atom|js|jpg|jpeg|gif|png|ico|zip|tgz|gz|rar|bz2|doc|xls|exe|ppt|tar|mid|midi|wav|bmp|rtf)))$ {
# proxy_pass https://mainhost.cdkeysell.ir/; ==> final `/` caused error
proxy_pass https://mainhost.cdkeysell.ir;
...
}

Related

nginx: Match multiple locations / disable access log without returning

I would like to disable access logging for some specific paths but still proxy it to another container. In other words "match multiple locations without returning/exiting" which is not possible as far as I know.
The following config will make nginx cancel the request without entering the proxy pass location.
server {
# ...
# do not log requests for /_nuxt/* and /_ipx/*
location ~ ^/(_ipx|_nuxt) {
access_log off;
}
# still proxy these paths
location ~* ^(\/|\/(foo|bar|_nuxt|_ipx)$ {
proxy_pass http://frontend:3000;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $server_name:$server_port;
}
}
Is there a more clean way of achieving the desired behavior other than duplicating the proxy configuration and adding the access log config line to that second location?
server {
# ...
# proxy pass without _nuxt and _ipx
location ~* ^(\/|\/(foo|bar)$ {
proxy_pass http://frontend:3000;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $server_name:$server_port;
}
# access log + proxy pass
location ~ ^/(_ipx|_nuxt) {
access_log off;
proxy_pass http://frontend:3000;
proxy_http_version 1.1;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $server_name:$server_port;
}
}
You're right, location working like switch case taking the first hit and break.
Maybe you can try something like that:
if ($request_uri ~ ^/(_ipx|_nuxt)) {
access_log off;
}
instead of your first location statement.
The access_log directive has the following syntax:
access_log path [format [buffer=size] [gzip[=level]] [flush=time] [if=condition]]; ...
...
The if parameter (1.7.0) enables conditional logging. A request will not be logged if the condition evaluates to “0” or an empty string. In the following example, the requests with response codes 2xx and 3xx will not be logged:
map $status $loggable {
~^[23] 0;
default 1;
}
access_log /path/to/access.log combined if=$loggable;
Applied to the asked question, that means the following config should achieve the desired goal:
map $uri $loggable {
~^/_(ips|nuxt) 0;
default 1;
}
server {
...
access_log /path/to/access.log <format> if=$loggable;
}

Hide a port from the content-security-policy so that script-src considers the url as self

I'm setting up the content-security-policy for my site , but the script url for my comments system contains a port number i.e http://dev.example.com:8080/client.js.
I have a few other URL's which use ports too and I'd prefer to hide which ports my site uses in case of possible security implications.
How can I hide these URL's containing port numbers so that the CSP considers the URL's as self?
For example instead of http://dev.example.com:8080/client.js I'd like it to be http://dev.example.com/comments/client.js.
I'm on an Nginx server and I've tried playing around with rewriting the URL, but can't get it working.
Here's what I've tried.
This allows me to change the script URL to http://dev.example.com/comments/client.js, but CSP still detects http://dev.example.com:8080/client.js.
location ~ /comments/ {
return 301 http://dev.example.com:8080/client.js;
}
Didn't work
location ~ /comments/ {
rewrite ^/comments/(.*) http://dev.example.com:8080/$1 last;
}
Just to be clear I know I can just throw the URL http://dev.example.com:8080/client.js straight into CSP e.g script-src 'self'
dev.example.com:8080/client.js;, but I want the URL to work with self e.g script-src 'self'.
Returns 502 bad gateway error
location /comments {
proxy_pass http://dev.example.com:8080/;
}
Also returns502 bad gateway.
location = /comments {
return 301 /comments/;
}
location ~ /comments/(?<ndpath>.*) {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_pass_request_headers on;
proxy_set_header Connection "keep-alive";
proxy_store off;
proxy_pass http://dev.example.com:8080/$ndpath$is_args$args;
gzip on;
gzip_proxied any;
gzip_types *;
}
What you are looking for is the concept of a reverse proxy. Described here.
In your case the simplest example is
location /comments {
proxy_pass http://localhost:8080/;
}

NGINX Rewrite ignored / not working with proxy_pass

Bug in the upstream vendor app. Wrote a route in our node app to proxy the request and avoid the bug but can't get the NGINX rewrite to work correctly. I've tried many variations of rewrite and now at my wit's end. Spent more time on the rewrite than the actual code... =(
IN: /Txtranscription/transcription/TranscriptionHandler.ashx?q=c3R1ZHlfaWQ...
OUT: /Txtranscription/transcription/TranscriptionHandler.ashx?q=c3R1ZHlfaWQ...
EXPECTED: /transcription?encoded=c3R1ZHlfaWQ...
### override handling of /Txtranscription/transcription/TranscriptionHandler.ashx
location /Txtranscription/transcription/TranscriptionHandler.ashx {
add_header Access-Control-Allow-Origin $cors_header;
access_log logs/vapi.proxy.log lfupstream;
error_log logs/vapi.error.log error;
rewrite ^/Txtranscription/transcription/TranscriptionHandler\.ashx\?q=(.*)$ /transcription?encoded=$1 break;
proxy_pass http://vapi;
}
You shouldn't need to rewrite the request at all, you can append a path to a proxy_pass directive and Nginx will replace the matching part of the location block from the original request URI with the URI of your proxy_pass directive.
So this should work:
location /Txtranscription/transcription/TranscriptionHandler.ashx {
set $args encoded=$arg_q;
....
proxy_pass http://vapi/transcription$is_args$args;
Example:
location ~ ^/connector(/?)(.*)$ {
proxy_buffer_size 64k;
proxy_buffers 16 32k;
proxy_http_version 1.1;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Connection "Keep-Alive";
proxy_set_header Proxy-Connection "Keep-Alive";
proxy_set_header Authorization "";
set $upstream_endpoint http://YOUR-END-POINT/$2$is_args$args;
proxy_pass $upstream_endpoint;
}
The magic is -> location ~ ^ /admin (/?)(.*)$
And then -> /$2$is_args$args;

Serving static assets for a single-page-app with nginx

I'm trying to figure out how to have nginx serve static assets, have a fallback to index.html, and forward to the API.
Currently, all I have working is the root route /, and the API forwarding.
This is the behavior that I'm after:
GET / -> nginx sends s3/index.html (current)
* /api -> nginx proxies to the puma server (current)
# Yet to figure out (and the reason for this question)
GET /sub-route -> nginx sends s3/index.html, and routing is handled by the js framework
GET *.css|.js|etc -> nginx forwards to the s3 bucket (all relative to index.html)
Here is my nginx.conf (it has some template things in it, cause (as part of the deploy process) I do:
envsubst '$S3_BUCKET:$NGINX_PORT' < /app/deployment/nginx.template.conf > /app/deployment/nginx.conf
pid /app/tmp/nginx.pid;
events { }
http {
upstream puma {
server unix:///app/sockets/puma.sock;
}
server {
listen ${NGINX_PORT} default_server deferred;
server_name aeonvera.com;
gzip on;
gzip_disable "msie6";
gzip_vary on;
gzip_proxied any;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;
root /app/public;
access_log /app/log/nginx.access.log;
error_log /app/log/nginx.error.log info;
client_max_body_size 20M;
keepalive_timeout 5;
location ~ ^/api {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://puma;
}
# Send all other requests to the index.html
# stored up on s3
location / {
# tell all URLs to go to the index.html
# I got an error with this about having proxy_pass within a location...?
# location ~* \.(?:ico|css|js|gif|jpe?g|png|woff2|woff|ttf)$ {
# proxy_pass "https://s3.amazonaws.com/${S3_BUCKET}/ember/"
#
# gzip_static on;
# expires max;
# add_header Cache-Control public;
# }
# Don't know what this does
rewrite ^([^.]*[^/])$ $1/ permanent;
# only ever GET these resources
limit_except GET {
deny all;
}
# use google as dns
resolver 8.8.8.8;
proxy_http_version 1.1;
proxy_set_header Host 's3.amazonaws.com';
proxy_set_header Authorization '';
# avoid passing along amazon headers
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_hide_header x-amz-delete-marker;
proxy_hide_header x-amz-version-id;
# cookies are useless on these static, public resources
proxy_hide_header Set-Cookie;
proxy_ignore_headers "Set-Cookie";
proxy_set_header cookie "";
proxy_buffering off;
# s3 replies with 403 if an object is inaccessible; essentially not found
proxy_intercept_errors on;
# error_page 500 502 503 504 /500.html;
# the actual static files
proxy_pass "https://s3.amazonaws.com/${S3_BUCKET}/ember/index.html";
}
}
}
Update 1
I've added this above location /
location ~ ^/(assets|fonts) {
rewrite (.*) $1 break;
proxy_pass "https://s3.amazonaws.com/${S3_BUCKET}/ember";
gzip_static on;
expires max;
add_header Cache-Control public;
}
but it gives an error:
nginx: [emerg] "proxy_pass" cannot have URI part in location given by regular expression, or inside named location, or inside "if" statement, or inside "limit_except" block in /app/deployment/nginx.conf:53
The thinking behind this change, is that since all my assets are in known locations, I could tell nginx to proxy to that location, and then have a rewrite (.*) / permanent; for my location /
Update 2
I thought maybe I could re-write the url for /dashboard,
so that nginx would proxy_pass the index.html. No success.
rewrite ^/(.*) / break;
proxy_pass "https://s3.amazonaws.com/${S3_BUCKET}/ember/index.html";
That just redirects to s3's marketing page
Content-Length:0
Content-Type:text/plain; charset=utf-8
Date:Thu, 08 Feb 2018 13:22:49 GMT
Location:https://aws.amazon.com/s3/
Server:nginx/1.12.0
Response Headers ^
I think I figured it out, but It'd be super helpful if someone could verify, as I'm still fairly new to nginx.
location / {
# ...
proxy_set_header Host 's3.amazonaws.com';
# ...
rewrite ^/(.*) /${S3_BUCKET}/ember/index.html break;
proxy_pass "https://s3.amazonaws.com/${S3_BUCKET}/ember/index.html";
}
Having the Host header is important here, because without it, rewrite will change the URL in the user's browser (which we don't want to do, cause that will more than likely mess with the SPA's routing.

Nginx proxy_pass returns [No Host] while reverse proxying

I'm trying to reverse proxy an api with Nginx. I have the following configuration:
worker_processes 4;
events { worker_connections 1024; }
http {
upstream some_upstream {
server 1.something.com;
server 2.something.com;
}
server {
listen 80;
location ~/proxyNow/(?<zvar>(\w+))/(?<xvar>(\w+))/(?<yvar>(\w+))/ {
proxy_pass http://some_upstream/hello/something/$zvar/$xvar/$yvar/somethingelese;
proxy_set_header Host $http_host;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_cache maps_cache;
proxy_cache_valid 200 302 365d;
proxy_cache_valid 404 1m;
proxy_redirect off;
}
}
}
When I try to call the following url http://localhost:82/proxyNow/1/2/3/?app_id=someAppId&app_code=someCode
I get the following error message:
Invalid URL
The requested URL
"http://%5bNo%20Host%5d/hello/something/1/2/3/somethingelese", is
invalid. Reference #9.be35dd58.1489086561.5c9bd3c
It seems that the host cannot be retrieved by nginx. But if I execute the call directly:
http://1.something.com/hello/something/$zvar/$xvar/$yvar/somethingelese?app_id=someAppId&app_code=someCode
http://2.something.com/hello/something/$zvar/$xvar/$yvar/somethingelese?app_id=someAppId&app_code=someCode
It seems that Nginx for some reason is not able to resolve the host
You should take a look into the doc.
http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass
When location is specified using a regular expression.
In this case, the directive should be specified without a URI.
I suggest the following solution without expensive regex location.
http://nginx.org/en/docs/http/ngx_http_core_module.html#location
location /proxyNow/ {
rewrite /proxyNow/(?<zvar>(\w+))/(?<xvar>(\w+))/(?<yvar>(\w+))/.* /hello/something/$zvar/$xvar/$yvar/somethingelese$is_args?$args break;
proxy_pass http://some_upstream;
... other nginx diretives;
}

Resources