I am having a problem with my $memcached_keys being too long in my .conf file for nginx. I am using the memcached module but some of my urls are too long. I am in the process of trying to user MD5 hashes of the urls instead but in the meantime I was just wondering if there was a way I could check on the length of a string stored in a variable.
so:
set $memcached_key "byp-$uri";
if ($args) {
set $memcached_key "byp-$uri?$args";
}
if (len($memcache_key) < 250) {
memcached_pass 127.0.0.1:11211;
error_page 404 = #cache_miss;
error_page 502 = #cache_miss;
}
else {
pass to #cache_miss;
}
Old question but ....
To do this sort of thing, you need a scripting setup such as the Lua Module:
location / {
set_by_lua $memcached_key '
if not ngx.var.args then
return "byp-" .. ngx.var.uri
else
return "byp-" .. ngx.var.uri .. "?" .. ngx.var.args
end
';
content_by_lua '
local string = string;
if string.len($memcached_key) < 250 then
ngx.exec("/memcached");
else
ngx.exec("/cache_miss");
end
';
}
location /memcached {
internal;
memcached_pass 127.0.0.1:11211;
error_page 404 = /cache_miss;
error_page 502 = /cache_miss;
}
location /cache_miss {
internal;
...
}
Suggest "internal" locations instead of named locations due to a few quirks with the latter but named locations can be used as well.
I'm not positive, but I don't think it can be done within the nginx config language. I suspect it would be documented here if it existed, and I don't see anything.
Related
How to forward the URL's all parameters through a proxy_pass with nginx?
Nginx config:
location /proxy/ {
if ($request_method = HEAD) { return 200; }
if ( $arg_address != "" ) {
proxy_pass $arg_address;
return 301 $arg_address;
}
proxy_ssl_verify off;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
}
this urls works:
https://localhost/proxy/?address=https://exemple.com/transfer/file.txt ==>> https://exemple.com/transfer/file.txt
or
https://localhost/proxy/?address=https://exemple.com/transfer/file.txt?host-id=1 ==>> https://exemple.com/transfer/file.tx?host-id=1
if I add multiple parameters, it will be truncated to the first "&"
https://localhost/proxy/?address=https://exemple.com/transfer/file.txt?host-id=1&password=123456&date=xxxxxx
==>> https://exemple.com/transfer/file.txt?host-id=1
How can I transfer the entire url?
Before we get to the answer, may I ask what do you want to achieve? Do you want to proxy the request or to generate HTTP 301 redirect? With the following construction
if ( $arg_address != "" ) {
proxy_pass $arg_address;
return 301 $arg_address;
}
you'll always get a redirect because the directives from ngx_http_rewrite_module are executed before any others, so the proxy_pass directive is useless here. ngx_http_rewrite_module is very special and different from most of the other modules. Although nginx configuration in general is declarative, rewrite module evaluates its instructions imperatively. This is always a source of confusion for every nginx novice. You can read more about the rewrite module internal implementation here.
If you want to proxy the request instead of generating a redirect, you'll need to remove return and add a resolver directive to your configuration. Here you can read why it is required.
A "dirty hack" solution
Being that said, get back to the question. Of course, when nginx receives the request
https://localhost/proxy/?address=https://example.com/transfer/file.txt?host-id=1&password=123456&date=20210520
arg_NAME variables will be filled the following way:
arg_address => https://example.com/transfer/file.txt?host-id=1
arg_password => 123456
arg_date => 20210520
It is correct and expected behavior.
What you can do to preserve all the other query arguments? The most simple is to assume that all query arguments following the address one are subject to pass to the upstream and use a map directive to get the required string:
map $args $address {
~(?:^|&)(address=.*) $1;
}
server {
...
location /proxy/ {
...
if ($address) {
# 'proxy_pass $address' or 'return 301 $address' here
}
...
}
...
}
Here is more strict check where we take the rest of the query string only if there is a question mark after address query parameter and only $arg_address value otherwise:
map $args $address {
~(?:^|&)(address=[^&?]+\?.*) $1;
default $arg_address;
}
Reliable solution
While the answer above is generally workable, I'd rather try to design my proxy solution using URL encoding on address query argument to avoid reserved characters usage as part of the query argument value. The above request being URL-encoded would look like
https://localhost/proxy/?address=https%3A%2F%2Fexample.com%2Ftransfer%2Ffile.txt%3Fhost-id%3D1%26password%3D123456%26date%3D20210520
The bad thing is that "vanilla" nginx doesn't have an ability to URL-decode an arbitrary string. However it can be done using OpenResty/lua-nginx-module:
location /proxy/ {
...
set_by_lua_block $address { return ngx.unescape_uri(ngx.var.arg_address) }
if ($address) {
# 'proxy_pass $address' or 'return 301 $address' here
}
...
}
or set-misc-nginx-module:
location /proxy/ {
...
if ($arg_address) {
set_unescape_uri $address $arg_address;
# 'proxy_pass $address' or 'return 301 $address' here
}
...
}
Perhaps the same can be done using njs, but I didn't use it and can't give you an example.
I have a requirement to do a proxy call to url delivered via a query parameter as per example:
My nginx proxy is deployed at: https://myproxy.net
if the redirect parameter is not url encoded I can do the call with this block:
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass $arg_redirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}
the error intercepts and #handle_redirects then take care of othe 30X codes that might pop up at new destination.
This works for a request:
GET: https://myproxy.net/basepath?redirect=https://destination.com/somepath/uuid
What do I need to do to make it work for:
GET: https://myproxy.net/basepath?redirect=https%3A%2F%2Fdestination.com%2Fsomepath%2Fuuid
Additionally as part of spec it has to be pure nginx, not additional modules, lua etc.
Thanks!
Actually, proxy_pass does normalisation by default, but it only affects $uri part. Thus you only need to decode the beginning of the passed string to get it working:
location / {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
if ( $arg_redirect ~ (.+)%3A%2F%2F(.+) ){ # fix :// between scheme and destination
set $arg_redirect $1://$2;
}
if ( $arg_redirect ~ (.+?)%3A(.*) ){ # fix : between destination and port
set $arg_redirect $1:$2;
}
if ( $arg_redirect ~ (.+?)%2F(.*) ){ # fix / after port, the rest will be decoded by proxy_pass
set $arg_redirect $1/$2;
}
proxy_pass $arg_redirect;
}
With the above I managed to access http://localhost/?redirect=http%3A%2F%2F127.0.0.1%3A81%2Fsfoo%20something%2Fs
The solution seems dirty and the only alternative using default modules is map (even less cleaner in my opinion). I'd rather split redirect argument into pieces: scheme (http or https), destination, port, and uri. With that you would be able to construct full address without rewriting:
proxy_pass $arg_scheme://$arg_dest:$arg_port/$arg_uri
Ok, there is very weird and curious solution
server {
listen 80;
resolver x.x.x.x;
location /basepath {
if ($arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
proxy_pass http://127.0.0.1:80/basepath/$arg_redirect;
}
location ~ ^/basepath/(?<proto>\w+):/(?<redir>.+)$ {
proxy_pass $proto://$redir;
}
}
Nginx does not encode path with variables in proxy_pass and send it as is. So, I make $arg_* part of proxy_pass uri, send request to self and nginx will receive new request which will be decoded.
But because Nginx will clean path and replace // to / I split protocol part in regexp.
And ... I would never recommend using this solution, but it works :)
try like this and let me know if it works
location /basepath {
if ( $arg_redirect = '') {
return 400 "Missing redirect directive in request";
}
set_unescape_uri $decodedredirect $arg_redirect;
proxy_pass $decodedredirect;
proxy_intercept_errors on;
error_page 301 302 307 = #handle_redirects;
}
I am trying to resolve proxy_pass value dynamically (through web api) in nginx.
I need something like below;
Example taken from: https://www.nginx.com/resources/wiki/start/topics/depth/ifisevil/
location /proxy-pass-uri {
set $urlToProxy = CallWebAPI("http://localhost:8081/resolver?url=" + $url);
proxy_pass $urlToProxy;
}
So, my question is that, is it possible to make HTTP request or to write method such as CallWebAPI?
I know it might be a bad practice, but the website I am dealing with has thousands of web urls, which are mapped as key-value pairs, and 90% of them does not obey any specific regex rules. So I have content mapped database, and I need to fetch incoming url with content dynamically.
I am trying to use a very light web service to look up URLs from redis, and return proxy url.
Would this be a valid scenario, or is there any other built in solution in nginx like this?
I doubt this can be done with "pure" nginx, but this definitely can be done with openresty or ngx_http_lua_module with the help of ngx.location.capture method. For example:
resolver 8.8.8.8;
location ~/proxy-pass-uri(/.*)$ {
set $url $1;
set $proxy "";
access_by_lua_block {
res = ngx.location.capture("http://localhost:8081/resolver?url=" .. ngx.var.url)
ngx.var.proxy = res.body
}
proxy_pass $proxy$url;
}
There is also an ngx_http_js_module (documentation, GitHub) which have an ability to do subrequests (example), but I never used it and cannot tell if it can be used this way.
Important update
After almost a three years since this answer was written, it comes that I needed the similar functionality myself, and it turns out that the above answer is completely broken and unworkable. You can't do a subrequest via ngx.location.capture to anything else but to some other nginx location. So the correct (checked and confirmed to be workable) example for the above question is
resolver 8.8.8.8;
location /resolver {
internal;
proxy_pass http://localhost:8081;
}
location ~ ^/proxy-pass-uri(/.*)$ {
set $url $1;
set $proxy "";
access_by_lua_block {
res = ngx.location.capture("/resolver?url=" .. ngx.var.url)
if res.status == ngx.HTTP_OK then
ngx.var.proxy = res.body
else
ngx.exit(res.status)
end
}
proxy_pass $proxy$url$is_args$args;
}
The above example assumes that the proxy resolution service is really expecting request in a /resolver?url=<uri> form. The location /resolver { ... } while being internal behaves like any other prefix location, so if the /resolver prefix for that location cannot be used for some reason, the same can be written as
resolver 8.8.8.8;
location /get_proxy {
internal;
proxy_pass http://localhost:8081/resolver;
}
location ~ ^/proxy-pass-uri(/.*)$ {
set $url $1;
set $proxy "";
access_by_lua_block {
res = ngx.location.capture("/get_proxy?url=" .. ngx.var.url)
if res.status == ngx.HTTP_OK then
ngx.var.proxy = res.body
else
ngx.exit(res.status)
end
}
proxy_pass $proxy$url$is_args$args;
}
I have the following configuration of nginx that hosts my image service:
upstream thumbor {
server localhost:8888;
}
server {
listen 80;
server_name my.imageserver.com;
client_max_body_size 10M;
rewrite_log on;
location ~ /images {
if ($arg_width="10"){
rewrite ^/images(/.*)$ /unsafe/$1 last;
}
rewrite ^/images(/.*)$ /unsafe/$1 last;
}
location ~ /unsafe {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header HOST $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://thumbor;
proxy_redirect off;
}
location = /favicon.ico {
return 204;
access_log off;
log_not_found off;
}
}
I am trying to rewrite the following urls:
from
http://my.imageserver.com/images/Jenesis/EmbeddedImage/image/jpeg/jpeg/9f5d124d-068d-43a4-92c0-1c044584c54a.jpeg
to
http://my.imageserver.com/unsafe/Jenesis/EmbeddedImage/image/jpeg/jpeg/9f5d124d-068d-43a4-92c0-1c044584c54a.jpeg
which is quite easy, the problem begins when I want to allow query string that should go to the path of the url like so:
from
http://my.imageserver.com/images/Jenesis/EmbeddedImage/image/jpeg/jpeg/9f5d124d-068d-43a4-92c0-1c044584c54a.jpeg?width=150&height=200&mode=smart
to
http://my.imageserver.com/unsafe/150x200/smart/Jenesis/EmbeddedImage/image/jpeg/jpeg/9f5d124d-068d-43a4-92c0-1c044584c54a.jpeg
Also it will be better if the order of the query strings won't matter.
I tried using:
$arg_width but it didn't seem to work.
Using nginx 1.6.1 on ubuntu.
Help would be much much appreciated.
Working with query arguments is always hard (this is also true with Apache).
But in your example when you do:
location ~ /images {
if ($arg_width="10"){
rewrite ^/images(/.*)$ /unsafe/$1 last;
}
rewrite ^/images(/.*)$ /unsafe/$1 last;
}
I do not see any difference between the 2 rewrites... so that's maybe why it is not working.
Anyway you could maybe try something like that (based on this thread):
location ^~ /images {
# get there only if we have a query string
if ($is_args) {
set $width "";
set $height "";
if ($args ~* "(?:^|&)width=([^&]+)") {
set $width $1;
}
if ($args ~* "(?:^|&)height=([^&]+)") {
set $height $1;
}
# string concatenation using sort of bash syntax
set $dim "${width}x${height}";
# maybe we should add a control here on $dim !='x'...
# the ? here prevent query string from being appended
rewrite ^/images(/.*)$ /unsafe/$dim/$1? last;
}
rewrite ^/images(/.*)$ /unsafe/$1 last;
}
location ~ /unsafe {
(...)
You can use the arg_name parameter, and you can get rid of location block:
rewrite ^/images/(.*)$ /unsafe/$1;
# WARNING: Here we suppose that when we have a "mode" parameter, we have
# width and height paremeters too
if ($arg_mode) {
rewrite ^/unsafe/(.*)$ /images/unsafe/smart/$arg_mode/${arg_width}x${arg_height}/$1 last;
}
You need as scripting option that can evaluate the input and construct the rewrite URL. I use the third party Nginx Lua Module for such logic.
You will need to compile the module into Nginx after installing Lua or preferably, LuaJit on your machine.
Alternatively, you can install Openresty which is Nginx bundled with a few modules that completely transforms the Nginx into a full web application server. Openresty will take care of Lua/LuaJit dependencies during installation.
If you have this in place, this should do the job for you:
upstream thumbor {
server localhost:8888;
}
server {
[...]
location ~ ^/images {
rewrite_by_lua '
-- Use local variables to limit their scope to this location block
local i, j, key, val, args, dimension, modetype, tempheight, replacement
-- We only go ahead if there are request arguments to start with
if ngx.var.is_args then
-- Get the request arguments
-- These are loaded into a Lua table of the form, { a = 1, b = 2, c = 3 }
args = ngx.req.get_uri_args()
-- Loop through the args table and build our replacement string
for key, val in pairs(args) do
if key == "width" then
if tempheight then
dimension = val .. "x" .. tempheight
else
dimension = val
end
elseif key == "height" then
if dimension then
dimension = dimension .. "x" .. val
else
tempheight = val
end
elseif key == "mode" then
modetype = val
end
end
-- Construct the replacement string.
replacement = "/unsafe/" .. dimension .. "/" .. modetype .. "/"
-- Replace "/images/" in the request url with the replacement string.
local newuri, n, err = ngx.re.sub(ngx.unescape_uri(ngx.var.request_uri), "/images/", replacement, "io")
-- If there is a new string, then redirect to the new URL
if newuri then
return ngx.redirect(newuri, 301)
end
end
';
}
location ~ /unsafe {
[...]
proxy_pass http://thumbor;
}
[...]
}
I'm trying to figure out how to do the following:
Request is coming in.
HttpLuaModule performs some action against the request. If request is valid than Lua will finish processing with ngx.exit(202). But there are some conditions that may (and will) occur during the processing and nginx might return 403 , 404, 503 Errors.
What I want to do is to write to access logs only requests that have 200 Status code.
Basically I would like to do something like this:
location /foo {
content_by_lua_file "/opt/nginx/lua/process.lua";
if (status == 200) {
access_log "/path/to/the/access_log"
}
I'm very new to both nginx and lua so for me it's a bit of a challenge to figure out where to place and if statement (ether after content_by_lua_file or in side lua file) and what this if statement should look like.
nginx 1.7.0+ allows using an if condition in access_log directive itself.
access_log path [format [buffer=size [flush=time]] [if=condition]];
The if parameter (1.7.0) enables conditional logging.
A request will not be logged if the condition evaluates to “0” or an empty string
Combined with map directive its possible to send log events to different logs based on various conditions.
http {
map $status $normal {
~^2 1;
default 0;
}
map $status $abnormal {
~^2 0;
default 1;
}
map $remote_addr $islocal {
~^127 1;
default 0;
}
server {
access_log logs/access.log combined if=$normal;
access_log logs/access_abnormal.log combined if=$abnormal;
access_log logs/access_local.log combined if=$islocal;
}
}
http://nginx.org/en/docs/http/ngx_http_log_module.html
http://nginx.org/en/docs/http/ngx_http_map_module.html
you can do it by using ngx.log and log_by_lua directives.
location /conditional_log{
log_by_lua 'if ngx.status == 200 then ngx.log(ngx.ERR, "It is 200") end';
content_by_lua 'ngx.say("I am ok") ngx.exit(200)';
}
In the above code, we use log_by_lua which is called while running in log phase. In that if ngx.status == 200, we use ngx.log to trigger the logging using ngx.log.
This will write to error_log. Not sure how to write it to access_log.
For reference
http://wiki.nginx.org/HttpLuaModule#ngx.log
http://wiki.nginx.org/HttpLuaModule#log_by_lua
In every question is a part of answer. You were very close:
if ($status != "200") {
access_log off;
}
Check info for version availability here.
http://nginx.org/en/docs/http/ngx_http_core_module.html#variables
Also, almost all access log format vars are available in "modern" versions:
http://nginx.org/en/docs/http/ngx_http_log_module.html
This is the solution I came up with:
auth.lua
-- Some logic goes here
-- ....
-- ....
ngx.var.return_status = 200
nginx.conf
http {
lua_package_path .....;
lua_package_cpath ....;
rewrite_by_lua_no_postpone on;
server {
set $return_status 1;
location /foo {
rewrite_by_lua_file "<apth_to_aut.lua";
if ($return_status = 200) {
access_log <path_to_access_log> format;
return 200;
}
}
}
}