I'm looking to improve nginx caching by removing irrelevant query parameters (that could come from web crawlers or similar) from the request. I have come across an unwieldy solution on the internet:
set $c_uri $args; # e.g. "param1=true¶m4=false"
# remove unwanted parameters one by one
if ($c_uri ~ (.*)(?:&|^)pd=[^&]*(.*)) { set $c_uri $1$2 ; }
if ($c_uri ~ (.*)(?:&|^)mid=[^&]*(.*)) { set $c_uri $1$2 ; }
if ($c_uri ~ (.*)(?:&|^)ml=[^&]*(.*)) { set $c_uri $1$2 ; }
if ($c_uri ~ (.*)(?:&|^)contact_eid=[^&]*(.*)) { set $c_uri $1$2 ; }
...
set $c_uri $scheme://$host$uri$c_uri;
...
location / {
# set $c_uri as cache_key
proxy_cache_key $c_uri;
...
}
It works, but it's not very concise, takes a lot of steps and from what I learned, if is evil.
I know there are maps, which can do basic regex things but they don't work in this scenario (because there can be any number of parameters in any order that I need to remove).
I also found this substitution module which can do regex replace but it's only made for specific operations and not for setting a variable.
So I have two questions:
Does anyone know whether there is some tooling to set a variable by doing a regex replace operation?
Is using if in this case really that bad? It's not inside a location context and I don't know whether many consecutive regexes are actually worse than one large regex replace.
I would be very thankful if someone with more nginx know-how could weigh in here and help me out. Thanks :)
Related
Please tell me the syntax to start with with $arg_name in Nginx
For example, I want to write if $arg_name starts with Test.
I want to write a syntax that matches the following
http://localhost/sss?name=Test1
http://localhost/sss?name=Test2
location /
{
if($arg_name=="Test") →I want to write the syntax starting with Test here
{
return 500;
}
}
Seems you can use regexes
This is completely untested. But maybe it will get you closer.
Using the suggestion here to use location instead of an if.
https://www.digitalocean.com/community/questions/nginx-rule-for-wildcard-url
location ~ /.*name=Test.*/ {
...
}
Here's my favorite regex playground if you don't already have one.
https://regex101.com/r/Q89zXg/2/
I have looked hi and low and found no such implementation and am wondering if what I am trying is even possible. I have 3 relative paths that serve up static content:
Path1: /usr/local/www/style1/static/...
Path2: /usr/local/www/style2/static/...
Path3: /usr/local/www/style3/static/...
The 3 different roots are static unto themselves but the content from /static on down is only semi-static (might be a bit different depending on the exact file being served up and may exist in one path and not in another). For example
/static/css/custom.css
/static/css/form/button.css
/static/css/form/images/buttondisabled.png
/static/images/ui/buttons/add_item.png
/static/images/ui/menu/help.png
The following is what I would like to do. Which is basically, if "/static" content is requested I want to check the relative path associated with path1 and if not found there then check the relative path associated with path2 and if not found check the relative path associated with path3. This seems fairly simple but I have not found any examples that outline how this might be done. Could I set the 3 different roots up as variables perhaps:
path1root /usr/local/www/style1;
path2root /usr/local/www/style2;
path3root /usr/local/www/style3;
location /static
{
try_files path1root/$uri path2root/$uri path3root/$uri (=404);
}
Or might that be done as follows since it is only needed for /static content:
location /static
{
path1root /usr/local/www/style1;
path2root /usr/local/www/style2;
path3root /usr/local/www/style3;
try_files path1root/$uri path2root/$uri path3root/$uri (=404);
}
Or can what I am attempting to do even be done at all ?? If I cannot do it with 3 roots could it be done with just 2 roots without defining one of them as an overall arching base root. If it is possible to do this and stay away from regular expressions that would be better -- but if that is needed then that is needed.
You could use a common root and try the three directories in the try_files statement:
location /static {
root /usr/local/www;
try_files /style1$uri /style2$uri /style3$uri =404;
}
My url format :
http://example.org/img?src=http://a.com/logo.png
I want nginx to rewrite this request to http://example.org/logo.png
(not url redriect)
how to do this?
The src argument is available as the $arg_src variable, however, you will need to use an if block to extract the part that you need. See this caution on using if.
For example (and you will need to adapt the regex to your specific needs):
location = /img {
if ($arg_src ~ \w(?<src>/\w.*)$) {
rewrite ^ $src last;
}
}
See nginx documentation here and a useful resource for regular expressions here.
I need to know how do i do a proxy pass in nginx for certain url pattern only
i have written following but i am not sure whether its working as i wanted. What i wanted is
1. if the url matches '/member-chat' it needs to be redirected the proxy pass
2. anything else needs to be re-written as below
is what have written is correct ?
location ^/member-chat {
proxy_pass http://lxx.com:5280/http-bind;
}
location !/member-chat {
rewrite ^/files/([^/]+)/([^/]+)$ /_files/$1/$2;
rewrite ^/plugins/([^.]+) http://www.lxx.com:9090/plugins/$1;
}
if i do this as below
location / {
rewrite ^/files/([^/]+)/([^/]+)$ /_files/$1/$2;
rewrite ^/plugins/([^.]+) http://www.lxx.com:9090/plugins/$1;
}
i get a error
nginx: [emerg] duplicate location "/" in /var/www/vhosts/system/lxx.com
/conf/vhost_nginx.conf:4
nginx: configuration file /etc/nginx/nginx.conf test failed
Several issues:
Your location #1 location ^/member-chat is wrong
because ^ to match the beginning of the path only works with regular expression matching (location ~ or location ~* for case-sensitive/-insensitive expression matching).
Either do location /member-chat which will also match locations like /member-chatABCDE or /member-chat/xyz
or use location = /member-chat to only match /member-chat.
You can also use regular expressions like location ~ ^/member-chat (prefix-match) or location ~* ^/member-chat$ (exact match), but avoiding regular expressions in favor of prefix or even better exact matches is recommended
(regular expressions have much worse performance and are compared at the very last in the matching process).
Location #2 is just plain wrong because there is nothing like a not operator for location matching.
nginx will process locations in a certain order, e.g. it will start with exact matches (=), then check for prefix-matches (no modifier) and afterwards check for regular expressions (~ or ~*).
However, if a regular expression match is found, it will be favored over the prefix-match.
Conclusion
location = /member-chat {
# exact match
# proxy stuff for chat goes here
}
location /files {
# match files
}
location /plugin {
# match plugin
}
I really recommend you to read the nginx docs to prevent you from asking one question after another.
E.g. location matching is a complex topic but so far well covered by the docs already.
I'm trying to modify my nginx access log format to include the request duration, in seconds.
I see two possible variables I could use:
1) $request_time
2) $upstream_response_time
However both of these variables are expressed in microseconds, and I need this value to be rendered in seconds. Is there any way to specify the output as an expression (i.e. $request_time * 1000) or accomplish this in some other way?
Thanks
The webserver is not a calculator or statistical program. It's logging function is to provide the raw data you can do your analysis with. If you analysis program is incapable of converting microseconds to seconds you should shop around for other software. In any case, it us unrealistic to expect a program's logging function to perform unit conversions for you. The goal of logging is not to format, yet to record what it has done without impacting the performance of it's core functionality.
If you use a reporter like LogStash (ELK stack) you can do some calculation when parsing the log. Here is my example to convert second into millisecond in my Logstash filter for Nginx:
grok {
match => {
"message" => "%{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"%{WORD:verb} %{URIPATHPARAM:logMessage} HTTP/%{NUMBER:httpversion}\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) (?:\"(?:%{URI:referrer}|-)\"|%{QS:referrer}) %{QS:agent} rt=%{NUMBER:duration_sec} cid=%{GREEDYDATA:correlationId}"
}
}
mutate { convert => [ "duration_sec", "float" ] }
ruby { code => "event['duration_ms'] = event['duration_sec'].to_f * 1000" }
Hope this helps.
As noted in the comments $request_time is already in seconds, however it is possible to convert to another unit purely in nginx config as follows.
I realise this is an old question but it seems to get a lot of traffic so perhaps the below will help someone else who, like me, might want to convert to a different unit, in my case nanoseconds (though if you want milliseconds then simply omit the 6 trailing zeros).
This is useful for sending to the Elastic event.duration field - to avoid the faff of an ingest pipeline or similar on the Elastic end (to multiply by 1000000) you can instead do some hideous regex as below.
The fact that $request_time is always zero-padded to 3 decimal places helps, so in the common cases you can append 6 zeros - but you also have to handle leading zeros (before & after the decimal point) to make sure the result doesn't end up with any leading zeros:
map $request_time $request_time_nanos {
# Simple case for 0.000
~^0\.000$ 0;
# If 0 before decimal place, must remove leading zeros after it, before adding 6 zeros
~^0\.(?:0*)([^0].*)$ $1000000;
# Otherwise just concatenate the full pre- & post-decimal parts, before adding 6 zeros
~^([^0][^.]*)\.(.*)$ $1$2000000;
}
Example transformations (commas inserted for readability only):
[sec] [nanoseconds]
0.000 => 0
0.110 => 110,000,000
0.010 => 10,000,000
1.010 => 1,010,000,000