In Flask, if you place a file in a directory called static/, then any URL of the form http://localhost/static/foo.jpg will serve that file from static/foo.jpg.
This can also be accomplished via an nginx config:
location /static {
alias /var/www/mywebsite/static;
}
However, I want to do dynamic URL rewriting.
If someone requests the URL http://localhost/username/foo.jpg, I want to tell nginx to fetch the static file from an arbitrary URL, say, /var/www/assets/11235/1bcd5.jpg. I want the user to see a pretty url, and I want the location to be transparent to the user.
Is there an easy way to do this? Ideally, I would be able to do something so that nginx serves the file. However, if Flask needs to serve it, then that is fine too (it isn't like my project has any users yet!)
What am I missing here?
If the files can be stored with the names that are directly referenced in the "pretty" URL, then you can do a simple rewrite in nginx.
However, it appears that you want to map URL path info to other representations on the disk, as in username -> 11235 and foo.jpg -> 1bcd5.jpg. If the content being served should be protected by authentication or sessions, then you should probably keep the mapping and rewriting inside your Flask app, since Flask provides the means to do that.
If the content can be treated as public and only needs the mapping done, then nginx can be configured to grab query string parameters, look them up in a datastore, and rewrite the URL.
Here's an example from agentzh that was originally posted on the nginx mailing list:
Consider you seo uri is /baz, the true uri is /foo/bar. And I have the
following table in my local mysql "test" database:
create table my_url_map(id serial, url text, seo_url);
insert into my_url_map(url, seo_url)values('/foo/bar', '/baz');
And I build my nginx 0.8.41 this way:
./configure \
--add-module=/path/to/ngx_devel_kit \
--add-module=/path/to/set-misc-nginx-module \
--add-module=/path/to/ngx_http_auth_request_module-0.2 \
--add-module=/path/to/echo-nginx-module \
--add-module=/path/to/lua-nginx-module \
--add-module=/path/to/drizzle-nginx-module \
--add-module=/path/to/rds-json-nginx-module
Also, I have lua 5.1.4 and the lua-yajl library installed to my system.
And here's the central part in my nginx.conf:
upstream backend {
drizzle_server 127.0.0.1:3306 dbname=test
password=some_pass user=monty protocol=mysql;
drizzle_keepalive max=300 mode=single overflow=ignore;
}
lua_package_cpath '/path/to/your/lua/yajl/library/?.so';
server {
...
location /conv-mysql {
internal;
set_quote_sql_str $seo_uri $query_string; # to prevent sql injection
drizzle_query "select url from my_url_map where seo_url=$seo_uri";
drizzle_pass backend;
rds_json on;
}
location /conv-uid {
internal;
content_by_lua_file 'html/foo.lua';
}
location /jump {
internal;
rewrite ^ $query_string? redirect;
}
# your SEO uri
location /baz {
set $my_uri $uri;
auth_request /conv-uid;
echo_exec /jump $my_uri;
}
}
Contents of foo.lua, the essential glue:
local yajl = require('yajl')
local seo_uri = ngx.var.my_uri
local res = ngx.location.capture('/conv-mysql?' .. seo_uri)
if (res.status ~= ngx.HTTP_OK) then
ngx.throw_error(res.status)
end
res = yajl.to_value(res.body)
if (not res or not res[1] or not res[1].url) then
ngx.throw_error(ngx.HTTP_INTERNAL_SERVER_ERROR)
end
ngx.var.my_uri = res[1].url;
Then let's access /baz from the client side:
$ curl -i localhost:1984/baz
HTTP/1.1 302 Moved Temporarily
Server: nginx/0.8.41 (without pool)
Date: Tue, 24 Aug 2010 03:28:42 GMT
Content-Type: text/html
Content-Length: 176
Location: http://localhost:1984/foo/bar
Connection: keep-alive
<html>
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx/0.8.41 (without pool)</center>
</body>
</html>
Related
I have two servers:
NGINX (it exchanges file id to file path)
Golang (it accepts file id and return it's path)
Ex: When browser client makes request to https://example.com/file?id=123, NGINX should proxy this request to Golang server https://go.example.com/getpath?file_id=123, which will return the response to NGINX:
{
data: {
filePath: "/static/..."
},
status: "ok"
}
Then NGINX should get value from filePath and return file from the location.
So the question is how to read response (get filePath) in NGINX?
I assume you are software developer and your have full control over your application so there is no need to force square peg in a round hole here.
Different kinds of reverse proxies support ESI(Edge Side Includes) technology which allow developer to replace different parts of responce body with content of static files or with response bodies from upstream servers.
Nginx has such technology as well. It is called SSI (Server Side Includes).
location /file {
ssi on;
proxy_pass http://go.example.com;
}
Your upstream server can produce body with content <!--# include file="/path-to-static-files/some-static-file.ext" --> and nginx will replace this in-body directive with content of the file.
But you mentioned streaming...
It means that files will be of arbitrary sizes and building response with SSI would certainly eat precious RAM resources so we need a Plan #B.
There is "good enough" method to feed big files to the clients without showing static location of the file to the client.
You can use nginx's error handler to server static files based on information supplied by upstream server.
Upstream server for example can send back redirect 302 with Location header field containing real file path to the file.
This response does not reach the client and is feed into error handler.
Here is an example of config:
location /file {
error_page 302 = #service_static_file;
proxy_intercept_errors on;
proxy_set_header Host $host;
proxy_pass http://go.example.com;
}
location #service_static_file {
root /hidden-files;
try_files $upstream_http_location 404.html;
}
With this method you will be able to serve files without over-loading your system while having control over whom do you give the file.
For this to work your upstream server should respond with status 302 and with typical "Location:" field and nginx will use location content to find the file in the "new" root for static files.
The reason for this method to be of "good enough" type (instead of perfect) because it does not support partial requests (i.e. Range: bytes ...)
Looks like you are wanting to make an api call for data to run decision and logic against. That's not quite what proxying is about.
The core proxy ability of nginx is not designed for what you are looking to do.
Possible workaround: extending nginx...
Nginx + PHP
Your php code would do the leg work.
Serve as a client to connect to the Golang server and apply additional logic to the response.
<?php
$response = file_get_contents('https://go.example.com/getpath?file_id='.$_GET["id"]);
preg_match_all("/filePath: \"(.*?)\"/", $response, $filePath);
readfile($filePath[1][0]);
?>
location /getpath {
try_files /getpath.php;
}
This is just the pseudo-code example to get it rolling.
Some miscellaneous observations / comments:
The Golang response doesn't look like valid json, replace preg_match_all with json_decode if so.
readfile is not super efficient. Consider being creative with a 302 response.
Nginx + Lua
sites-enabled:
lua_package_path "/etc/nginx/conf.d/lib/?.lua;;";
server {
listen 80 default_server;
listen [::]:80 default_server;
location /getfile {
root /var/www/html;
resolver 8.8.8.8;
set $filepath "/index.html";
access_by_lua_file /etc/nginx/conf.d/getfile.lua;
try_files $filepath =404;
}
}
Test if lua is behaving as expected:
getfile.lua (v1)
ngx.var.filepath = "/static/...";
Simplify the Golang response body to just return a bland path then use it to set filepath:
getfile.lua (v2)
local http = require "resty.http"
local httpc = http.new()
local query_string = ngx.req.get_uri_args()
local res, err = httpc:request_uri('https://go.example.com/getpath?file_id=' .. query_string["id"], {
method = "GET",
keepalive_timeout = 60,
keepalive_pool = 10
})
if res and res.status == ngx.HTTP_OK then
body = string.gsub(res.body, '[\r\n%z]', '')
ngx.var.filepath = body;
ngx.log(ngx.ERR, "[" .. body .. "]");
else
ngx.log(ngx.ERR, "missing response");
ngx.exit(504);
end
resty.http
mkdir -p /etc/nginx/conf.d/lib/resty
wget "https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http_headers.lua" -P /etc/nginx/conf.d/lib/resty
wget "https://raw.githubusercontent.com/ledgetech/lua-resty-http/master/lib/resty/http.lua" -P /etc/nginx/conf.d/lib/resty
I want to create a link in a banner to redirect to one of two pages. The link points to subdomain.example.com/email. If the browser language is "de" then go to www.example.de/banner else go to www.example.com/banner. My nginx conf language skills are not good, but I know that any browser language of German has the first two characters are "de" (see https://www.metamodpro.com/browser-language-codes). No other language has this.
location /email {
if $http_accept_language === "de" { return 301 https://www.example.de/banner }
else { return 301 https://www.example.com/banner}
}
A cleaner and extensible solution uses the map directive.
For example:
map $http_accept_language $redirect {
default https://www.example.com/banner;
~de https://www.example.de/banner;
}
server {
...
location /email {
return 301 $redirect;
}
...
}
See this document for details.
A similar use case where the locale is to be determined by the browser. There are only two languages offered, so I use http_accept_language for this. A standard language has been defined, in this case it is English.
map $http_accept_language $lang {
default en;
~de de;
}
server {
...
rewrite ^/$ /$lang/ redirect;
...
}
After a reload of the configuration the behaviour can be checked with curl.
curl -I https://www.your-site.com/ -H "Accept-Language: fr"
curl -I https://www.your-site.com/ -H "Accept-Language: de-CH"
curl -I https://www.your-site.com/ -H "Accept-Language: en-US"
Now tests the server repsonse, the header parameter location now shows the correct suffix depending on the Accept-Language.
location: https://www.your-site.com/en/
Short note, Accept-Language allows a listing using q-list, e.g. de, en-US;q=0.9, es;q=0.1. The above configuration does not support this. In my case this was not necessary at all.
https://www.w3.org/International/questions/qa-accept-lang-locales.en
I have 2 Nginx servers serving static files from 2 different subdomains of an unknown parent domain, let's say <env>.foo.<domain>.com and <env>.bar.<domain>.com.
I want to configure the nginx server for <env>.foo.<domain>.com so that if the url has the subdirectory cat or dog I want to redirect to <env>.bar.<domain>.com/<subdirectory>/<rest of url>.
E.g.
http://dev.foo.mydomain1.com/cat/22 -> http://dev.bar.mydomain1.com/cat/22
http://dev.foo.mydomain1.com/dog/22 -> http://dev.bar.mydomain2.com/dog/22
http://dev.foo.mydomain2.com/dog/22 -> http://dev.bar.mydomain2.com/dog/22
http://dev.foo.mydomain1.com/bird/22 -> [no redirect]
The <env> and <domain> portions of the domain are dynamic depending to the environment to which the servers are deployed, but are common between the 2 nginx boxes.
I imagine it being something like:
server {
location ??? /(cat|dog) {
return 301 $scheme://???/$1$is_args$query_string;
}
}
But my nginx skills are not quite there...
rewrite ^/oldlocation$ http://www.newdomain.com/newlocation permanent;
Refer digital oceans's article on "How To Create Temporary and Permanent Redirects with Nginx" for further details.
What you can do is below
http {
map $http_host $host_to_send_to {
default $http_host;
dev.foo.mydomain1.com dev.bar.mydomain1.com;
dev.foo.mydomain2.com dev.bar.mydomain2.com;
}
server {
server_name dev.foo.mydomain1.com;
location /(cat|dog) {
return 301 $scheme://$host_to_send_to$request_uri$is_args$query_string;
}
}
}
You will need to add it to each server block which needs to handle the redirect.
Edit-1: Dynamic host name handling
You can handling host names dynamically also user re patterns and groups
map $hostsname $host_to_send_to {
default $http_host;
~(.*).foo.mydomain1.com $1.bar.mydomain1.com;
~(.*).foo.mydomain2.com $1.bar.mydomain2.com;
}
Original URL: /api/url%2Fencoded%2F/?with=queryParams
nginx:
location /api {
client_max_body_size 2G;
proxy_pass https://oursite;
}
With this configuration, I was able to preserve the URL encoding when passing through the proxy. If I add a "/" after "oursite", it will decode the URL.
Problem:
Now the URL after being proxied still contains "/api/". I need to remove "/api/" only while still preserving the URL encoded parts.
Not a long time ago there was identical question without an answer. In my opinion, you should rething api to not have such weird URLs. Another way is to have api on subdomain. – Alexey Ten Mar 11 '15 at 22:58
stackoverflow.com/q/28684300/1016033 – Alexey Ten Mar 11 '15 at 23:01
Year-old challenge accepted!
location /api/ {
rewrite ^ $request_uri;
rewrite ^/api/(.*) $1 break;
return 400;
proxy_pass http://127.0.0.1:82/$uri;
}
That's it, folks!
More details at Nginx pass_proxy subdirectory without url decoding, but it does work even with the query string, too:
% curl "localhost:81/api/url%2Fencoded%2F/?with=queryParams"
/url%2Fencoded%2F/?with=queryParams
%
Disclaimer: I am sure this looks like an hack - and maybe it is. It is using the auth-subrequest feature for something else than auth, but it works!
If you want to keep any url-encoded part after /api/ from the original $request_uri I use NJS to set a variable and use it afterwards in the proxy_pass
js_import /etc/nginx/conf.d/http.js; # Import your njs file here
js_set $encodedUrlPart 'empty'; # Define a variable
location ~* api\/(.*)$ {
auth_request /urlencode; #This will get executed before proxy_pass
proxy_pass http://127.0.0.1:82/$encodedUrlPart;
}
and the http.js can look like this
function urlencode(r){
let regex = "(?<=\/api\/)(.*$)";
let url = r.variables.request_uri # this holds the original, non touched url
let lastPart = url.match(regex);
r.variables.encodedUrlPart = lastPart;
r.log("The encoded url part: " + r.variables.encodedUrlPart);
r.return(200); // need to return 200 so the 'auth' doesn't fail
}
export default {urlencode};
Is this considered unsafe? We could do some checking in the njs part though!
I have nginx 1.0.8 installed.
here is my problem:
I have 2 files : file1.js and file2.js. the requested path is something like this:
www.mysite.com/files_dir/%user%/file.js
If the requested header : "X-Header" exists and has the value "OK" then the responded content should be file1.js else file2.js.
The files are situated in "html/files_dir" and %user% is a set of directories that represents the usernames registered through my service.
How do I configure this in nginx? I'm not interested in php, asp or similar technologies only if it's possible with nginx.
Thanks
map lets you define a variable's value based on another variable. map should be declared at http level (i.e. outside of server):
map $http_x_header $file_suffix {
default "2";
OK "1";
};
Then the following location should do the trick using your new variable $file_suffix
location ~ ^(/files_dir/.+)\.js$ {
root html;
try_files $1$file_suffix.js =404;
}
You could do this with nginx very easily. This is example:
location /files_dir/ {
set $file = file2.js;
if ( $http_x_header = OK ) {
set $file = file1.js;
}
rewrite ^(/files_dir/.*)/file.js$ $1/$file last;
}
You could read about HTTP variables in NGINX here , and about nginx rewrite module here