nginx hook before upload - nginx

I've created an intranet http site where users can upload their files, I have created a location like this one:
location /upload/ {
limit_except POST { deny all; }
client_body_temp_path /home/nginx/tmp;
client_body_in_file_only on;
client_body_buffer_size 1M;
client_max_body_size 10G;
proxy_set_header X-upload /upload/;
proxy_set_header X-File-Name $request_body_file;
proxy_set_body $request_body_file;
proxy_redirect off;
proxy_pass_request_headers on;
proxy_pass http://localhost:8080/;
}
Quite easy as suggested in the official doc. When upload is complete the proxy_pass directive calls the custom URI and makes filesystem operations on newly created temp file.
curl --request POST --data-binary "#myfile.img" http://myhost/upload/
Here's my problem: I need to have some kind of custom hook/operation telling me when the upload begins, something nginx can call before starting the http stream, is there a way to achieve that ? I mean, before uploading big files I need to call a custom url (something like proxy_pass) to inform the server about this upload and execute certain operations.
Is there a way to achieve it ? I have tried with echo-nginx module but it didn't succeed with these http POST (binary form-urlencoded). I don't want to use external scripts to deal with the upload and keep these kind of operations inside nginx (more performant)
Thanks in advance.
Ben

Self replying.
I have found this directive in order to solve my own request.
auth_request <something>
So I can do something like:
location /upload/ {
...
# Pre auth
auth_request /somethingElse/;
...
}
# Newly added section
location /somethingElse/ {
...
proxy_pass ...;
}
This seems to be fine and working, useful for uploads as well as for general auth or basic prechecks

Related

Multiple "relays" in NGINX - is it possible?

Is it possible to configure NGINX to something like multiple reverse-proxy? So, instead of one proxy_pass:
location /some/path/ {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://localhost:8000;
}
to have multiple proxy_passs (relays)? I need something like load balancer but to send not to one of them, but TO ALL (see the scheme below).
TO BE ACCURATE: IT'S NOT REVERSE PROXY AND IT IS NOT LOAD BALANCING AS WELL.
The response may be retrieving from any of them, maybe the last one or even to be "hardcoded" with some configuration directives - it does not matter (it's enough to be HTTP 200 and will be ignored)... So, the scheme should look like:
.----> server 1
/
<---> NGINX <-----> server 2 (response from 2nd server, but it maybe any of them!)
\
`----> server 3
Maybe some extension for NGINX? Is it possible at all and how to do it if it is?
Yes, it is possible.
What you are searching for is called mirroring. And nginx implements it since version 1.13.4, see the directive mirror for more info.
Example:
location = /mirror1 {
internal;
proxy_pass http://mirror1.backend$request_uri;
}
location = /mirror2 {
internal;
proxy_pass http://mirror2.backend$request_uri;
}
...
location /some/path/ {
mirror /mirror1;
mirror /mirror2;
proxy_pass http://primary.backend;
}
(you can also specify it for whole server (or even http) and disable for locations where you don't need it).
Alternatively you could try post_action (but this is undocumented feature and if I remember correctly is deprecated).

How do you send an auth_request to a variable URI in nginx?

I am trying to use the auth_request module to check whether a user is allowed to access a certain file. The user posts the request at /my/download/uri/<File ID>. I want the authorisation request to be posted at auth_service:9999/files/<File ID>. The relevant part of my config is as follows:
location /my/download/uri {
auth_request /auth/files/$uri;
alias /my/file/directory;
}
location /auth {
internal;
proxy_pass http://auth_service:9999/;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
}
The request is received by the authorisation service, but at literally /files/$uri; the variable is not placed. I have tried getting the URI ready via a set variable first, but to no avail. How can I get nginx to properly direct the authorisation request?
(Note: I am aware I can include the original request in the header of the authorisation request via X-Original-URI. However, this would mean I have to do additional processing of the full URI on the authorisation server to get the relevant data, which I would rather not do if there is a way to post the authorisation request to the correct URI in the first place.)
You cant use variables in auth_request this is apparently a design choice for nginx
https://trac.nginx.org/nginx/ticket/761
I came up with this through trial and error. I had trouble putting $uri in a variable, not sure why.
location ~ /my/download/uri/(.*) {
set $key $1
auth_request /auth;
alias /my/file/directory;
}
location /auth {
internal;
proxy_pass http://auth_service:9999/$key;
proxy_pass_request_body off;
proxy_set_header Content-Length "";
}

nginx file upload with client_body_in_file_only

Good evening,
I need to upload static content to nginx server 1.9 (so upload module didn't work with this version). I've read the article "Nginx direct file upload without passing them through backend" and followed instructions step by step. Everything works for me, except file names at the nginx data directory. File names look like '0000000001', '0061565403' and so on. What should I do to save files with their correct names?
Here is my nginx location config:
location /upload {
limit_except POST { deny all; }
client_body_temp_path /data/;
client_body_in_file_only on;
client_body_buffer_size 128K;
client_max_body_size 50M;
proxy_pass_request_headers on;
proxy_set_header content-type "text/html";
proxy_set_body $request_body_file;
proxy_pass http://localhost:8080/
proxy_redirect off;}
You can use HTTP header in the client to pass the correct name (whatever that is), e.g.:
Correct-Filename: my-correct-filename
And since you're using proxy_pass_request_headers on, the header is visible in the back end where you can use it when saving the file. However when using headers the filename is limited to using ASCII characters, see this answer.
The only way I have been able to do this is to send the original filename as a parameter (I use JS to copy the filename to a hidden field), and then, on the server, if I am storing the temp file to our file system, I use that parameter to rename the file in the process of saving it to its "proper" location.
Not beautiful, but it works.

docker-registry nginx rest api

I am trying to build a docker-registry server from source (not as a container) on Ubuntu 14.04.1. I was able to get most of the way there using the instructions found on digitalocean.
I am able to curl http://localhost:5000 and https://user:password#localhost:8000 with no problems
When I try to open a web browser to see hopefully more than just that, that is when the issues seem to happen.
Here is my docker-registry file in /etc/nginx/sites-available/:
# For versions of Nginx > 1.3.9 that include chunked transfer encoding support
# Replace with appropriate values where necessary
upstream docker-registry {
server 192.168.x.x:5000;
}
server {
listen 8000;
server_name docker-registry;
ssl on;
ssl_certificate /etc/nginx/ssl/docker-registry.crt;
ssl_certificate_key /etc/nginx/ssl/docker-registry.key;
proxy_set_header Host $http_host; # required for Docker client sake
X-Real-IP $remote_addr; # pass on real client IP
client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads
# required to avoid HTTP 411: see Issue #1486 (https://github.com/dotcloud/docker/issues/1486)
chunked_transfer_encoding on;
location / {
# let Nginx know about our auth file
auth_basic "Restricted";
auth_basic_user_file docker-registry.htpasswd;
proxy_pass http://docker-registry;
}
location /_ping {
auth_basic off;
proxy_pass http://docker-registry;
}
location /v1/_ping {
auth_basic off;
proxy_pass http://docker-registry;
}
}
I have my docker registry stored locally in /var/docker-registry and ensured that it was readable by the www-data user. Why can I not see my images on the web browser?
If I tag an image and push it to my repository it works, I can see it in the web browser:
https://192.168.x.x:8000/v1/repositories/ubuntu-test/tags/latest
I see the following:
"5ba9dab47459d81c0037ca3836a368a4f8ce5050505ce89720e1fb8839ea048a"
When I try to get to:
https://192.168.x.x:8000/v1
Or:
https://192.168.x.x:8000/v1/repositories
Or:
https://192.168.x.x:8000/v1/images
I get a "not found" error
How would I be able to see everything in my /var/docker-registry folder (which is where these are stored....and yes, they are owned by the www-data user) through the web interface?
This is by design. Not only is there no reason one would implement the entire url path, but there are severe security implications with implementing it.
I'm assuming you don't have much experience with web programming. There is no directory '/v1/repositories'... etc. Instead, there is a program (in this case either Python or Ruby) that is listening for the url path and has logic built-in to determine what to do.
i.e. if url = /v1/_ping: return 'ok'

Nginx - how to return a proxy generated file?

Description:
I want to implement an http server (using nginx) that serves static files.
If the requested file doesn't exist, nginx shall send a request to a service (REST API) that will create the file and return its path.
After that, I want nginx to return the static file that was created.
Question:
What is the best way to return the file after its creation?
So far I managed to do this by changing the REST API in order to return the created file path with the 302 status code and with a location header as a redirect, but I am not sure if this is a good thing to do. Is it?
Is there any nginx-side solution for this? Do I have to create a custom module?
Conf file:
http {
server {
listen 80;
location /files {
try_files $uri #rest;
}
location #rest {
rewrite ^(.*)$ /api/ break;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_pass http://localhost:8080;
}
}
}
Edit: Actually, on balance, this should be even simpler:
location #rest {
...
proxy_intercept_errors on;
error_page 404 = $uri;
}
Configure the named location to intercept an "error" coming back (I chose 404), and then using the error_page directive will cause the given URI to be loaded again. Since the file now exists, the request should succeed.
Side note: I had thought try_files $uri #rest $uri would have worked, but an internal redirection only happens for the last argument.
The simplest option here is probably for your REST service to use X-Sendfile/X-Accel to return the relevant URI that Nginx should serve once the file is created. Your REST service could return the target URI using the header X-Accel-Redirect.
In your case, your API could actually just return the same URI it received as the X-Accel-Redirect header, and then Nginx would re-use the same location block and find the file for the subrequest occurring.
If this fails, however, using an internal Nginx location as per the examples at http://wiki.nginx.org/XSendfile and http://wiki.nginx.org/X-accel:
location /files-protected {
internal;
root /path/to/files;
}
and returning the relevant URI to that would also work.

Resources