I am trying to work out how to modify $args before using them in a set variable command.
I am moving from a service that uses the addresses formatted like this
http://server/proxy/account?mp=/stream
and need them to be sent to the new service like this
http://server:2000/system/proxy.php?unique_id=account&mounturl=stream
so far i have been able to use
this location block to get the stream account name
`location ~ ^/proxy//?([^/]+)/?([^/]+)? {
set $proxy_url https://127.0.0.1:2000/system/proxy.php?unique_id=$1&mounturl=$2;
proxy_buffering off;
proxy_ignore_client_abort off;
proxy_intercept_errors off;
proxy_redirect off;
proxy_next_upstream error timeout invalid_header;
proxy_pass_request_headers on;
proxy_set_header Cache-Control no-cache;
proxy_set_header User-Agent "$http_user_agent [ip:$remote_addr]";
proxy_set_header X-Forwarded-For $remote_addr;
proxy_connect_timeout 5;
proxy_send_timeout 15;
proxy_read_timeout 15;
proxy_max_temp_file_size 0;
proxy_pass $proxy_url;
expires off;
client_max_body_size 1M;
tcp_nodelay on;
}
however i cannot work out what i need to use to change the mount point from /stream to just stream
im aware i can just use $arg_mp as a variable to use it directly but i need to present it without the leading slash, and frankly i dont have any hair left to pull out, can anybody point me in the right direction please?
Related
I have minIO storage with a video folder inside the bucket where all the videos are uploaded.
I have configured Nginx proxy_pass to this folder to access the videos. I use the http://nginx.org/en/docs/http/ngx_http_mp4_module.html module. Everything works fine, I mean videos are available through the link (http://localhost:8888/videos/1.mp4), and playback also works.
The problem is, I want to request some range of the video, but the start-end parameters don't work (for example http://localhost:8888/videos/1.mp4?start=100&end=220), every time it gives me a complete video. Is it possible to use them in my case or for any workaround?
my Nginx configuration
server {
listen 8888;
server_name localhost;
# To allow special characters in headers
ignore_invalid_headers off;
# Allow any size file to be uploaded.
# Set to a value such as 1000m; to restrict file size to a specific value
client_max_body_size 0;
# To disable buffering
proxy_buffering off;
#charset koi8-r;
#access_log logs/host.access.log main;
# Proxy requests to the bucket "photos" to MinIO server running on port 9000
location /videos {
mp4;
mp4_buffer_size 5m;
mp4_max_buffer_size 10m;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
#add_header Accept-Ranges;
add_header Accept-Ranges bytes;
proxy_connect_timeout 300;
# Default is HTTP/1, keepalive is only enabled in HTTP/1.1
proxy_http_version 1.1;
proxy_set_header Connection "";
chunked_transfer_encoding off;
proxy_pass http://127.0.0.1:9000/mybucket/videos;
}
}
Is it possible with nginx to rewrite "&" to "&" if it appears in a URL the client sends to nginx?
I'm currently kinda stuck, as some parts of my application (where I have plenty influence on) are calling s3 download links from my minio backend with "&" and others with "&" argument separation in the URL. I'm not sure why this issue exactly occurs, but my idea is to simply make nginx fix these URL calls internally, as they are practically the same. The problem is that minio (my S3 Server) interprets these wrong and denies access to a given resource as s3 signatures do not match anymore with & in the URL. They always have to be "&" instead of "&" !
To be a bit more specific, how the URL differs in some cases, please see the following example:
String I pass:
https://localhost/test/sprites.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=2PsuWGctH4UQmGvEQYjTDsZ2HqGM%2F20220601%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20220601T172937Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=6da7dc0137d25730d09bebbd54b0e1f0132d58cba318b3cfe11bcde5af608e05
Browser calls:
https://localhost/test/sprites.png?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=2PsuWGctH4UQmGvEQYjTDsZ2HqGM%2F20220601%2Fminio%2Fs3%2Faws4_request&X-Amz-Date=20220601T172937Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=6da7dc0137d25730d09bebbd54b0e1f0132d58cba318b3cfe11bcde5af608e05
The location at my NGINX config looks like this at the moment of writing:
location / {
auth_jwt_enabled off;
limit_req zone=s3 burst=100 nodelay;
proxy_buffer_size 256k;
proxy_buffers 4 512k;
proxy_busy_buffers_size 512k;
proxy_cache_convert_head off;
proxy_connect_timeout 180;
proxy_hide_header Set-Cookie;
proxy_hide_header x-amz-id-2;
proxy_hide_header x-amz-request-id;
proxy_http_version 1.1;
proxy_ignore_headers Set-Cookie;
proxy_pass http://127.0.0.1:7777;
proxy_read_timeout 86400;
proxy_send_timeout 86400;
proxy_set_header Authorization $http_authorization;
proxy_set_header Connection "";
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Connection keep-alive;
add_header Content-disposition "attachment; filename=$1";
default_type application/octet-stream;
}
My guess here is that the browser converts the & to & for some reason, as the strings I'm passing does not contain & at all ...
Unfortunately nginx does not have an ability of globally replacing some strings. A workaround is possible, based on the rewrite ... last, when being triggered inside a location, starts over the NGX_HTTP_FIND_CONFIG_PHASE to find a new location for the rewritten URI. However there is a limitation of maximum 10 iteration rounds of this process, or nginx will throw an internal HTTP 500 error. Here is the idea:
location / {
if ($args ~ "^(?<prefix>.+)&(?<suffix>.+)$") {
rewrite ^ $uri?$prefix&$suffix? last;
}
...
}
Nevertheless I doubt the problem is on the nginx side, so most likely this won't help.
I have been battling this issue for some days now. I found a temporary solution but just can't wrap my head around what exactly is happening.
So what happens is that one request is handled immediately. And if I send the same request right after it hangs on 'waiting' for 60 seconds. If I cancel the request and send a new one it is handled correctly again. If I send a request after this one it hangs again. This cycle repeat.
It sounds like a load-balancing issue but I didn't set it up. Does nginx have some sort of default load balancing for connection to the upstream server?
The error received is upstream timed out (110: Connection timed out).
I found out that changing these proxy parameters, it only hangs for 3 seconds and every subsequent request now handles fine (after the waited one). Because of a working keep-alive connection I suppose.
proxy_connect_timeout 3s;
It looks like setting up a connection to the upstream is timing out and then after the timeout it tries again and succeeds. Also in the "(cancelled)request - ok request - (cancelled)request" cycle described above there is no keep-alive being setup. Only if I wait for the request to complete. Which takes 60 seconds without the above settings and is unacceptable.
It happens for both domains..
NGINX conf:
worker_processes 1;
events
{
worker_connections 1024;
}
http
{
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 65;
types_hash_max_size 2048;
gzip on;
# Timeouts
client_body_timeout 12;
client_header_timeout 12;
send_timeout 10;
include /etc/nginx/mime.types;
default_type application/octet-stream;
include /etc/nginx/conf.d/*.conf;
server
{
server_name domain.com www.domain.com;
root /usr/share/nginx/html;
index index.html index.htm;
location /api/
{
proxy_redirect off;
proxy_pass http://localhost:3001/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
#TEMP fix
proxy_connect_timeout 3s;
}
}
DOMAIN2 conf:
server {
server_name domain2.com www.domain2.com;
location /api/
{
proxy_redirect off;
proxy_pass http://localhost:5000/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection keep-alive;
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
#TEMP fix
proxy_connect_timeout 3s;
}
}
I found the answer. However, I still don't fully understand why and how. I suspect setting up the keep-alive wasn't working as it should. I read to the documentation and found the answer there: https://nginx.org/en/docs/http/ngx_http_upstream_module.html#keepalive
For both the configuration files I added a 'upstream' block.
i.e.
DOMAIN2.CONF:
upstream backend
{
server 127.0.0.1:5000;
keepalive 16;
}
location /api/
{
proxy_redirect off;
proxy_pass http://backend/;
proxy_http_version 1.1;
proxy_set_header Connection "";
...
# REMOVED THE TEMP FIX
}
Make sure to:
Clear the Connection header
Use 127.0.0.1 instead of localhost in upstream block
Set http version to 1.1
I use next config to save files on nginx. It works fine but filenames becomes like 0000001234. Is there a way to change filenames to original?
Here is the same question, the reply suggests to use additional header with original filename but there's no clear answer how to use it and how to rename the file.
limit_except POST { deny all; }
client_body_temp_path /www/sitename/uploads;
client_body_in_file_only on;
client_body_buffer_size 128K;
client_max_body_size 100000M;
proxy_pass_request_body off;
proxy_pass_request_headers on;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-FILE $request_body_file;
proxy_set_body off;
proxy_redirect off;
proxy_pass http://farm1;
It seems it is not supported as a module
https://github.com/fdintino/nginx-upload-module/issues/69
https://serverfault.com/questions/475755/nginx-upload-module-and-file-names
I want to use NGINX to as a proxy to get to Deluge which is inside my home network (NGINX is publically available).
This configuration works:
location 8112;
location / {
proxy_pass http://deluge_address:8112;
}
However I'd like to use an address in form of http://nginx_address/deluge to be proxied to internal http://deluge_address:8112.
I tried the following:
location /deluge/ {
proxy_pass http://deluge_address:8112/;
}
(I tried different combinations of trailing / - none work).
But I get 404 Not found instead.
I have some knowledge about networks, but not too much.
Does anybody have any idea what I'm doing wrongly?
I did find a solution for this, but found a bug also in Nginx in the same time
https://trac.nginx.org/nginx/ticket/1370#ticket
Edit-1
Seems like bug i logged was an invalid one, which even helped me understand few more things. So I edited the config a bit.
You need to use below config
location ~* /deluge/(.*) {
sub_filter_once off;
sub_filter_types text/css;
sub_filter '"base": "/"' '"base": "/deluge/"';
sub_filter '<head>' '<head>\n<base href="/deluge/">';
sub_filter 'src="/' 'src="./';
sub_filter 'href="/' 'href="./';
sub_filter 'url("/' 'url("./';
sub_filter 'url(\'/' 'url(\'./';
set $deluge_host 192.168.33.100;
set $deluge_port 32770;
proxy_pass http://$deluge_host:$deluge_port/$1;
proxy_cookie_domain $deluge_host $host;
proxy_cookie_path / /deluge/;
proxy_redirect http://$deluge_host:$deluge_port/ /deluge/;
}
The key was to insert a base url into the pages using below
sub_filter '<head>' '<head>\n<base href="/deluge/">';
And then make replacement in src and href attributes in html. And also url(' in css entries.
Luckily deluge has a JavaScript config which has the base url. So we can override the same by adding
sub_filter '"base": "/"' '"base": "/deluge/"';
I faced the same problem, luckily I found a better and official solution:
Reverse Proxy with Deluge WebUI
proxy_set_header X-Deluge-Base "/deluge/";
add_header X-Frame-Options SAMEORIGIN;
My final settings:
location /deluge {
proxy_pass http://127.0.0.1:8112/;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "Upgrade";
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_connect_timeout 75;
proxy_send_timeout 3650;
proxy_read_timeout 3650;
proxy_buffers 64 512k;
client_body_buffer_size 512k;
client_max_body_size 0;
# https://dev.deluge-torrent.org/wiki/UserGuide/WebUI/ReverseProxy
proxy_set_header X-Deluge-Base "/deluge/";
add_header X-Frame-Options SAMEORIGIN;
}