I'm trying to have a self hosted sourcegraph server being served on a subdirectory of my domain using a reverse proxy to add an SSL cert.
The target is to have http://example.org/source serve the sourcegraph server
My rewrites and reverse proxy look like this:
location /source {
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Scheme $scheme;
rewrite ^/source/?(.*) /$1 break;
proxy_pass http://localhost:8108;
}
The problem I am having is that upon calling http://example.org/source I get redirected to http://example.org/sign-in?returnTo=%2F
Is there a way to rewrite the response of sourcegraph to the correct subdirectory?
Additionally, where can I debug the rewrite directive? I would like to follow the changes it does to understand it better.
-- Edit:
I know my approach is probably wrong using rewrite and I'm trying the sub_filter module right now.
I captured the response of sourcegraph using tcpdump and analyzed using wireshark so I am at:
GET /sourcegraph/ HTTP/1.0
Host: 127.0.0.1:8108
Connection: close
Upgrade-Insecure-Requests: 1
DNT: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.99 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Referer: https://example.org/
Accept-Encoding: gzip, deflate, br
Accept-Language: de,en-US;q=0.9,en;q=0.8
Cookie: sidebar_collapsed=false;
HTTP/1.0 302 Found
Cache-Control: no-cache, max-age=0
Content-Type: text/html; charset=utf-8
Location: /sign-in?returnTo=%2Fsourcegraph%2F
Strict-Transport-Security: max-age=31536000
Vary: Cookie
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Trace: #tracer-not-enabled
X-Xss-Protection: 1; mode=block
Date: Sat, 07 Jul 2018 13:59:06 GMT
Content-Length: 58
Found.
Using rewrite here causes extra processing overhead and is totally unnecessary.
proxy_pass works like this:
proxy_pass to a naked url, i.e. nothing at all after domain/ip/port and the full client request uri gets added to the end and passed to the proxy.
Add anything, even just a slash to the proxy_pass and whatever you add replaces the part of the client request uri which matches the uri of that location block.
so if you want to lose the source part of your client request it needs to look like this:
location /source/ {
proxy_pass http://localhost:8108/;
.....
}
Now requests will be proxied like this:
example.com/source/ -> localhost:8108/
example.com/source/files/file.txt -> localhost:8108/files/file.txt
It's important to point out that Nginx isn't just dropping /source/ from the request, it's substituting my entire proxy_pass URI, It's not as clear when that's just a trailing slash, so to better illustrate if we change proxy_pass to this:
proxy_pass http://localhost:8108/graph/; then the requests are now processed like this:
example.com/source/ -> localhost:8108/graph/
example.com/source/files/file.txt -> localhost:8108/graph/files/file.txt
If you are wondering what happens if someone requests example.com/source this works providing you have not set the merge_slashes directive to off as Nginx will add the trailing / to proxied requests.
If you have Nginx in front of another webserver that's running on port 8108 and serve its content by proxy_pass of everything from a subdir, e.g. /subdir, then you might have the issue that the service at port 8108 serves an HTML page that includes resources, calls its own APIs, etc. based on absolute URL's. These calls will omit the /subdir prefix, thus they won't be routed to the service at port 8108 by nginx.
One solution is to make the webserver at port 8108 serve HTML that includes the base href attribute, e.g
<head>
<base href="https://example.com/subdir">
</head>
which tells a client that all links are relative to that path (see https://www.w3schools.com/tags/att_base_href.asp)
Sometimes this is not an option though - maybe the webserver is something you just spin up provided by an external docker image, or maybe you just don't see a reason why you should need to tamper with a service that runs perfectly as a standalone. A solution that only requires changes to the nginx in front is to use the Referer header to determine if the request was initiated by a resource located at /subdir. If that is the case, you can rewrite the request to be prefixed with /subdir and then redirect the client to that location:
location / {
if ($http_referer = "https://example.com/subdir/") {
rewrite ^/(.*) https://example.com/subdir/$1 redirect;
}
...
}
location /subdir/ {
proxy_pass http://localhost:8108/;
}
Or something like this, if you prefer a regex to let you omit the hostname:
if ($http_referer ~ "^https?://[^/]+/subdir/") {
rewrite ^/(.*) https://$http_host/subdir/$1 redirect;
}
Related
I am implementing a web interface using React and Flask. One component of this interface is a server sent event that is used to update data in the front end whenever it is updated in a database. This data is quite large and one event could contain over 16000 characters.
The React front end uses a reverse proxy to the flask back end in order to forward API requests to it. When accessing this back end directly, this works fine with the SSEs and the data is pushed as expected. However, when using Nginx to serve the reverse proxy, something weird happens. It seems like nginx buffers and chunks the event stream and does not send the results until it has filled around 16000 characters. If the data from the event is smaller than this, it means the front end will have to wait until more events are sent. If the data from the event is larger, it means the new-lines that tell the EventSource that a message has been received aren't part of the event. This results in the front end receiving event X when event X+1 is sent (that is, when the new-lines actually appear in the stream).
This is the response header when using nginx:
HTTP/1.1 200 OK
Server: nginx/1.14.0 (Ubuntu)
Date: Thu, 19 Nov 2020 13:10:49 GMT
Content-Type: text/event-stream; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
This is the response header when running flask run and accessing the port directly (I'm going to use gunicorn later but I tested with flask run to make sure nginx was the problem and not gunicorn):
HTTP/1.0 200 OK
Content-Type: text/event-stream; charset=utf-8
Cache-Control: no-transform
Connection: keep-alive
Connection: close
Server: Werkzeug/1.0.1 Python/3.7.6
Date: Thu, 19 Nov 2020 13:23:54 GMT
This is the nginx config in sites-available:
upstream backend {
server 127.0.0.1:6666;
}
server {
listen 7076;
root <path/to/react/build/>;
index index.html;
location / {
try_files $uri $uri/ =404;
}
location /api {
include proxy_params;
proxy_pass http://backend;
proxy_set_header Connection "";
proxy_http_version 1.1;
proxy_buffering off;
proxy_cache off;
chunked_transfer_encoding off;
}
}
This config is based on the answer mentioned here.
As you can see, both proxy_buffering and chunked_transfer_encoding is off, so I don't understand why this is happening. I have also tried changing the buffer sizes but without any luck.
Can anybody tell me why this is happening? How do I fix it such that using nginx results in the same behaviour as when I don't use it? Thank you.
The above mentioned configuration actually did work. However, the server I was using contained another nginx configuration that was overriding my configuration. When the configuration parameters specific for SSEs were added to that configuration as well, things started working as expected. So this question was correct all along.
I have an Nginx server which redirect requests to another external sites.
The location configuration below :
location ~* ^\/h\/([a-zA-Z0-9.-]+)\/(.*) {
proxy_connect_timeout 20s;
set $remote_host $1;
proxy_set_header Host $remote_host;
proxy_pass http://$remote_host/$2;
So , the server receive a request like :
http://localhost/h/test.com/testinguri.txt
and the proxy pass will be http://test.com/testinguri.txt
Everything works fine until we use two underscores and the server will respond with 502 :
curl -I http://localhost/h/test.com/testing_ur_i.txt
HTTP/1.1 502 Bad Gateway
Server: nginx/1.10.3
Content-Type: text/html
Content-Length: 173
Connection: keep-alive
The error log with debug mode :
2020/11/05 00:00:11 [error] 40859#40859: *66328156 upstream prematurely closed connection while reading response header from upstream, client: ****, server: _, request: "GET /h/test.com/testing_ur_i.txt HTTP/1.1", upstream:
The request failed after least than one second , so it is not a timeout problem. And the distant site works correctly. (i changed the real domain name by test.com for security reasons)
Thank you
Given error actually says that connection was closed by your backend (test.com in your example).
Please try to add it to your location section:
proxy_read_timeout 300s;
proxy_connect_timeout 75s;
https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
So , it wasn't related to the use of underscore.
The problem was the http version supported by the distant site!
Changing the http to 1.1 for the proxy-pass fixed the issue :
proxy_http_version 1.1;
Thank you.
I am trying to build a reverse proxy with nginx to make all Is in my project reachable from single address.
For a single service the configuration below works without problem
/etc/nginx/sites-enabled/reverse-proxy.conf
server {
listen 80;
listen [::]:80;
location / {
resolver 127.0.0.1;
allow "x.x.x.x";
deny all;
proxy_pass http://consul:8500;
}
}
So when I call server's ip x.x.x.x in my browser I see the Consul UI and the URL showing x.x.x.x/ui/dc1. Besides that, I see that the UI did requests for asset files successfully.
My question; is it possible two host different services on the same server and just reference to them with different location? For example, if I want to include Vault UI then I would think of doing something like this:
server {
listen 80;
listen [::]:80;
location /consul {
resolver 127.0.0.1;
allow "x.x.x.x";
deny all;
proxy_pass http://consul:8500;
}
location /vault {
resolver 127.0.0.1;
allow "x.x.x.x";
deny all;
proxy_pass http://vault:8200;
}
}
However I am not sure if this could be done this way. The farest I got, is to open the Consul UI with all other sub requests not found (i.e. loading assets).
UPDATE
I think my problem is that I am wrongly using location and proxy_pass
observing the first configuration (which is working)
server {
listen 80;
listen [::]:80;
location / {
resolver 127.0.0.1;
allow "x.x.x.x";
deny all;
proxy_pass http://consul:8500;
}
}
If I look at the curl command curl localhost -L -vvvv
* Trying 127.0.0.1:80...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.18.0 (Ubuntu)
< Date: Fri, 10 Jul 2020 16:24:38 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 39
< Connection: keep-alive
< Location: /ui/
<
* Ignoring the response-body
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost/ui/'
* Found bundle for host localhost: 0x557b754549e0 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /ui/ HTTP/1.1
> Host: localhost
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.18.0 (Ubuntu)
< Date: Fri, 10 Jul 2020 16:24:38 GMT
< Content-Type: text/html; charset=utf-8
< Content-Length: 7806
< Connection: keep-alive
< Accept-Ranges: bytes
< Last-Modified: Fri, 10 Jul 2020 07:37:44 GMT
<
<!DOCTYPE html>
<html lang="en" class="ember-loading">
...
and I can see the html already. However, if I changed the conf file to this:
server {
listen 80;
listen [::]:80;
location /consul/ {
resolver 127.0.0.1;
allow "x.x.x.x";
deny all;
proxy_pass http://consul:8500;
}
}
and then try to call it like curl localhost/consul -L -vvvv, I get the following:
* Trying 127.0.0.1:80...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /consul HTTP/1.1
> Host: localhost
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx/1.18.0 (Ubuntu)
< Date: Fri, 10 Jul 2020 16:32:35 GMT
< Content-Type: text/html
< Content-Length: 178
< Location: http://localhost/consul/
< Connection: keep-alive
<
* Ignoring the response-body
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost/consul/'
* Found bundle for host localhost: 0x55ba7959f9e0 [serially]
* Can not multiplex, even if we wanted to!
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET /consul/ HTTP/1.1
> Host: localhost
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: nginx/1.18.0 (Ubuntu)
< Date: Fri, 10 Jul 2020 16:32:35 GMT
< Content-Length: 0
< Connection: keep-alive
I would appreciate any ideas on this issue
You are right, you are using location and proxy_pass a wrong way. When you use the
location /vault {
proxy_pass http://vault:8200;
}
construction, you are passing your URI to the upstream as-is, while most likely you want to strip the /vault prefix from it. To do it, you should use this one:
location /vault/ {
proxy_pass http://vault:8200/;
}
You can read more about the difference of the first and the second one here. However this still can prevent the assets from loading correctly.
This question - how to proxy some webapp under some URI prefix - is being asked again and again on stackoverflow. The only right way to do it is to made your proxied app request its assets via relative URLs only (consider assets/script.js instead of /assets/script.js) or using the right prefix (/vault/assets/script.js). Some well-written apps are able to detect if they are used under such an URI prefix and use it when an asset link is being generated, some apps allows to specify it via some settings, but some are not suited for the such use at all. The reason why the webapp won't work without fulfilling these requirements is quite obvious - any URL not started with /vault won't match your location /vault/ { ... } block and would be served via main location block instead. So the best way to do it is to fix your webapp, however several workarounds can be used if you really cannot.
Some web frameworks already builds their webapps with relative URLs, but uses a <base href="/"> in the head section of index.html. For example, React or Angular use this approach. If you have such a line within your webapp root index.html, just change it to <base href="/vault/">.
Using conditional routing based on HTTP Referer header value. This approach works quite well for a single page applications for loading assets, but if a webapp contains several pages this approach won't work, it's logic for the right upstream detection would break after the first jump from one page to another. Here is an example:
map $http_referer $prefix {
~https?://[^/]+/vault/ vault;
# other webapps prefixes could be defined here
# ...
default base;
}
server {
# listen port, server name and other global definitions here
# ...
location / {
try_files "" #$prefix;
}
location /vault/ {
# proxy request to the vault upstream, remove "/vault" part from the URI
proxy_pass http://vault:8200/;
}
location #vault {
# proxy request to the vault upstream, do not change the URI
proxy_pass http://vault:8200;
}
location #base {
# default "root" location
proxy_pass http://consul:8500;
}
}
Update # 2022.02.19
Here is one more possible approach using conditional rewrite:
server {
# listen port, server name and other global definitions here
# ...
if ($http_referer ~ https?://[^/]+/vault/)
# rewrite request URI only if it isn't already started with '/vault' prefix
rewrite ^((?!/vault).*) /vault$1;
}
# locations here
# ...
}
Rewriting the links inside the response body using sub_filter directive from ngx_http_sub_module. This is the ugliest one, but still can be used as the last available option. This approach has an obvious perfomance impact. Rewrite patterns should be determined from your upstream response body. Usually that type of configuration looked like
location /vault/ {
proxy_pass http://vault:8200/;
sub_filter_types text/css application/javascript;
sub_filter_once off;
sub_filter 'href="/' 'href="/vault/';
sub_filter "href='/" "href='/vault/";
sub_filter 'src="/' 'src="/vault/';
sub_filter "src='/" "src='/vault/";
sub_filter 'url("/' 'url("/vault/';
sub_filter "url('/" "url('/vault/";
sub_filter "url(/" "url(/vault/";
}
Update # 2022.02.19
Related thread at the ServerFault: How to handle relative urls correctly with a nginx reverse proxy.
Possible caveats using sub_filter on the JavaScript code: Nginx as reverse proxy to two nodejs app on the same domain.
I've done some searching on this issue and have found other posts similar to this one, but none of the solutions have helped me.
I've got a location entry in my Nginx config which contains a proxy_pass rule, but it only seems to work when I add a trailing slash to the URL.
Here's the location stanza
location /webApp {
proxy_pass https://webapp.service.consul/webApp;
proxy_set_header Host webapp.service.consul;
proxy_intercept_errors on;
}
You can see, when I use curl and go out to /webApp/ (with trailing slash), it works fine - and the cookie is set as expected:
MacBook-Pro:~ $ curl -I https://example.com/webApp/
HTTP/1.1 302 Found
Server: nginx/1.9.11
Date: Thu, 28 Apr 2016 14:45:55 GMT
Content-Length: 0
Connection: keep-alive
Set-Cookie: JSESSIONID=440B8D729469BBD80FC92796754D9475; Path=/providerApp/; HttpOnly
Location: http://webapp.service.consul/webApp/login.do;jsessionid=440B8D729469BBD80FC92796754D9475
However, when I go out to /webApp (no trailing slash), I get a 302 Found, but I don't get redirected to the /webApp/login.do page as I did with the trailing slash:
MacBook-Pro:~$ curl -I https://example.com/webApp
HTTP/1.1 302 Found
Server: nginx/1.9.11
Date: Thu, 28 Apr 2016 14:48:44 GMT
Connection: keep-alive
Location: http://webapp.service.consul/webApp/
I have tried add a rewrite rule, something like:
rewrite ^(.*[^/])$ $1/ permanent;
But it doesn't seem to make a difference. This was working before, and I haven't really messed with this location stanza, so I'm wondering if it has something to do with one of my other locations.
Any tips?
You didn't actually specify IN WHICH WAY is it not working, but it sounds like an issue that should be fixable by the proxy_redirect directive.
proxy_redirect http://webapp.service.consul/webApp/ http://example.com/webApp/
(However, the best solution might be to just ensure that the hostnames are correctly specified in the rest of the configuration files.)
I'm setting up a private docker registry with NGINX in front for authentication. Both in a container which are linked. The nginx image I'm using is jwilder/nginx-proxy. I can ping the registry just fine:
>http zite.com:5000/v1/_ping
HTTP/1.1 200 OK
Cache-Control: no-cache
Connection: keep-alive
Content-Length: 2
Content-Type: application/json
Date: Thu, 02 Apr 2015 12:13:32 GMT
Expires: -1
Pragma: no-cache
Server: nginx/1.7.11
X-Docker-Registry-Standalone: True
But pushing an image gives me:
FATA[0001] HTTP code 401, Docker will not send auth headers over HTTP
I've tried marking the registry as insecure but to no avail:
--insecure-registry zite.com:5000
I have been able to get this setup running without NGINX in the middle.
My NGINX config file is (where 'dockerregistry' is the name of the linked container):
upstream dockerregistry {
server dockerregistry:5000;
}
server {
listen 80;
server_name zite.com;
proxy_set_header Host $http_host;
client_max_body_size 0;
location / {
proxy_pass http://dockerregistry;
auth_basic "Docker Registry";
auth_basic_user_file /etc/nginx/dockerregistry_users;
}
location /v1/_ping {
auth_basic off;
proxy_pass http://dockerregistry;
}
}
I think I've read almost every article about this setup but one thing I cannot figure out is whether HTTP only access to a private docker repo is a no-go at all. Is it at all possible to get it working? Or do I have to use SSL certificates? If so, who knows a good guide for this setup?
Yes, you need SSL if you want to use (basic) authentication against your registry (and there is no way around that).
This was a deliberate design decision: the reasoning was that basic authentication over plain http would give a false sense of security, while credentials would really be transmitted in the clear and be extremely easy to compromise.
Not allowing for false security was indeed on purpose (though a questionable move, judging by the number of people being confused by that).
About setting up SSL, I would just go with the example nginx files in the repo:
https://github.com/docker/docker-registry/tree/master/contrib/nginx