For example, I have two servers in the same network, with identical code/software. If the primary server goes down, I want the second one to become primary.
I heard about the following approaches:
Run a proxy server (nginx/haproxy/etc.) in front of these two.
Run CARP - Common Address Redundancy Protocol.
Round-robin DNS.
What are the pros and cons of the above approaches? And what are the best practices to achieve this?
I'm not too familiar with CARP but I can try to help with the remaining two options:
Round-Robin DNS gives you load balancing but if a server fails it will still receive requests (which will fail too)
i.e : the DNS www.example.com points to both x.x.x.1 and x.x.x.2 if x.x.x.2 dies the DNS will still be resolved to x.x.x.2 and clients will still try to request from it, so this brings your fail rate to half your requests during the downtime (not good)
Even if you change the DNS to point to only x.x.x.1 during the downtime; DNS propagation will take long and you will still loose requests.
In my honest opinion placing a load balancer (proxy server) in front of your stack is the only way to go
I'm really fond of HAProxy but its by no means the only solution (find what works for you)
Proxy-Servers gives you a lot more control over your application stack in the form of High Availability (HA)
you can load balance between 2 to N backend servers and loose any number of them and still be running.
you can schedule downtime anytime of the day to do maintenance or deployments and not influence your clients.
Built in health checks poll the backend servers and take them out of the load as needed and place them back when they've recovered.
The cons to HA Load Balancing is usually the number of rules that have to be setup in order to keep sessions correct or routing of special cases. yes it can get complex but there is A LOT of support in the community and its easily learn-able.
another con to HA Load Balancing is that the proxy server itself become a single point of failure but this can be overcome easily with heartbeatd and a second proxy server.
Hope this answers some of your questions
A good way for making your apps fault tolerant would be using nginx as your load balancer. You can make a config like
upstream some_name {
server server_ip;
server server_ip2;
};
server {
listen 80;
location / {
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-NginX-Proxy true;
proxy_pass http://some_name
}
}
plus this nginx upstream object takes further flags like max_fails=10 fail_timeout=20s and is smart enough to know if one server goes down, it switches to the next server that's online and so much more than that.
Please check this official nginx website for more information about it.
Related
Good evening everyone,
I have developed a web app for our school that is loaded into a kiosk app on ChromeOS and Windows so that students can take digital tests and exams in a protected environment.
The web app also allows you to consult sources, these sources are links to, for example, a news site, wikipedia, you name it..
Unfortunately, many external links do not work because they are loaded into the web app via iFrame. And nowadays many websites do not allow this by passing this in the headers such as x-frame options.
I had hope at https://github.com/niutech/x-frame-bypass but unfortunately it no longer works.
I also come to the conclusion that a reverse proxy could offer a solution here, but I have no experience with this and research does not make it any easier for me. Or are there even better/other solutions?
As a test I was able to realize through the following that google.be can be loaded within an iFrame, however I encounter 2 problems that I hope I can find a solution for this way.
Issue 1: Images and CSS not loading
The content links to the proxy server, of course that content does not exist on the reverse proxy server.
Issue 2: Every teacher can create exams/tests with their own external sources, it is impossible to add all those external URLS to the reverse proxy every time
That's why I thought of getting the url for the proxy_pass from the url of the reverse proxy url.
Reverse proxy url: http://sitekiosk.xyz/bypass/google.be
google.be gets used in the proxy_pass
Reverse proxy url: http://sitekiosk.xyz/bypass/wikipedia.be
wikipedia.be gets used in the proxy_pass
And so on...
location /bypass {
proxy_set_header Host www.google.be;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_pass https://google.be/;
proxy_hide_header Content-Security-Policy;
proxy_hide_header X-Frame-Options;
add_header X-Frame-Options "ALLOWALL";
}
Is this technically possible? Can someone help me with this?
Thank you in advance for taking the time to go through this!
My apologies for my Google Translate English :-)
I have a node.js server running behind an nginx proxy. node.js is running an HTTP 1.1 (no SSL) server on port 3000. Both are running on the same server.
I recently set up nginx to use HTTP2 with SSL (h2). It seems that HTTP2 is indeed enabled and working.
However, I want to know whether the fact that the proxy connection (nginx <--> node.js) is using HTTP 1.1 affects performance. That is, am I missing the HTTP2 benefits in terms of speed because my internal connection is HTTP 1.1?
In general, the biggest immediate benefit of HTTP/2 is the speed increase offered by multiplexing for the browser connections which are often hampered by high latency (i.e. slow round trip speed). These also reduce the need (and expense) of multiple connections which is a work around to try to achieve similar performance benefits in HTTP/1.1.
For internal connections (e.g. between webserver acting as a reverse proxy and back end app servers) the latency is typically very, very, low so the speed benefits of HTTP/2 are negligible. Additionally each app server will typically already be a separate connection so again no gains here.
So you will get most of your performance benefit from just supporting HTTP/2 at the edge. This is a fairly common set up - similar to the way HTTPS is often terminated on the reverse proxy/load balancer rather than going all the way through.
However there are potential benefits to supporting HTTP/2 all the way through. For example it could allow server push all the way from the application. Also potential benefits from reduced packet size for that last hop due to the binary nature of HTTP/2 and header compression. Though, like latency, bandwidth is typically less of an issue for internal connections so importance of this is arguable. Finally some argue that a reverse proxy does less work connecting a HTTP/2 connect to a HTTP/2 connection than it would to a HTTP/1.1 connection as no need to convert one protocol to the other, though I'm sceptical if that's even noticeable since they are separate connections (unless it's acting simply as a TCP pass through proxy). So, to me, the main reason for end to end HTTP/2 is to allow end to end Server Push, but even that is probably better handled with HTTP Link Headers and 103-Early Hints due to the complications in managing push across multiple connections and I'm not aware of any HTTP proxy server that would support this (few enough support HTTP/2 at backend never mind chaining HTTP/2 connections like this) so you'd need a layer-4 load balancer forwarding TCP packers rather than chaining HTTP requests - which brings other complications.
For now, while servers are still adding support and server push usage is low (and still being experimented on to define best practice), I would recommend only to have HTTP/2 at the end point. Nginx also doesn't, at the time of writing, support HTTP/2 for ProxyPass connections (though Apache does), and has no plans to add this, and they make an interesting point about whether a single HTTP/2 connection might introduce slowness (emphasis mine):
Is HTTP/2 proxy support planned for the near future?
Short answer:
No, there are no plans.
Long answer:
There is almost no sense to implement it, as the main HTTP/2 benefit
is that it allows multiplexing many requests within a single
connection, thus [almost] removing the limit on number of
simalteneous requests - and there is no such limit when talking to
your own backends. Moreover, things may even become worse when using
HTTP/2 to backends, due to single TCP connection being used instead
of multiple ones.
On the other hand, implementing HTTP/2 protocol and request
multiplexing within a single connection in the upstream module will
require major changes to the upstream module.
Due to the above, there are no plans to implement HTTP/2 support in
the upstream module, at least in the foreseeable future. If you
still think that talking to backends via HTTP/2 is something needed -
feel free to provide patches.
Finally, it should also be noted that, while browsers require HTTPS for HTTP/2 (h2), most servers don't and so could support this final hop over HTTP (h2c). So there would be no need for end to end encryption if that is not present on the Node part (as it often isn't). Though, depending where the backend server sits in relation to the front end server, using HTTPS even for this connection is perhaps something that should be considered if traffic will be travelling across an unsecured network (e.g. CDN to origin server across the internet).
EDIT AUGUST 2021
HTTP/1.1 being text-based rather than binary does make it vulnerable to various request smuggling attacks. In Defcon 2021 PortSwigger demonstrated a number of real-life attacks, mostly related to issues when downgrading front end HTTP/2 requests to back end HTTP/1.1 requests. These could probably mostly be avoided by speaking HTTP/2 all the way through, but given current support of front end servers and CDNs to speak HTTP/2 to backend, and backends to support HTTP/2 it seems it’ll take a long time for this to be common, and front end HTTP/2 servers ensuring these attacks aren’t exploitable seems like the more realistic solution.
NGINX now supports HTTP2/Push for proxy_pass and it's awesome...
Here I am pushing favicon.ico, minified.css, minified.js, register.svg, purchase_litecoin.svg from my static subdomain too. It took me some time to realize I can push from a subdomain.
location / {
http2_push_preload on;
add_header Link "<//static.yourdomain.io/css/minified.css>; as=style; rel=preload";
add_header Link "<//static.yourdomain.io/js/minified.js>; as=script; rel=preload";
add_header Link "<//static.yourdomain.io/favicon.ico>; as=image; rel=preload";
add_header Link "<//static.yourdomain.io/images/register.svg>; as=image; rel=preload";
add_header Link "<//static.yourdomain.io/images/purchase_litecoin.svg>; as=image; rel=preload";
proxy_hide_header X-Frame-Options;
proxy_http_version 1.1;
proxy_redirect off;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://app_service;
}
In case someone is looking for a solution on this when it is not convenient to make your services HTTP2 compatible. Here is the basic NGINX configuration you can use to convert HTTP1 service into HTTP2 service.
server {
listen [::]:443 ssl http2;
listen 443 ssl http2;
server_name localhost;
ssl on;
ssl_certificate /Users/xxx/ssl/myssl.crt;
ssl_certificate_key /Users/xxx/ssl/myssl.key;
location / {
proxy_pass http://localhost:3001;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host;
}
}
NGINX does not support HTTP/2 as a client. As they're running on the same server and there is no latency or limited bandwidth I don't think it would make a huge different either way. I would make sure you are using keepalives between nginx and node.js.
https://www.nginx.com/blog/tuning-nginx/#keepalive
You are not losing performance in general, because nginx matches the request multiplexing the browser does over HTTP/2 by creating multiple simultaneous requests to your node backend. (One of the major performance improvements of HTTP/2 is allowing the browser to do multiple simultaneous requests over the same connection, whereas in HTTP 1.1 only one simultaneous request per connection is possible. And the browsers limit the number of connections, too. )
I've found some interesting reading on the X-Forwarded-* headers, including the Reverse Proxy Request Headers section in the Apache documentation, as well as the Wikipedia article on X-Forwarded-For.
I understand that:
X-Forwarded-For gives the address of the client which connected to the proxy
X-Forwarded-Port gives the port the client connected to on the proxy (e.g. 80 or 443)
X-Forwarded-Proto gives the protocol the client used to connect to the proxy (http or https)
X-Forwarded-Host gives the content of the Host header the client sent to the proxy.
These all make sense.
However, I still can't figure out a real life use case of X-Forwarded-Host. I understand the need to repeat the connection on a different port or using a different scheme, but why would a proxy server ever change the Host header when repeating the request to the target server?
If you use a front-end service like Apigee as the front-end to your APIs, you will need something like X-FORWARDED-HOST to understand what hostname was used to connect to the API, because Apigee gets configured with whatever your backend DNS is, nginx and your app stack only see the Host header as your backend DNS name, not the hostname that was called in the first place.
This is the scenario I worked on today:
Users access certain application server using "https://neaturl.company.com" URL which is pointing to Reverse Proxy. Proxy then terminates SSL and redirects users' requests to the actual application server which has URL of "http://192.168.1.1:5555". The problem is - when application server needed to redirect user to other page on the same server using absolute path, it was using latter URL and users don't have access to this. Using X-Forwarded-Host (+ X-Forwarded-Proto and X-Forwarded-Port) allowed our proxy to tell application server which URL user used originally and thus server started to generate correct absolute path in its responses.
In this case there was no option to stop application server to generate absolute URLs nor configure it for "public url" manually.
I can tell you a real life issue, I had an issue using an IBM portal.
In my case the problem was that the IBM portal has a rest service which retrieves an url for a resource, something like:
{"url":"http://internal.host.name/path"}
What happened?
Simple, when you enter from intranet everything works fine because internalHostName exists but... when the user enter from internet then the proxy is not able to resolve the host name and the portal crashes.
The fix for the IBM portal was to read the X-FORWARDED-HOST header and then change the response to something like:
{"url":"http://internet.host.name/path"}
See that I put internet and not internal in the second response.
For the need for 'x-forwarded-host', I can think of a virtual hosting scenario where there are several internal hosts (internal network) and a reverse proxy sitting in between those hosts and the internet. If the requested host is part of the internal network, the requested host resolves to the reverse proxy IP and the web browser sends the request to the reverse proxy. This reverse proxy finds the appropriate internal host and forwards the request sent by the client to this host. In doing so, the reverse proxy changes the host field to match the internal host and sets the x-forward-host to the actual host requested by the client. More details on reverse proxy can be found in this wikipedia page http://en.wikipedia.org/wiki/Reverse_proxy.
Check this post for details on x-forwarded-for header and a simple demo python script that shows how a web-server can detect the use of a proxy server: x-forwarded-for explained
One example could be a proxy that blocks certain hosts and redirects them to an external block page. In fact, I’m almost certain my school filter does this…
(And the reason they might not just pass on the original Host as Host is because some servers [Nginx?] reject any traffic to the wrong Host.)
X-Forwarded-Host just saved my life. CDNs (or reverse proxy if you'd like to go down to "trees") determine which origin to use by Host header a user comes to them with. Thus, a CDN can't use the same Host header to contact the origin - otherwise, the CDN would go to itself in a loop rather than going to the origin. Thus, the CDN uses either IP address or some dummy FQDN as the Host header fetching content from the origin. Now, the origin may wish to know what was the Host header (aka website name) the content is asked for. In my case, one origin served 2 websites.
Another scenario, you license your app to a host URL then you want to load balance across n > 1 servers.
The situation I have is that I have a dev box full of different applications like a minecraft server, a couchdb server, and a basic wordpress blog behind nginx, which handles forwarding.
Now they all have their own way of handling logins, but what i'd like to set up is somekind of authentication proxy.
In a sense, intercept all the HTTP requests coming to the server and check if they are authenticated, if not return a login page, if they are let the request through to wordpress or couchdb. I could have a list of users in the server to let my friends login with 1 log.
I've tried googling with many different key words but haven't found out how this could be done with for example NGINX? Im a bit of a newbie when it comes to networking please help!
Here's a rough example to do HTTP auth in nginx and then proxy the connection to the original source:
location /couchdb {
auth_basic "Restricted";
auth_basic_user_file htpasswd;
rewrite /couchdb/(.*) /$1 break; # Optional, depends on what you're proxying to
proxy_pass http://localhost:5984;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
The above is from the couchdb docs, but the general idea is applicable to any HTTP-based app you want to protect. You'd need to repeat this config block for every distinct app.
Note the rewrite in the above setup lets you work with apps that don't expect to live anywhere but the URL root. It is not required.
Also note that if you want to have a single page where users log in and then have that login be shared across all your apps, that is much more complicated. That is commonly referred to as Single Sign On and requires a specific configuration for each app that you'd intend to integrate.
I'm using nginx on OpenWRT to reverse-proxy a motion-jpeg feed from an IP camera, but I'm experiencing lag of up to 10-15 seconds, even at quite low frame sizes and rates. With the OpenWRT device removed from the path, the camera can be accessed with no lag at all.
Because of the length of the delay (and the fact that it grows with time), this looks like some kind of buffering/caching issue. I have already set proxy_buffering off, but is there something else I should be watching out for?
Thanks.
I installed mjpg-streamer on an Arduino Yun, and then in my routers settings setup port forwarding whitelisted to my webserver only.
Here is my Nginx config which lives in the sites-enabled directory.
server {
listen 80;
server_name cam.example.com;
error_log /var/log/nginx/error.cam.log;
access_log /var/log/nginx/access.cam.log;
location / {
set $pp_d http://99.99.99.99:9999/stream_simple.html;
if ( $args = 'action=stream' ) {
set $pp_d http://99.99.99.99:9999/$is_args$args;
}
if ( $args = 'action=snapshot' ) {
set $pp_d http://99.99.99.99:9999/$is_args$args;
}
proxy_pass $pp_d;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $host:$server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Request-Start $msec;
}
}
I never got this working to my satisfaction with nginx. Depending on your specific needs, two solutions which may be adequate:
if you can tolerate the stream being on a different port, pass it through using the port forwarding feature of OpenWRT's built-in firewall.
use the reverse-proxy capabilities of tinyproxy. The default package has the reverse-proxy capabilities disabled by a flag, so you need to be comfortable checking out and building it yourself. This method is definitely more fiddly, but does also work.
I'd still be interested to hear of anyone who gets this working with nginx.
I have Nginx on Openwrt BB (wndr3800) reverse-proxying to a dlink 932LB1 ip cam, and it's working nicely. No significant lag, even before I disabled proxy_buffering. If I have a lot of stuff going over the network, the video can get choppy, but no more than it does with a straight-to-camera link from the browser (or from any of my ip cam apps). So... it is possible.
Nginx was the way to go for me. I tried tinyproxy & lighttpd for the reverse proxying, but each has missing features on OpenWrt. Both tinyproxy and lighttpd require custom compilation for the full reverse proxy features, and (AFAIK) lighttpd will not accept FQDNs in the proxy directive.
Here's what I have going:
Basic or digest auth on public facing Nginx provides site-wide access control.
I proxy my CGI scripts (shell, haserl, etc) to Openwrt's uhttpd.
Tightly controlled reverse-proxy to the camera mjpeg & jpeg API, no
other camera functions are exposed to the public.
Camera basic-auth handled by Nginx (proxy_set_header), so no backend
authorization code exposed to public.
Relatively small footprint (no perl, apache, ruby, etc).
I would include my nginx.conf here, except there's nothing unusual about it... just the bare bones proxy stuff. You might try tcpdump or wireshark to see what's cluttering your LAN, if traffic is indeed your culprit.
But it sounds like something about your router is the cause of the delay. Maybe the hardware just can't handle the cpu/traffic load, or there could be something else on your Openwrt setup that is hogging the highway. Is your video smooth and just delayed? Or are you seeing seriously choppy video? The lengthening delay you mention does sound like a buffer/cache thing... but I don't know what would be doing that.