I have an application that accepts TCP traffic (not HTTP) and I'd like the ability to have the traffic load balanced to it. However, one requirement is that when a client makes a connection, we do not close that connection under any circumstances (ideally) since we are dealing with some clients with older technology.
I've set up the kubernetes nginx ingress controller, but it isn't behaving how I'm hoping/expecting. What I would like is: If the connection to one of the upstream servers closes, then the client connection remains open for some amount of time while nginx picks a new upstream server and starts sending data to it. I am not concerned about the stream's data being split across different upstream servers, I just need the connection to stay open from the client's perspective during something like a redeploy.
What is actually happening is that from my client's perspective, currently when the upstream server app closes the connection, my connection is closed and I have to reconnect.
The ingress controller has this configuration, which I thought would accomplish what I want, but it doesn't seem to be working as expected:
server {
preread_by_lua_block {
ngx.var.proxy_upstream_name="tcp-my-namespace-my-service-7550";
}
listen 7550;
proxy_timeout 600s;
proxy_next_upstream on;
proxy_next_upstream_timeout 600s;
proxy_next_upstream_tries 3;
proxy_pass upstream_balancer;
}
Any help at all is greatly appreciated and I'm happy to provide more info.
What you describe is how nginx works out of the box with http. However
Nginx has a detailed understanding of http
HTTP is a message based protocol i.e. uses requests and replies
Since nginx knows nothing about the protocol you are using, even if it uses a request/reply mechanism with no implied state, nginx does not know whether it has received a request not to to replay it elsewhere.
You need to implement a protol-aware mitm.
Unfortunately I haven't been able to get this functionality working with nginx. What I've ended up doing is writing my own basic TCP reverse-proxy that does what I need - if a connection to a backend instance is lost, it attempts to get a new one without interrupting the frontend connection. The traffic that we receive is fairly predictable in that I don't expect that moving the connection will interrupt any of the "logical" messages on the stream 99% of the time.
I'd still love to hear if anyone knows of an existing tool that has this functionality, but at the moment I'm convinced that there isn't one readily available.
I think you need to configure your Nginx Ingress to enable the keepalive options as listed in the documentation here. For instance in your nginx configuration as:
...
keepalive 32;
...
This will activate the keepalive functionality with a cache of upto 32 connections active at a time.
Related
Basically what I'm trying to do is have a secure websocket connection start life at a client, go through nginx where nginx would terminate the tls, and instead of just proxying the websocket connection to a server, have nginx handle the websocket upgrade and just send the socket stream data to a tcp server or a unix domain socket.
Is that possible with the existing nginx modules and configuration?
proxy_pass can connect to a server via a unix domain socket
proxy_pass http://unix:/tmp/backend.socket:/uri/;
But the implication is that it still speaks http over the unix domain socket and the server is responsible for handling the websocket upgrade. I'm trying to get nginx to do the upgrading so that only the raw socket stream data gets to my server.
Sorta like a mix between proxy_pass and fastcgi_pass.
Do I have to modify one of these modules to make that possible or is there some way to configure this to work?
So what I eventually came to realize is that proxies just proxy and don't parse protocols. There's nothing built into nginx (although mod_ws in apache might do it) that can actually process the websockets protocol, the nginx proxy function just forwards the stream to the back end server. I'm working on another approach for this as the hope of having the webserver do the heavy lifting is not going to work easily.
I need to use http health checks on a Elastic Beanstalk application, with proxy protocol turned on. That is currently not possible, and the health check fails with a an error --> *58 broken header while reading PROXY protocol
I figured I have two options
Perform the health check on another port, and setup nginx to listen to http requests on that port and proxy to my app.
If it is possible to catch the broken header errors, or detect regular http requests in the proxy_protocol server block, then redirect those requests to a port that listens to http.
I would prefer the latter(#2), if possible. So is there any way to do this?
Ideally, I would prefer not to have to do any of this. A feature request to fix this has been submitted to AWS, but it has no ETA.
The proxy protocol specification says:
The receiver MUST be configured to only receive the protocol described in this
specification and MUST not try to guess whether the protocol header is present
or not. This means that the protocol explicitly prevents port sharing between
public and private access. Otherwise it would open a major security breach by
allowing untrusted parties to spoof their connection addresses.
I think this means that option 2 is a sufficiently bad idea that it's not even supported by conforming implementations of the proxy protocol.
Option 1, on the other hand, seems pretty reasonable. You can set up a security group so that only legitimate health checks can come in on the port without proxy protocol enabled.
Another couple of options spring to mind too:
Simply point your health checks at the thing that's adding the header (i.e. ELB?), rather than directly at your Nginx instance. Not sure if this is possible with Elastic Beanstalk, it's not a service I use.
Use something else to add the proxy protocol header before forwarding the health-check traffic on to your Nginx, which would avoid having to duplicate your Nginx config. For instance a HAProxy running on the same machine as your Nginx could do this. Again, use security groups to ensure that only legitimate traffic gets through.
I'm set up a live broadcast website. I use nginx as reverse proxy, and deploy multiple flv-live-stream process behind nginx(binary program writen by C++). In my flv-live-stream program. Clients maintain long connection with nginx. I count video frame that alreay sent to predict whether the client play smoothly.
But I found there is a strange buffer in upstream module. Even if the client 100% loss packets, back-end process can still send to nginx for 2~3 seconds, almost 2.5~3MBytes.
If there is a method that response can pass to a client synchronously, as soon as it is received from the back-end. And when nginx is unable to send data to client(exp. client loss packets...), nginx donot accept data from the back-end immediately.
I'm already set
listen 80 sndbuf=64k rcvbuf=64k;
proxy_buffering off;
fastcgi_buffering off;
Anyone can help? thanks!
We use nginx with an application server as a backend.
We need to limit number of simultaneous connections per IP to backend. We used limit_conn nginx directive for this purpose. But it doesn't work well in all cases.
If user generates a lot of connections from one IP and quickly closes them, then nginx passes this request to a backend, but because client connection is already closed, this connection is not count in limit_conn.
Is it possible to limit number of simultaneous connections per IP to backend server with nginx?
You may want to set
proxy_ignore_client_abort off;
Determines should the connection with a proxied server be closed if a
client closes a connection without waiting for a response.
from the documentation
Another suggestion is to use limit_req to limit the request rate.
I'm afraid this facility is not yet available for nginx out of the box. According to the Nginx FAQ
Many users have requested that Nginx implement a feature in the load
balancer to limit the number of requests per backend (usually to one).
While support for this is planned, it's worth mentioning that demand
for this feature is rooted in misbehaviour on the part of the
application being proxied
I've seen some 3rd parties module for that nginx-limit-upstream but I've never tried.
I have a server block listening to 80 port requests on a specific server name along with some location directives. How can I make nginx treat any request that doesnt match as if it hadnt received it, that is let it time out? Currently those requests are treated with an 404 error
There is a way to ignore every request and tell nginx to respond nothing:
server {
listen 80 default_server;
return 444;
}
Documented here: http://nginx.org/en/docs/http/ngx_http_rewrite_module.html
The non-standard code 444 closes a connection without sending a response header.
How can I make nginx treat any request that doesnt match as if it hadnt received it, that is let it time out?
You can't unwind the clock. To find out what server name is being requested (from TLS-SNI and/or Host header), nginx must first accept the connection (which will cause the operating system's TCP/IP stack to send a SYN-ACK packet back to the requestor, in order to advance the TCP handshake). Once the connection has been accepted, the other side knows something is listening. So you've already foregone the opportunity for a connection timeout.
Having irrevocably lost that opportunity, the next opportunity you have is to hit the socket receive timeout. Since the client will be waiting for the server to respond with something, you could theoretically trigger its receive timeout by not responding at all. There is no mechanism in standard nginx currently to tell it to sit there and do nothing for a while, but if you are willing to involve another piece of software as well, then you could use the proxy_pass directive or similar to offload that responsibility to another server. Beware, however, that this will keep some amount of operating system resources in use by your server for the life of the connection. Nginx operates on sockets, not raw packets, and the operating system manages those sockets.
Having answered the question as asked, I think it's better to question the premise of the question. Why do you want the client to time out? If your goal is to mess with or "pay back" a malicious attacker, nginx is the wrong tool. This is the space of computer security and you're going to need to use tools that operate at a lower layer on the networking stack. Nginx is designed to be a webserver, not a honeypot.
If your goal is simply to hide the fact that there's a server listening at all, then your goal is impossible to achieve with the way HTTP works currently. Firewalls are only able to achieve ghosting like that by rejecting the connection before it's accepted. Thus, the TCP handshake never proceeds, no SYN-ACK packet gets sent back to the client, and as far as anyone can tell that entire port is a black hole. But it's not possible to do this after determining what server name has been requested. The IP address and port are available from the very first packet; the server name is not.
Someday in the not-so-distant future, HTTP may use UDP instead of TCP, and the request parameters (such as the server name) may be presented up front on the first packet. Unless and until that becomes the norm, however, the situation I describe above will remain.
I'm assuming you are trying to deflect malicious requests. Something like this might work (untested).
Set up a default server (catches any requests that don't match an existing server name), and then redirect the client back to itself:
server {
listen 80 default_server;
rewrite ^ http://127.0.0.1/;
}
You'd have to setup a similar catch-all for invalid locations inside your valid server blocks. Might be more of a headache than you want.
I don't know how useful this would really be in practice. Might be better to consider fail2ban or some other tool that can monitor your logs and ban clients at the firewall.