I have an ASP.NET Web API application running behind a load balancer. Some clients keep an HTTP busy connection alive for too much time, creating unnecessary affinity and causing high load on some server instances. In order to fix that, I wish to gracefully close a connection that is doing too much requests in a short period of time (thus forcing the client to reconnect and pick a different server instance) while at same time keeping low traffic connections alive indefinitely. Hence I cannot use a static configuration.
Is there some API that I can call to flag a request to "answer this then close the connection" ? Or can I simply add the Connection: close HTTP header that ASP.NET will see and close the connection for me?
It looks like the good solution for your situation will be the built-in IIS functionality called Dynamic IP restriction. "To provide this protection, the module temporarily blocks IP addresses of HTTP clients that make an unusually high number of concurrent requests or that make a large number of requests over small period of time."
It is supported by Azure Web Apps:
https://azure.microsoft.com/en-us/blog/confirming-dynamic-ip-address-restrictions-in-windows-azure-web-sites/
If that is the helpful answer, please mark it as a helpful or mark it as the answer. Thanks!
I am not 100% sure this would work in your situation, but in the past I have had to block people coming from specific IP addresses geographically and people coming from common proxies. I created an Authorized Attribute class following:
http://www.asp.net/web-api/overview/security/authentication-filters
In would dump the person out based on their IP address by returning a HttpStatusCode.BadRequest. On every request you would have to check a list of bad ips in the database and go from there. Maybe you can handle the rest client side, because they are going to get a ton of errors.
Write an action filter that returns a 302 Found response for the 'blocked' IP address. I would hope, the client would close the current connection and try again on the new location (which could just be the same URL as the original request).
Related
I had the following issue on a system that I supported ~7 years ago. We never got to the bottom of it, and focus shifted onto other issues. I was recently reminded of it, and wondered if anyone would know what was going on. But alas I'll be a little short on details. Sorry.
The Setup
I had a farm of web servers sitting behind a load balancer. The servers were hosting a system that would receive HTTP requests (XML &/or SOAP) from clients, then for each one kick-off a bunch of further HTTP requests to 3rd-party-suppliers, wait for the suppliers' responses, process and combine the results and respond to the client's request.
Think insurance comparison, but as Business-To-Business XML service.
The whole processing would take 5s of seconds, from receiving the initial client request to them sending back a response to that original HTTP request, and the server would be processing 10s or 100s of requests in parallel (i.e. at any given point, a given webserver would have many client Requests that had come in, and been logged, but not yet been responded to.)
We had detailed logging which record the reciept of the requests, including origin IP and which server was processing the request, and record when a response was sent.
All client requests were sent to a single IP address (well, URL), which was the address of the loadbalancer, which would then forward requests to the webservers, which weren't individually accessible to the internet (they didn't have public IP addresses).
Our load balancer would allow us to take individual web-servers out of rotation, for maintenance.
When we did that we could watch the DB logs, and see new requests stop coming in, and the existing request gradually get completed, until there were now outstanding requests and the server was idle.
The problem
We found that sometimes, when we took a server out of rotation ... it wouldn't entirely stop receiving requests. You could see the large bulk of request suddenly stop coming in, but it would still receive a trickle of fresh requests (I don't know ... maybe 0.1% of normal load, maybe less?). I think the longest we left it going was maybe ... 10 minutes?
Notably we realised that all of those requests were coming from a single client/IP address (I don't remember which).
I forget whether other (still-in-rotation) webservers were still receiving requests from this client, but I think they were?
If we rebooted the webserver, no further requests would come in after restarting.
Web stack was Windows, IIS, ASP.NET; pretty old school even at the time. All servers individually owned and configured.
What was happening?
We vaguely waved our hands and asserted that the client's integration with us was "holding an HTTP tunnel open and sending multiple requests through it", rather than sending each request separately, and thus was maintaining that tunnel even after the LB stopped sending new requests to that server. But that was BS-waffle, and since we never needed to actually understand what was going on, we ignored it and moved on with our lives :)
But I'd still like to know what we were seeing, if anyone can diagnose it from that description.
We vaguely waved our hands and asserted that the client's integration with us was "holding an HTTP tunnel open and sending multiple requests through it", rather than sending each request separately, and thus was maintaining that tunnel even after the LB stopped sending new requests to that server.
That sounds like a good explanation.
Normally, a LB will refuse new connections to a removed server, but will allow open connections to live on until they naturally close. This is known as "connection draining" or "graceful shutdown".
If one of your clients had HTTP keepalive on, and was holding a TCP connection open and sending HTTP requests through it for a long time, it would give the symptoms you describe.
Most LBs will have a configuration knob for how long to wait for connections to close before force-closing them during this "connection draining" time. You can set a timeout here to avoid this scenario if it is a problem for you.
The HTTP connection handling behaviour of clients will vary at the client's discretion, to a large extent. Perhaps most of your clients were of one type (say, web browsers) and weren't holding open a single connection for 10 mins, but perhaps one client was different (say, a programmatic HTTP API client)?
Further reading about "connection draining" on AWS Load Balancers here (the exact details will vary by LB vendor): https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-conn-drain.html
Further reading about HTTP keep alive here: https://en.wikipedia.org/wiki/HTTP_persistent_connection
I have a website that is frequently overloaded with multiple requests from thousands of clients. I cannot scale to infinity my servers and the application in current state is not possible to handle the traffic. For a better comfort I would like to let firstly the clients that started the transaction to complete it, and after that allow other clients to start the transaction. I am looking for a solution how to divide HTTP requests to two groups: the first requests that are able to finish the transaction and the others that should receive the 503 Server busy web page.
I can handle some amount of transactions concurrently. The rest transactions I would like to hold for a while with Server busy web page. I thought that I can use varnish for that. Bud I cannot think up the right condition in VCL for that.
I would like to find in varnish the number of current connections to the backend. If the current number of connections will be higher than some value (eg. 100) and the request didn't have a session cookie, the response will be 503 Server busy. If the number of connections will even greater than 100, but the session cookie exists, the requests will be passed to the backend.
AFAIK in varnish VCL I can get only the health of the backend (director) that should be true/false. But when backend is considered not healthy, the requests are not passed to it. When I use max_connections to the backend, all connections up to the limit will got 503 error.
Is there a way how to achive this behavior with varnish, ngingx, apache or any other tool?
Does your content have to be dynamic no matter what? I run a site that handles 3 to 4 million unique a day and use features like grace mode to handle invalidation.
Maybe another option is ESI, Edge Side Includes, that may help reduce load by caching everything that isn't dynamic.
My job is to write a distributed client/server application with some concurrent tasks. So i decided to use akka.net for the concurrency issues. To implement the ipc between server and client akka remote is used. For some reasons there may run more than one client of the same type on a workstation. So i configured these clients for dynamic assignment of a tcp port. This worked fine for sending messages to the server.
My problem is to push some information to the clients. To accomplish this task an actor on the client exist. Now the server creates a reference for this actor. Therefor it needs the port the client is listening on . My idea is to send the tcp port the client uses to the server in some sort of connection procedure using a actor on the server.
After searching for some hours I didn't find any hint where to find the dynamically assigned tcp port. So how would the client get the assigned tcp port?
Ok, I could use akka.cluster. But using akka.cluster is breaking a fly on the wheel, I think. And if it solves my issue reamins to be seen.
Two suggestions, assuming that it is your client that makes the first contact with the server.
I'd have the server keep track of which clients are connected. I'd probably have a heartbeat message that gets sent once every few seconds from each client system. This way you can store an IActorRef for each alive client and send messages back without the need for finding the port. IActorRefs are preferable wherever possible for location transparency.
If you actually need to explicitly find the port, you may be able to extract it from the Path property of the IActorRef of one of the actors on the client system.
Thanks to patricks suggestions my issue is solved.
The solution is to extract the needed information from the senders path available while executing the hello message. With this information the server is able to maintain a list of all connected clients and theire network address.
Thanks a lot # patrick.
Regards Gregor
I have developed serve-client model based on UDP. Client are connected to server on random basis. I mean number of clients alive at a time is not fixed.
Any new client can communicate any time. It means, there could be 1 live client or 100 clients or any number of clients.
Now in such model, I need to add HTTP requests. Browser could send request to server and then server will forward that to any of client based on some identification.
Is there any method or readymade server(like nginix or lighttpd), which I can use for this requirement.
My big worry is that, destination client are not fixed, they keep changing. Most of server (nginix or lighttd) have static entries for destination address.
I visualize your scenario as multiple sensors that connect to the servers when they have something to say, and then they send a request and wait for the answer.
I visualize you also want to somehow administer such modules so that you want to access to them via HTTP.
You could leave the new configuration items on the regular server so that upon any update connection the response would include (in a piggy-backed fashion) the changes to the node.
Or the server could mark somehow your interest in accessing a certain node, and then, when this connects, the server could notify the interested client. The sensor should pay attention to clients wanting to connect to them during a window time.
Certainly, more information would help us help you.
I have found this tool online: http://www.unleashnetworks.com/products/unsniff.html
How does this work? Are they assuming that all HTTP traffic for a session occurs in the same TCP session, and then just clumping all that data together? Is that a safe assumption?
I was under the impression that when I load a page, multiple TCP sessions could be running for that single page load (images, videos, flash, whatever).
This seems to get complicated when I think about having two browser tabs open that are loading pages at the same time..how could I differentiate one http "session" from another? Especially true if they are hitting the same page, right?
For that matter, how does the browser know which data incoming belongs to which tab? Does it keep track of TCP sessions belonging to an individual tab?
Edit:
When HTTP session is mentioned above, I am referring to all of the related HTTP transactions that it takes to, say, load a page.
By TCP session, I am literally referring to the handshake's SYN -> FIN packet lifetime.
Although it might not be visible, the HTTP Session tracker is being passed to the server from the client as a parameter or as e cookie (header)
You might need to read about HTTP session token
A session token is a unique identifier that is generated and sent from a server to a client to identify the current interaction session. The client usually stores and sends the token as an HTTP cookie and/or sends it as a parameter in GET or POST queries. The reason to use session tokens is that the client only has to handle the identifier—all session data is stored on the server (usually in a database, to which the client does not have direct access) linked to that identifier. Examples of the names that some programming languages use when naming their HTTP cookie include JSESSIONID (JSP), PHPSESSID (PHP), and ASPSESSIONID (ASP).
I am not familiar with the "Unsniff" app you link to, but I have used a few packet sniffers before (my favorite is Wireshark). Usually you can differentiate sessions based on what host they are connected to. So, for instance, if you have 2 tabs open and one is opened to www.google.com and the other is www.facebook.com, the packet sniffer should be able to tell you which session is pointed at which host (or at least give you an IP address, which you can then use to find the host. see: reverse lookup).
Most times, multiple HTTP sessions will be open to one host. This is the case when you're loading a site's various resources (CSS files, images, javascript, etc.). Each of these resources will show up as a separate HTTP session (unless, of course, the connection is persistent... but your sniffer should be able to separate them anyway). In this case, you (or the sniffer) will need to determine what was downloaded by looking at the actual data within the HTTP packet.