Is it possible to configure IIS in such a way that it can handle multiple HTTP requests that arrive on the same TCP socket in HTTP pipelining mode in parallel?
We have a problem where multiple requests are done by a web client in a single TCP socket, using HTTP pipelining. The client basically sends let's say 10 requests at once, and then the server sends 10 responses (in the same order as the requests). Our server takes quite some time for each request, mostly waiting for external IO. It would be much more efficient if IIS could start to work on all 10 requests in parallel, then serialize the responses in the correct order back to the client. Obviously, the server would need some way to cache responses if e.g. response 3 is available earlier than response 2.
Is that possible somehow? Maybe this is not possible in IIS, or I'm just searching for the wrong keywords... We are running IIS 7.5 and ASP.NET 4.5 on Windows Server 2008 R2.
We came across the same issue in IIS 7.5.
Our solution was to enable "Web Garden"... and it really really works well! It's just that you can't have a "session" based web site. So if you have clients "logging in", you will have to re-configure the process. (We used cookies to store an encrypted token - anyway that's besides the point).
Go to:
Internet Information Service > Applications Pools
Select the Pool being used (you should have a pool per site)
Click Advanced Settings...
Find "Maximum Worker Processes" and crank that sucker!
The amount of processes that you push it up to now depends entirely on how much RAM your system has. You can of course monitor and control this your self.
With a "Web Garden" enabled, you will notice (with Process Explorer or something similar), IIS will spawn a new instance of w3wp.exe for each request, up to the max number you specified. New requests simply get processed by the next available Worker Process available, enabling true IIS parallel request processing. If two requests come in within moments of each other, and request 2 is completed before request 1, request 2 is sends its response.
IIS uses the HTTP server api (that uses HTTP.sys); so I did a simple test -
wrote an HTTP server using this API,
wrote a Winsock client that opens a connection and sends 2 http requests
I observed that if I called HttpReceiveHttpRequest twice on the server (without sending the response for the first request), it doesn't receive the second request (basically, the second call blocks). This holds true for both PUT and GET requests.
It appears that HTTP.sys is in fact serializing requests to IIS on a single connection; I couldn't find any configuration on HTTP.sys that might modify this behavior.
As you can see while the requests from all users all over the web are just being added to the queue, and building up and up (Green) - only 1 single Request is Executing (Blue).
This doesn't really answer the question - but its an beautiful illustration of this disastrous situation.
Related
I am working testing the auto-scaling feature of OpenStack. In my test set-up, java servlet applications are deployed in tomcat web servers behind a HAproxy load balancer. I aim at stressing testing the application, to see how it scales and the response time using JMeter as the stress tester. However the I observe that HAProxy (or something else) terminates the connection immediately the onComplete signal is sent by one of the member instances. Consequently, the subsequent responses from the remaining servers are reported as failures (timeouts). I have configured the HAProxy server to use a round-robin algorithm with sticky sessions. See attached JMeter results tree , I am not sure of the next step to undertake. The web applications are asyncronous hence my expectation was that the client (HAProxy in this case) should wait until the last thread is submitted before sending the response.
Is there be some issues with my approach or some set up flaws ?
We have recently implemented a new ASP.NET site to our webservers to replace our old Classic ASP site(Both severs are Windows 2008 R2 Using IIS 7.5). They are hosted on a Load Balancer.
This one .NET webform application is used for approximately 30 clients (each with their own URL. client1.mysite.biz, client2.mysite.biz etc...)
Our original plan was deploy our new application into 3 "WebSites" each with their own app pools and BIND the clients to the relevant Website.
When binding we bound to both Http and Https for the URL (we have certificates for each of the sites)
INITIAL PROBLEM:
We noticed that after we bound more than half the sites and tested, we were suddenly being greeted with "Service Unavailable. Service is Temporarily Unavailable" (NO NUMBER just the words) every time. We unbound everything and tried again (meticulously testing each time we bound a site). Each time after binding a certain number of sites the same thing happened.
We ran out of down time and went to Plan B. We put the whole thing in the "Default Website" as a virtual directory (No bindings) (This is how the Classic ASP site was setup)
OUR PROBLEM NOW:
Occasionally we get the same dreaded white screen with "Service Unavailable. Service is Temporarily Unavailable" (NO NUMBER just the words).
It seems to happen randomly (not load or time dependent as far as we can tell). If using AJAX it simply is caught in the "Error" portion of the AJAX code but I believe it is the same problem. The error occurs INSTANTLY when it does happen. If the user attempts to repeat the action that caused the problem everything is fine (they are not logged out and they proceed on their way).
However this is happening MULTIPLE times a day and it's across ALL of our sites (not just this new one).
One more item of great importance. This appears to be happening to ALL of our sites (Virtual Directories and custom WebSites on BOTH of our web servers). That seems to rule out a "bad" server (both are in the cloud did I mention?) and it also "seems" to rule out App Pool settings but what do I know?
About our IIS servers: We have multiple application pools running multiple different instances of websites (different code). Some are testing sites. Some are using classic ASP and others and using ASP.NET.
What we've tried: We scoured the web looking for answers and have edited our machine.config file to increase all manner of things such as "Threads, Max-Connections etc...". We've edited our App Pool settings by increasing our Queue Length and turning on ALL the logs.
Anyone seen anything like this before? My theory is it has something to do with the bindings and the frequency of the error is increased for each binding I initiate but that is difficult to test when it happens on my production servers only.
We have finally solved this problem. As mentioned previously, we noticed that the IIS logs contained a sc-win32-status 64 error when we experienced the Service Unavailable problem in the browser when (and only when) our site was using the Load Balancer.
To help look into this further, we did a network capture of the traffic on the Load Balancer while testing. We reproduced the random Service Unavailable problem, saw the associated win32-status 64 error in the IIS logs, and identified the specific packet of traffic on the network capture for this event.
Using Wireshark, we followed the TCP stream and noticed that the TCP connection was reset by the Load Balancer immediately after this packet. We reproduced the problem three times and every time there was a TCP reset immediately afterwards.
Walking backwards through the TCP stream, we noticed in all three instances a packet for HTTP/1.1 200 (accplication/octet-stream) and prior to that a request to download a document (ie. .pdf or .xlsx or .docx) from one of our sites. The server that contains all our documents is not a web server and does not have the IIS role active. The document server does not have a way to define the content/media type for the document that is being downloaded. Hence the generic (application/octet-stream) packet in the network capture. The Load Balancer treated the request for a document as potentially malicious and decided to reset the TCP connection if another request is made. To fix the problem, we added a content type library function to our application using this post as a guide. Sorted!
In Summary:
A document was requested from our document server via our web
application
The document was sent back to the user with a generic content type =
application/octet-stream
The Load Balancer flagged this activity to be potentially malicious
Another request within this TCP connection was made
The Load Balancer reset the TCP connection
This results in a Service Unavailable
Lesson Learned:
Always define your content/media types if you are serving content from a non web server or a web server running an IIS version less than 7 (Heaven forbid).
A UC Certificate was originally meant for Microsoft Exchange, but it can also be used to cover multiple domains. We use one and it covers about 60+ domains (actually 4 or 5 domains with lots of subdomains). We also apply the certificate to a load balancer and two web servers and we have multiple sites. So far as I can tell the certificates operate as expected. you can view it from any of the 60+ domains. One odd thing about our setup is that in the IIS UI, you can't bind the same certificate to more than one site so we had to use the appcmd command line interface to bind multiple sites to the same certificate.
After looking more closely at our IIS logs it appears that there is indeed something that coincides with this behavior. We get an error of 200 0 64 which is the sc-win32-status 64: "the specified network name is no longer available".
Now our 2 IIS servers are hosted in the cloud on Sungard, and we are using a load balancer that they setup for us. It was our theory that the load balancer was "losing" the proper session id of the user when this 64 error occurs and has no idea where it was supposed to be.
We ran some controlled tests. One group we took OFF the load balancer and sent them directly to one of the servers and another group used the load balancer but made sure to connect to the same server. Both teams conducted the tests of trying to reproduce the error (which is to say we clicked a popup on the site over and over).
The results were interesting. The group that was NOT on the load balancer NEVER received the "Service Unavailable" error! BUT the logs indicated they were getting 64 errors 45 times. The group that WAS on the load balancer was able to produce the "Service Unavailable" message twice and the logs confirmed that there were exactly 2 instances of the 64 error that coincided to the exact moment that the errors were observed.
So what does this mean?
1.) Load balancer has some settings "Sticky Sessions?" that aren't keeping the sessions in right (but we can't find the right settings. It's not even our load balancer it's SunGard's). Anyone have any advice on these settings for ASP.NET?
2.) 64 errors are a part of web life? We gave more cpu power to one of our Virtual IIS servers and received less 64 errors. This is all I can come up with. We've sunk too much time and money trying to solve this, but it appears that I have an option at least of taking people off the load balancer and just routing them to one or the other server and in addition I can at least beef up the server to handle more traffic and reduce the 64 errors.
My question is on ASP.net session management. In the current web application we have "sticky sessions" (user is always redirected to server it started talking to). Below is my problem statement.
From one of our client there are huge number of request hitting our servers. Somehow requests are sent from 1 or at most 2 IPs. We have 5 servers running to serve those request. Now the problem here is that 1-2 server might be heavily getting hits while other servers might be idle because sticky sessions will not allow request to be processed by serverB which was initially answered by serverA
What we need is exactly the opposite. Any server should be able to process incoming request maintaining the continued conversation.
I have put my problem in very plain words. Any pointer will be appreciated.
Why not just store the session state on a SQL server?
We have a biztalk server that makes frequent calls to a web service that we also host.
The web service is hosted on 4 servers with a DNS load balancer sitting between them. The theory is that each subsequent call to the service will round robin the servers and balance the load.
However this does not work presumably because the result of the DNS lookup is cached for a small amount of time on the client. The result is that we get a flood of requests to each server before it moves on to the next.
Is that presumption correct and what are the alternative options here?
a bit more googling has suggested that I can disable client side caching for DNS: http://support.microsoft.com/kb/318803
...however this states the default cache time is 1 day which is not consistent with my experiences
You need to load balance at a lower level with NLB Clustering on Windows or LVS on Linux (or other equivalent piece of software). If you are letting clients to the web service keep an HTTP connection open for longer than a single request/response you still might not get the granularity of load balancing you are looking for so you might have to reconfigure your application servers if that is the case.
The solution we finally decided to go with was Application Request Routing which is an IIS extension. In tests this has shown to do what we want and is far easier for us (as developers) to get up and running as compared to a hardware load balancer.
http://www.iis.net/download/ApplicationRequestRouting
I ask because I had heard that Microsoft may have implemented an optimized kernel level driver for handling HTTP traffic that then ushers it off to various worker processes through IPC and non socket mechanism. Is this true and if so is there any information on how this works. I'm asking in general for IIS6 and IIS7.
Microsoft may have implemented an optimized kernel level driver for handling HTTP traffic that then ushers it off to various worker processes through IPC and non socket mechanism
They have, is is called HTTP.SYS. Applications which wish to process HTTP requests (including IIS) use the API to subscribe to requests.
And documented on MSDN.
Useful information:
Also added by WinXP SP2, but IIS 5.1 does not use it.
Allows multiple processes to receive HTTP requests on the same port (so just because IIS is running does not stop another server using port 80... but URL prefix must be different).
Not limited to port 80 (HTTP) or 434 (HTTPS) (thus it can be used on XP while IIS is running).
Additional (based on comment):
Is it possible to intercept all requests that are inbound, but then hand them off to their originally intended target URLs? I would just like to capture the fact that the request took place and some information, and then let the request be processed as usual. – Leeks and Leaks
Based on the documentation (particularly this page): no indication of interception capabilities. But the ETW events might provide the information you are looking for (this will likely be Vista/2008/IIS7 only)