This question is related to a connection reset question I asked recently. I am also pursuing the info in this SO article and this SO article. What I want to know is the best way to track down the cause of an IIS Thread Pool crash. I am also pursuing this problem with our vendor. Here are the details.
I have an IIS application. It uses a very simple ASP page, modeled on a sample supplied by our vendor, when we purchased our first COM toolkit quite a few years ago. This application runs on IIS on Windows Server 2003. The application is a separate web service, not a virtual root, and it's at a different port number than 80. In other words, we're not using another NIC or virtual IP address. The site has little traffic, but was configured to have a DMZ. The whole configuration looks a bit funky to me.
Address verification requests are sent to the IIS application from browser-less applications that use http protocol. Street number, street name, city, and state are sent with a few other pieces of identification, and a response is sent back. One of the applications is written in C; the other is written in Clojure.
Sending "single-shot" requests to the newly built IIS application works just fine from both applications. Sending a lot of requests, and it's been hard to pin down whether that's over 25 or some other number, causes the IIS thread pool to crash. That's more or less what the error logs say.
This application has run without incident for years on a W2K/IIS server. The ASP page talks to an address verification engine by means of a COM toolkit that we purchased from the vendor. We had to upgrade to a new server (and 2012 COM toolkit) only because the newest COM toolkit would not install on W2K, and the new COM toolkit contained support for the new USPS address verification regulations. We'll validate fewer addresses without the new toolkit, and the only way you can barcode addresses and get that discount is to validate addresses. (When you send out 29,000 motor vehicle excise bills, the discount helps.)
What is the best way to track down this problem, so I can find the culprit? I'm looking for links to good information as much as answers. I am sorry to be vague; I know the rules of SO, and endeavor to provide as much detail as possible. I can re-edit this post and provide log entries, if someone wants to see those. The bottom line is my Clojure client (handles the batch requests) starts getting resets, and crashes, when the IIS thread pool crashes.
Epilogue:
We believe the rapid opening and closing of the COM object is the issue, and our writing the ASP page this way was sanctioned by our vendor. To get around this problem and the fact we also needed to install MS Access for additional address verification functions, we wound up writing an ActiveState Perl program to solve multiple needs.
First, we have purchased an ActiveState product that will the our Perl program run as a service. The service will start with parameters and a port number that allow the http-based requests to be sent, just like the were before. In this case the COM object will be opened once when the Perl program starts, and close when the Perl program exits.
The other functions provided by the Perl program are not germane to this post, but the writing of this Perl program obviates the need for IIS and MS Access, which were required with the old configuration.
So, if you want to take something from this post, it would be open a COM object on opening of a program and close on program close, at least for W2K Server 2003 and higher.
We had an issue at work with respect to application slowness. We did an application pool memory dump and then analysed the dump.
http://support.microsoft.com/kb/919790
Related
Background: We are trying to migrate a large, complex web application written in classic ASP from Windows Server 2003 to Windows Server 2012 R2.
Everything is working without errors, but the new server is extremely slow to serve the ASP pages. With a single user on the site, response times in the order of 2-3 seconds for ASP pages are common. Equally large AJAX calls and JavaScript pages are served and process in under 100ms.
When the site receives a moderate level of load (more than approx. 50 users) it becomes unusably slow. Normal load for the the production site is several thousand users.
There does not appear to be a correlation between the amount of data returned or the database connection. We are using SQL Server 2008 R2 for the database.
The Web application server is in a DMZ and uses a hosts file entry for the database server which is in our general intranet. Database queries process extremely fast (within milliseconds).
I've tried profiling the web server memory usage, disk I/O and network usage, and found no evidence of memory leaks. Query profiling shows no lag in processing database calls.
Update after running Failed Request Tracing
I set up tracing to be triggered for classic ASP requests taking longer than 1 second
Maximum time shown by detail logs for each trace from request start to request completion: 140ms
Total request times logged ranged from 1094ms to 1453ms - so the actual request is taking an order of magnitude longer than the events logged by the failed request trace.
What are common fixes for this performance problem?
There are reports of classic ASP sites being slow if the connection string uses the machine name\instance name instead of an IP address, especially if SQL is running on a non-standard port. Maybe try changing the connection string, e.g.:
Server=10.10.10.123,1433\myInstanceName;
Reference: forums.iis.net
I am unable to comment since I do not have enough reputation, so asking question as an answer. I will remove this once I get the answer
What is the driver you are using to connect in your connection string ?
I did see your comment on host file, can you please try direct IP in the connection string. Please do not remove the host file.
Can you try a small new web application in asp just with minimal database listing. is that also slow?
Again try the same new application without a database connection and time the difference.
Do you have on error resume next in the code, are you failing on any file / log permission but that is not getting reported.
Try disabling Microsoft Defender. There are serious slowdowns after an update. They modified vbscript.dll which is what executes ASP code.
If you cannot live without Windows Defender, you can replace vbscript.dll with an older version.
By reading documents on MSDN, I realized that it is recommended to create separate hosts by functionality (Sending hosts, Receiving hosts and Processing hosts). And if there is only one host in this bizTalk server, this host can perform all receiving, sending, and processing messages functionality.
My question is: Is it possible to have multiple hosts that each host can perform its own sending, receiving and processing function , and not affect each other?
This is for multiple developers working on the same project, because our current situation doesn't allow us to have a full set of SQL Server Database and SQL server for each developer or using VM.
Thanks a lot!
Multiple hosts is not a solution for letting multiple developers work on a single server. A single send/receive adapter can only be assigned to one host.
You will also run into other problems, as all the configuration settings are shared in a single database, a change from 1 developer will effect the others.
This same question was asked and answered at MSDN. What you are trying to do is not supported and will not work. There is no way around this.
You must deploy the same application code to each computer in a BizTalk Group.
Sharing a BizTalk computer for development work is not a workable or productive solution and will have a definite negative affect on productivity.
You are correct, the best way to handle DEV is a VM with the entire stack. This is the issue you must address in your environment.
I'm not sure if I'm asking the question properly. I'm referring to locally installed software, often called an "Agent" that keeps in regular communication with some host via HTTP. e.g. When you install LogMeIn, the Agent keeps in communication with the logmein.com server so that when you visit logmein.com with your web browser and connect to the agent, the server is able to initiate communication. The Agent, however, isn't a webserver, nor are any ports forwarded to the Agent. So, is the Agent constantly polling the server asking like a broken record, "Can I help you? Can I help you? Can I help you?" Or is the http connection from Agent to server somehow kept open? I know you can keep an http connection open, but A) how, and B) for how long? Does the Agent need to act like a less annoying broken record asking, "Can I help you? Yet? Yet? Yet?" with much more time in between each question? Or can the Agent ask once and wait indefinitely, asking again only once it learns that the connection has been dropped?
Bottom line is, I'd like to create a small little sample program for trying my hand at writing a client/server application that communicates via the Internet using HTTP. Either side needs to be able to initiate commands / requests. The Agent would likely communicate with the Server using some sort of API, perhaps RESTful. When I start the experiment, I'll be using Perl. It'd be fun to create a Hello World project that would have samples in many languages for many platforms how to write the agent and how to communicate with the server. The agent code would do client side things (e.g. determine public IP address) and send the data to the server. The server would act on the data (e.g. store IP address in a database). The server might also initiate a command to the Agent (e.g. Hey, Agent! What's your CPU type?) Proper authentication / authorization between Agent and Server is of course a necessity.
Are there any existing projects to model off of? Any existing documents? Perhaps I'm just missing terminology and if I just knew that everything I was asking can be summarized by the term foo, then the doors would be opened wide for what I could find in searches!
I looked into the code of Ubuntu's Landscape. It uses Python's Twister -- a web server for HTML5 Websockets. So I'd say what I was looking for in an answer is Websockets (bi-directional communication). That now has opened up a wealth of options, node.js, twister, mojolicious, and many many more as web servers. Turns out using Ajax to poll every few seconds is a very bad idea -- an overwhelming slam on web servers. Keep the connection open.
I'm currently reading a lot about node.js. There is a frequent comparison between servers using a traditional thread per request model (Apache), and servers that use an event loop (Nginx, node, Tornado).
I would like to learn in detail about how a request is processed in ASP.NET - from the point it is received in http.sys all the way up to it being processed in ASP.NET itself. I've found the MSDN documentation on http.sys and IIS a little lacking, but perhaps my google-fu is weak today. So far, the best resource I have found is a post on Thomas Marquardt's Blog.
Could anyone shed more light on the topic, or point me to any other resources?
(For the purposes of this question I'm only interested in IIS7 with a typical integrated pipeline)
From my research so far, its my understanding that when a request comes in it gets put into a kernel-mode request queue. According to this, this avoids many of the problems with context switching when there are massive amounts of requests (or processes or threads...), providing similar benefits to evented IO.
Quoted from the article:
"Each request queue corresponds to one
application pool. An application pool
corresponds to one request queue
within HTTP.sys and one or more worker
processes."
So according to that, every request queue may have more than one "Worker Process." (Google cache) More on worker processes
From my understanding:
IIS Opens creates a request queue
(see the http.sys api below)
A "Web Site" configured in IIS corresponds to one Worker Process
A Web Site/Worker Process shares the Thread Pool.
A thread is handed a request from the request queue.
Here is a lot of great information about IIS7's architecture
Here is some more information about http.sys.
HTTP Server I/O Completion Stuff
Typical Server Tasks
Open questions i still have:
How the heck does IIS change the Server header if it Uses HTTP.SYS? (See this question)
Note: I am not sure if/how a "Kernel-mode request queue" corresponds to an IO completion port, I would assume that each request would have its own but I don't know, so I truly hope someone will answer this more thoroughly. I just stumbled on this question and it seems that http.sys does in fact use IO Completion ports, which should provide nearly all of the same benifits that evented IO (node.js, nginx, lighttpd, C10K, etc...) have.
First of all thanks for taking the time to read my question. Here is what I am trying to accomplish followed by what I have so far on this.
What I want to do is create a Windows application (or server of sorts) that would listen for requests from an ASP.NET application. The windows application would be installed and would listen for messages from ASP.NET application and then do some processing. The flow is like this:
A user downloads the desktop application and registers their IP address on my web site. After downloading the desktop app, the ASP.NET application can then send requests to that particular desktop client for further processing. I think further processing is independent of the resolution in this case that's why i have skipped over details on what processing would be done. But if you think it is important, please let me know and I will add those details as well.
I have looked into creating a TCP server that would listen for requests. Because the user has already registered their IP address on my web site, my web site assigns them a unique identifier and stores the ID alongwith IP address in database. Now, the ASP.NET site can send requests to that desktop application.
I have looked into creating a TCP server for this purpose. While researching I also came across PNRP and it seems something like what I am trying to do.
Can you guys recommend some solutions or where I should be looking at for this scenario? Should I create a simple TCPLISTENER or may be go with PNRP approach? Or something else?
The basic requirement is for a web application to be able to communicate with a desktop application. The web application would be servicing numerous users and each user would have a desktop application installed. Which user for which desktop client question would be addressed by the web application that would maintain a database of unique user id's and their corresponding IP Address.
Thanks in advance for your help.
You could use .NET remoting or a web-service in the desktop app. Use WCF or WSE for the latter. You can use COM to add windows firwall rules.
Whatever you do, take firewalling/NAT into account. It might be easier for the client application to poll the server (initiate the connection) otherwise you open a can of worms by trying to have a remotely-accessible server in your user's computer without having to do some very manual configurations on the user's networking equipment.
Once you have that part sorted out, what I used in your situation was .NET Remoting. At the time WCF had not come out and when it did it was to crippled for my needs. TCP IP sockets were too raw (I had to write too much code) and so Remoting solved my problem ideally (a hand full lines of code to set up the connection, and everything was automatic from there on).
EDIT: I use an excellent third party library that makes Remoting even more flexible (flexible enough that I am still waiting for WCF to catch up with the featureset so that I stop using Remoting, and no luck yet!). Check out http://www.genuinechannels.com/ to see all the features they have. It includes making calls from server to client, and that sounds exactly like what you need to do. Check it out.