I have a webservice which communicates with a desk top application using .Net remoting. The clients or users are using this webservice to insert or update any data to data base (through desk top application after some processing). The problem i'm facing is, during peaks time, that means calling 2000 - 3000 times this webservice to insert/update data with in 15 to 20 mins.. i can see that the number of threads increases upto around 2000. (In the TaskManager of w3wp.exe). What could be the possible reason for creating these much threads? As i can see other webservices are showing only less than 50 threads.
NB: The web service which is causing the issue is in a diff application pool.
Thanks
It means many calls are waiting for response.
Since ASP.NET uses thread pool, you should use them for fast operations or instead use ASYNC handlers so the thread backs to the pool and ASP.NET assigns a new thread as soon as the result gets ready.
It can be result of deadlock in BRL or DLL or anything else. Try putting a break-point in web service code and find where it gets blocked.
Related
IIS (or maybe ASP.NET) takes longer time to respond requests when they are sent simultaneously with other requests. For example if a web page sends request A simultaneously along with 20 other requests, it takes 500 ms but when this request is sent lonely, it takes 400 ms.
Is there a name for this feature? It is in IIS or ASP.NET? Can I disable or change it? Is there any benefits using it?
Notes:
I am seeing this issue on a ASP.NET Web API application.
I have checked IIS settings (IIS 8.5 on Windows Server 2012 R2) and found nothing that limit its throughput. All constraints like band-with and CPU throttlings are at high number. Also the server have good hardware.
Update 1:
All requests are going to read something from database. I have checked them in Chrome developers' console. Also created a simple C# application that makes multiple parallel requests to the server. When they are really parallel, they take a large time, but when makes wait between each call, the response time decreases dramatically.
Update 2:
I have a simple method in my application that just sends an Ok:
[AllowAnonymous]
public IHttpActionResult CheckOnline()
{
return Ok();
}
Same behavior exists here. In my custom C# tester, if I call this route multiple times simultaneously it tokes more than 1000 ms to complete but when wait 5 seconds between each call, response time drops below 20 ms.
This method is not IO or CPU bound. Seems that IIS detects that these requests are from a single specific user/client so do not make too much attention to it.
If you use ASP.NET Session in your application, requests are queued and processed one by one. So, the last request can stay holt in the queue while the previous requests are being processed.
Another possible reason is that all threads in the ASP.NET Thread Pool are busy. In this case, a new thread will be created to process a new request that takes additional time.
This is just a theory (or my thoughts). Any other cause is possible.
We have created ASP.NET MVC app which accept file upload(up to 80mb) and has result pooling implemented by AsincController. Hosted on Windows 2008 R2 IIS7.5 .NET 4. Server 2 Cores 2.6GHZ, 2GB Ram, fast HDD.
The web site has many users and Performance Monitor show ASP.NET Requests/Sec ~15 and Request Current ~270
After several minutes ASP.NET starts queuing request and ASP.NET Request Queued counter starts growing and application become extremely slow. I am hunting the problem almost a month, tried to profile code, no performance issues and no memory leaks. Increased maxWorkerThreads to 400 and maxIoThreads to 400. Set maxConcurrentRequestsPerCPU to 5000 and MaxConcurrentThreadsPerCPU to 0 but that either didn't helps.
One thing which seems helps is increasing app pool Maximum Worker Processes to two or three processes and making app pool web garden. After that Request Current jumps to ~350 and no request queuing. But web garden introduce several new issues which we will not mention here.
Please post any suggestions how we could increase application performance without making our app run in IIS pool web garden?
I will say to you to double check the pages that take long time to proceed a work, or the pages that download or upload files, and disable the session on this pages. If this is not possible, then you may need to write a totally custom way to handle the session.
Why, because session is locks everything until the page fully return.
You can read relative to session and to this issue:
call aspx page to return an image randomly slow
ASP.NET Server does not process pages asynchronously
Trying to make Web Method Asynchronous
Web app blocked while processing another web app on sharing same session
What perfmon counters are useful for identifying ASP.NET bottlenecks?
Replacing ASP.Net's session entirely
I have an ASP.NET Web Service (SOAP style) that is running in our production environment.
Our server guys have set things up such that things like starting and stopping of Windows services, etc., are sent via email to the appropriate parties.
Lately my boss has been getting emails about my ASP.NET web service:
The (My Web Service's Name) Application_Start method was called
Now I figure that what's happening here is that the service has gone so long since being called last that the server has unloaded it from memory and now it's being re-loaded again (the product that consumes this web service has declined in popularity, so this isn't too far fetched a theory).
However my boss tells me he's been getting this email "dozens" of times per day.
I suppose it's still possible that my theory above is accurate, especially given how it's spread out over 3-4 servers in our web tier, but is there any other explanation for why this might be happening so frequently?
At this point in time I don't know whether or not Application_End calls are being similarly emailed or not, or what the ratio is.
The application could be downloaded when some of the settings in <processModel> are exceeded. idleTimeout could be the one in your case, but also requestLimit and memoryLimit.
Also, and this is based on a true story, if you start any thread, run anything in a separate threadpool thread, or use the TPL, make sure that you catch any exception that might be thrown. Uncaught exceptions from those threads will kill the worker process. Check the application logs in the Windows event log. If this is the case, you should see the red icon application error signs around the same time that the emails go out.
As best I can tell when a worker process recycles:
a) a new one spins up before the old one shuts down
b) the old one shuts down once all the active requests its servicing completes
Is the above accurate?
If so, I have data that I store in SQL once Application_End() fires from the global.ascx file. I pull this data back in when Application_Start() fires.
The problem is based on my testing, the new worker process fires the Application_Start() before my old worker process gets a chance to complete its Application_End().
What are best practices for handling this situation?
cheers in advance
edit: I just noticed a feature on IIS 7 'Disabled Overlapped Recycle' - I'm guessing this is the best route
Your description of overlapped recycling is accurate, yes (1); and there is a setting for disabling it, but it's intended to prevent HTTP errors which you would be re-introducing. App pool recycles are a normal occurrence for managed apps (stems from, among other things, a CLR limitation that prevents the unloading of assemblies in the same memory space) that you must design for.
Your technique would be difficult to manage in a web-farm or web-garden scenario.
I think a better design would be to rely on out-of-process storage for the data (using distributed cache products like ScaleOut, App Fabric, and the like) so that all app pools have the same view of the cached data.
(1) -
http://mvolo.com/blogs/serverside/archive/2008/02/25/Starting_2C00_-stopping-and-recycling-IIS-7.0-Web-sites-and-application-pools.aspx
Under windows server 2008 64bit, IIS 7.0 and .NET 4.0 if an ASP.NET application (using ASP.NET thread pool, synchronous request processing) is long running (> 30 minutes). Web application has no page and main purpose is reading huge files ( > 1 GB) in chunks (~5 MB) and transfer them to the clients. Code:
while (reading)
{
Response.OutputStream.Write(buffer, 0, buffer.Length);
Response.Flush();
}
Single producer - single consumer pattern implemented so for each request there are two threads. I don't use task library here but please let me know if it has advantage over traditional thread creation in this scenario. HTTP Handler (.ashx) is used instead of a (.aspx) page. Under stress test CPU utilization is not a problem but with a single worker process, after 210 concurrent clients, new connections encounter time-out. This is solved by web gardening since I don't use session state. I'm not sure if there's any big issue I've missed but please let me know what other considerations should be taken in your opinion ?
for example maybe IIS closes long running TCP connections due to a "connection timeout" since normal ASP.NET pages are processed in less than 5 minutes, so I should increase the value.
I appreciate your Ideas.
Personally, I would be looking at a different mechanism for this type of processing. HTTP Requests/Web Applications are NOT designed for this type of thing, and stability is going to be VERY hard, you have a number of risks that could cause you major issues as you are working with this type of model.
I would move that processing off to a backend process, so that you are OUTSIDE of the asp.net runtime, that way you have more control over start/shutdown, etc.
First, Never. NEVER. NEVER! do any processing that takes more than a few seconds in a thread pool thread. There are a limited number of them, and they're used by the system for many things. This is asking for trouble.
Second, while the handler is a good idea, you're a little vague on what you mean by "generate on the fly" Do you mean you are encrypting a file on the fly and this encryption can take 30 minutes? Or do you mean you're pulling data from a database and assembling a file? Or that the download takes 30 minutes to download?
Edit:
As I said, don't use a thread pool for anything long running. Create your own thread, or if you're using .NET 4 use a Task and specify it as long running.
Long running processes should not be implemented this way. Pass this off to a service that you set up.
IF you do want to have a page hang for a client, consider interfacing from AJAX to something that does not block on IO threads - like node.js.
Push notifications to many clients is not something ASP.NET can handle due to thread usage, hence my node.js. If your load is low, you have other options.
Use Web-Gardening for more stability of your application.
Turn-off caching since you don't have aspx pages
It's hard to advise more without performance analysis. You the VS built-in and find the bottlenecks.
The Web 1.0 way of dealing with long running processes is to spawn them off on the server and return immediately. Have the spawned off service update a database with progress and pages on the site can query for progress.
The most common usage of this technique is getting a package delivery. You can't hold the HTTP connection open until my package shows up, so it just gives you a way to query for progress. The background process deals with orchestrating all of the steps it takes for getting the item, wrapping it up, getting it onto a UPS truck, etc. All along the way, each step is recorded in the database. Conceptually, it's the same.
Edit based on Question Edit: Just return a result page immediately, and generate the binary on the server in a spawned thread or process. Use Ajax to check to see if the file is ready and when it is, provide a link to it.