using SignalR in procution. at startup everything works fine, but after 8-9 hours, service stops working, without any exception, or any log information in Event Logs.
Info:
Online Users (who uses this service in this 8-9 hours): 3000
Online Concurrent Users Max (at same time): 200
Hubs Count: 1 (try catch in every method for logging)
after browser timeout, it returns "404 not found".
do u have any ideas?
What version of SignalR are you using? You should be using v0.5.2 as previous versions had issues with zombie connections which would shut down your app by causing either an OutOfMemoryException or exceed the allowed number of requests for the app pool.
Essentially what would happen is that the # of requests would get backed up (use the performance monitor to view ASP.NET/{Requests Current, Requests Queued, Requests Rejected} - See Performance Tuning) and/or you would max out your IIS requests and the service would shut down. You can manually override this and increase your Requests Current for ASP.NET. I increased this to about 20,000 on our production box.
If you're not maxing out your requests, your app pool may be shutting down due to increased memory usage or # of exceptions shutting down the app pool.
In the IIS manager, under Management, double-click on Configuration Editor.
Next, at the top, click on system.applicationHost/applicationPools then click on the RHS of the line that says "(Collection)". This will open up your application pool collection editor.
Select your SignalR app pool and check the Properties at the bottom. Here you can set the periodicRestart/memory threshold to whatever you want.
In our application I'm finding that we're good for about 45 minutes, due to the high traffic nature of the application.
Hopefully that helps.
Related
We have a webserver (WinSrv2019) running a few ASP.NET 4.8 websites with a huge traffic.
Each site is running in its personal 64-bit Application Pool (Integrated Pipeline), .NET CLR Version v4.0.
One of the websites gets stuck apparently and stops responding.
Sometimes it continues a few minutes, sometimes it can remain stucked for a hour.
During this period our inter-application logs are empty.
Moreover, IIS log that logs incoming requests is empty as well and incoming requests during this period aren't being logged.
On the client side browser doesn't get any error, but just continue waiting for response.
Since it happens with the single site only, we've excluded the option that the problem refers server hardware. The server resources usage aren't get over 50%
On the second hand we don't have idea what can cause Application Pool to get stuck in this way.
Please advice.
How we can investigate this issue?
Is there any tool allowing to "debug" what happens inside IIS on the low level?
Thanks in advance
I have a ASP.NET application running on IIS 8. My application sometimes does not respond to request with clients logging a timeout and then IIS starts sending out 503 Unavailable response codes. Each outage seems to last 5 minutes which corresponds to a 5 minute limit interval for Kill w3wp.
I have the App Pool configured for a Kill w3wp limit action with limit percentage of 80%.
While it might seems the pool is getting killed due to high CPU usage, I could not find any records in the event viewer. In fact, it had gone to idle state because of no requests in about 25 minutes.
The crash happened with 3 simple identical requests issued when the pool was idle.It cannot be reproduced manually in any environment.
This is happening about once a week. Since its a critical application, its running on a separate server with no other applications installed.
I have changed the Limit from Kill to throttle but I am not sure its a good idea. Anyways I've got to find a way to reproduce this.
Can the pool fail to exit an idle state?
If there is no traffic, w3wp.exe may go idle aka stop running. But that is not the same as it being truly stopped and it would not cause a 503. It would just start back up on the next request.
A 503 is more likely because your IIS App Pool is crashing or literally being stopped.
I would check Windows Event Viewer to see if says anything around that time frame around w3wp crashing. You can also set IIS to do more detailed logging anytime your w3wp app pool is stopped, started, recycled.
Docs about enabling more app pool logging:
https://technet.microsoft.com/en-us/library/cc753412%28v=ws.10%29.aspx?f=255&MSPPError=-2147217396
You could also search various IIS logs for errors:
https://stackify.com/beyond-iis-logs-find-failed-iis-asp-net-requests/
If you set auto recycle option then remove it and make it schedule recycle instead.
It seems as the memory consumption by your application tends to saturation level, IIS being recycled
I have a w3wp.exe that is restarting on my IIS server (see specs below). Memory gradually climbs to ~3G then it randomly restarts itself about every 1-2min.
Memory Usage:
The odd thing is that once this memory drop (what looks like a restart - btw...the app pool does not get recycled/restarted) happens GET requests are queued but then serviced as soon as the service warms/starts up (causing a delay in responses to our clients - who were initially reporting delayed reponse times on occasion).
I have followed this link to get a stack dump once the .exe restarts (private bytes go to ~0) but nothing gets logged (no .dmp file) with diag debug once the service restarts.
I see tons of warnings in my webserver (IIS) log but that's it:
A process serving application pool 'MyApplication' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '1732'. The data field contains the error number.
ASK: I'm not sure if this is a memory limitation, if cacheing is not playing well with my threads/tasks, if cacheing is blowing up, if there is watchdog service restarting my application, etc,. Has anybody run across something similar with w3wp.exe restarting? It's hard to tell because diagdebug is not giving me a dump once it restarts.
SPECS:
MVC4 Web API servicing GET requests (code is debug build with debug=true)
Uses MemoryCache with Model and Business Objects with cache eviction set to 2hrs...uses
Task (TPS) for each new request.
Database: SQL Server 2008R2
Web Servers: Windows Server 2008R2 Enterprise SP1 (64bit, 64G RAM)
IIS 7.5
One application pool...no other LOB applications running on this server
Your first step is to reproduce the problem in your test environment. Setup some kind of load generation app (you can write it yourself pretty easily) and get the same problem happening. Then turn off debug in web.config and see if that fixes the issue. Then change it to be a release build and test again.
I've never used memorycache - try reducing the cache eviction time or just turn it off and see if that fixes the issue. Good luck :)
After deploying an ASP.Net application to a new server, the first user to hit the app gets a long pause, presumably because the app is performing its initial compilation. However, this pause seems to also also occur after the application has timed out and unloaded itself from memory.
The first compilation would be tolerable as it only happpens once, but the 2nd one seems to me like it should be unnecessary....is there a workaround to address this issue? It would be nice to be able to extend the application timeout, but I only see a way to extend the session timeout, which I would rather not do, as this would keep all user sessions in memory for a long period of time.
That's IIS shutting down the worker process - the default is a 20 min idle time. In IIS6, you can configure the app pool and turn off the Idle timeout (on Performance tab). In IIS7, it's in the Advanced Settings.
A common way (though looks hackish to me) to keep the worker process alive is to have some program issue web requests to some URL in the application on a regular basis.
You can reduce the initial startup time by precompiling the ASP.NET app with aspnet_compiler.exe. This won't reduce the initial request time to zero (it'll still need to create the worker process and do other housekeeping.) but it'll noticeably reduce it.
I did find some recommended settings for the App Pools in IIS. The intended outcome of these settings is to reduce the number of times your application needs to go through the startup cycle. These settings should apply to IIS 6 as well as IIS 7:
Recycle Worker Process: Disable. Use this only if you have multiple websites and are trying to isolate processes.
Shutdown worker processes after being idle for (time in minutes): Disable unless you are sharing resources on your server. With this setting disabled, your website should not unload, ensuring that startup time for your site is as minimal as possible.
I have a ASP.NET web application running on an IIS6 server. The application is making potentially long running calls to a xml service on a remote machine. Some of the service calls on the remote machine are taking a long time to execute (sometimes up to 4 minutes). The long term solution would be to make the calls asyncronous, but as a short term solution we want to increase the timeout for the calls and the overall httpRequest time out.
My fear with this is that the long running calls will fill up the request queue and prevent the "normal" page requests from completing. How can the server, IIS and application settings be tuned to temporarely resolve the issue?
Currently there are approximately 200 page requests/minute and this results in 270 service requests/minute.
The current executionTimeout is 360 (6 minutes)
The current service call time out is 2 minutes
There's this article on Microsoft's Knowledgebase that has some pretty much all the information you might need:
* Contention, poor performance, and deadlocks when you make Web service requests from ASP.NET applications
I will give you some research I have done regarding some of the specific items handled in the article above. This information below applies to IIS6, comments for IIS7 where applicable.
Increase the Processor Worker Thread pool from 25 to at least 100
The default values for the Threadpool size is 100 because the default value for autoConfig is true.
The values covered by autoConfig is
maxWorkerThreads
maxIoThreads
maxConnection
There is one value that is still 25 that must change is – ASPProcessorThreadMax, this can only be set in the IIS metabase (via adsutil tool) in IIS6. [IIS7’s equivalent is the processorThreadMax value]
So I'm opting not to change the machine.config settings as they are fine and there are other paramaters that would be affected by turning off autoconfig, but rather change ASPProcessorThreadMax from 25 to 100 via the IIS metabase (the only way to change this value).
e.g.
cscript %SYSTEMDRIVE%\Inetpub\AdminScripts\<nowiki>adsutil.vb</nowiki>s SET W3SVC/AspRequestQueueMax 100
Max connections per server
maxconnection
The autoconfig sets this value to 12*number of cpu’s, that’s how many connections can be made to each address you are connecting to at one time.
Debugging
Here are some things you can do:
Monitor if requests are waiting in the queue
Monitor the following counter:
Run perfmon
Add counter: ASP.NET Applications/Requests In Application Queue
This will show us if work items are queued because of a shortage of workers.
Check Identity Used by Application Pool
Open IIS Manager
Check which application pool is used by your site in IIS manager.
Choose the Application pool being used in the Application Pools list, then Right Click -> properties and see what account identity is being used.
It should be Network Service by default.