I have an internal website which calls multiple calls to the WCF service which is hosted in IIS 7 Windows 2008 R2 server.
During the heavy loading period when it is being used by 50-75 users, it returns FaultException in WCF calls. I highly doubt that it is because of the user loading because we are using the same website for almost a year and we haven't got the same error before. Some of the calls might take 2 or 3 sec to execute.
So, I added the following lines in web.config file in the Host Service. Do I need to add anything anywhere? Do I need to do anything in the client website too? Thanks.
<serviceBehaviors>
<behavior>
<serviceThrottling
maxConcurrentCalls="100"
maxConcurrentSessions="100"
maxConcurrentInstances="100"
/>
</behavior>
</serviceBehaviors>
Before changing configurations arbitrarly I would look for the reason of why it fails.
Since your calls take 2/3s to return we can safely assume that you achieve more than 10 instances created (10 is the default) at a given time. When that happens the calls queue up and if your load don't drop so every call can be replied to you are going to have problems.
You can do that diagnose by looking at performance counters that WCF provides, specially at Instance counters related to ServiceModelService 4.0.0.0.
If that is the problem (looks like it could be) then you can take action and increase the number of instances. However, the ultimate fix should be decrease response times because otherwise you will just be postponing the problems that comes with scaling.
Related
I am having some performance issues with my iis webserver. It is hanging randomly and I am trying to figure out how to speed up the server. I enabled Failed request tracing on the server and set it to generate a log when the request is over 3 seconds.
The resulting logs(xml) dont show much but there is a point in the compact performance log that indicates what part of the log the server is hanging on. Below is the part of the log where the large time loss is occurring.
65. i GENERAL_GET_URL_METADATA PhysicalPath="", AccessPerms="513" 17:46:32.577
66. i HANDLER_CHANGED OldHandlerName="", NewHandlerName="ExtensionlessUrlHandler-Integrated-4.0", NewHandlerModules="ManagedPipelineHandler", NewHandlerScriptProcessor="", NewHandlerType="System.Web.Handlers.TransferRequestHandler" 17:46:32.577
67. i VIRTUAL_MODULE_UNRESOLVED Name="FormsAuthentication", Type="System.Web.Security.FormsAuthenticationModule" 17:46:47.771
I am not sure what Handler changed is but it is taking a long time, any tips would be great on where to start looking.
It is hard to come up with a solution without having any piece of code in sight. Here are some general hints/tips you can follow in order to have great performances with an ASP.NET application.
The fastest way to do a request is to not do it in the first place. Try caching everything that can be cached. There are server-side caches and client-side caches. Each have their own uses, but you are not limited to only one type.
Make sure you do not cache and/or keep references of any request-related objects into memory. ASP.NET have a limited number of concurrent requests and keeping a request reference in memory will hang your server if it runs out of threads
Close the request as soon as you are done with it
Everything that is not needed by the client at the time of the request should be done in the background
Make sure you have no memory leak in your application. Garbage Collections are often the cause of hangs in ASP.NET application. When garbage collecting, all running threads are paused. This is especially true for Gen 2 garbage collections. You can enable background generation 2 garbage collections.
Isolate the problematic code. Use a profiler and see which type of request is CPU-intensive. Then dig deeper and see what inside that request makes it slow.
In any well-balanced application, objects should either be short-lived and live forever. In the case of an ASP.NET application, the objects created during the course of a request should ideally die within that request or during the next GC gen 0.
Consider object pooling for large objects and objects that are long to initialize
Make sure your app pool doesn't totally crash and restarts (look the IIS logs and/or the Windows Events)
Some useful debugging tools you can use:
LeanSentry. Great for diagnosing ASP.NET server hangs
windbg. High learning curve but by far the most powerful debugging tool you can use
PerfView. Useful for analyzing ETW events like I/O or CPU usage
There are many ways to improve server performance. But before that you should start with checking CPU usage during the "hang". An infinite loop in the application code may cause this behavior. Unless there is I/O, locking, or sleeps in the loop, you will be able to see it from the CPU usage as you will get exactly one full core's worth of CPU usage for each infinite loop.
Help link to improve server performance
More Info:
I can see entry related to VIRTUAL MODULE UNRESOLVED: which is related to bad use of Response.Redirect(url); Also make sure you have deployed your app on integrated mode on IIS.
here's a simple checklist you might want to reconsider:
Always pre-compiling your site, as opposed to copying it! you might gain a significant performance boost compiling your website before deployment: ASP.NET Precompilation Overview
Do not run the production application with debug="true" enabled, when debug flag is true in your web.config, Much more memory is used within the application at runtime, and since some additional debug paths are enabled, codes can execute much slower
Check your Web.config file to ensure trace is disabled in the section
IIS 7.5 comes with the Auto-Start Feature. WAS (Windows Process Activation Service) starts all the application pools that are configured to start automatically, ensure that your application pool is configured to AlwaysRunning in the IIS 7.5 applicationHost.config, check out here for more detail.
Every asp.net server can be well configured by aspnet.config file located in the root of the framework folder. Ensure that Publisher Evidence for Code Access Security (CAS) is set to false in your aspnet.config file, This might increase the initial page load when you restart the ASP.NET app pool. you can read more about it here.
Also you might want to try Application Initialization Module for IIS 7.5, this module also available on IIS 8.0 can decrease the response time for first requests by pre-loading worker processes
I'm trying to use the IIS 7.5 Application Initialization extension to configure a warmup process for my web application. This is an approach I am taking to minimize slow downs caused by application pool recycling, which is a problem explained well in other questions on Stack Overflow.
What I would like, is to gain the benefits of application initialization, without remapping requests anywhere else.
What I've done so far
I followed the IIS 8 instructions for the basic use case, and it works great! I created a splash page called app_starting.htm and by using this code, it gets displayed while the app initializes:
<applicationInitialization remapManagedRequestsTo="app_starting.htm" skipManagedModules="true" >
<add initializationPage="/" />
</applicationInitialization>
Why this isn't good
I want to use initialization to speed up requests to a REST-based web service written using ASP.NET MVC. This web service is a backend for several applications. When they make a request to a resource (i.e. /client/1/addresses), they can't handle receiving a splash page instead.
What I've tried
I removed the remapManagedRequestsTo attribute. However, now when I request a resource during initialization, I get a 500 error until initialization is completed. After which, responses go back to normal. The applications which rely on this this service also wouldn't respond well to a 500 error, since initialization should not be an error condition.
What I need
Without performing any remapping, I expect the request behavior to go back to normal. Even if initialization is in progress, other requests to the application should be queued and wait until after initialization has completed.
Is there something I am missing? Can I accomplish this?
Thanks for the help!
I think I answered my own question. I removed the skipManagedModules attribute and it worked. This code accomplishes application initialization, and during warmup, requests seem to wait for it to complete before being processed:
<applicationInitialization>
<add initializationPage="/" />
</applicationInitialization>
I couldn't find any documentation for why it behaves this way and don't really understand what skipManagedModules means. If anyone can further explain this, I can mark the explanation as an answer. Thanks!
I have a WCF service that has a complex operationcontract that has to executed atomically i.e. either the entire operation succeeds or fails. The WCF service is hosted on IIS server in an ASP .NET application. This operation has a sequence of SQL commands that execute in a transaction. During tests I found that with concurrent access by 4 - 5 users, atleast one user gets "Transaction Deadlock" error.
I then looked at the serviceThrottling settings which I had set to
<serviceThrottling maxConcurrentCalls ="5" maxConcurrentInstances ="50" maxConcurrentSessions ="5" />
and changed it to
<serviceThrottling maxConcurrentCalls ="1" maxConcurrentInstances ="1" maxConcurrentSessions ="1" />
I have turned off session since I don't need in the service contract. So I don't know whether maxConcurrentSessions will be having any effect at all
<ServiceContract([Namespace]:="http://www.8343kf393.com", SessionMode:=SessionMode.NotAllowed)>
This way I was queuing up the requests so that the request are processed serially instead of concurrently. While the transaction issue got away, the process time increased which was expected.
I was wondering
Whether serviceThrottling is the only way to resolve this issue ?
How can I set serviceThrottling such that while the service will accept many requests at the same time but will process one at a time?
Is setting the InstanceContextMode=InstancePerContext.PerCall relevant here since the application is ASP .Net application which in itself is multithreaded ?
Well, i think your going about this the wrong way trying to solve a database deadlock with WCF throttling.
you should try to understand why your database operations causes a deadlock, and try to avoid it (by using maybe locking hints.)
a singleton will do what you ask , but that isnt very scalable.
it is relevant but i think you get my drift , solve the deadlock in the database not in WCF.
if its SQL server that you are using , theres a great tool to analyze deadlocks (and a lot more) and its called the SQL Profiler. Also its a fairly well documented topic in the SQL Books Online
Your changes caused the WCF service to function as a singleton instance. That fixed your database concurrency issue but it only pushed the process blocking into the client.
I'd recommend using a different approach to remove the client blocking penalty. You should consider making this service, or at least extracting that operation into a new service that uses a netMsmqBinding (a good overview is here). This means the client will never be blocked and it guarantees delivery of the request to the service. The tradeoff is there can be no immediate response to the request, you'll need to add another operation to poll for completion status and to retrieve any expected results. It does require more work to spin up an MSMQ based service but the reliability is usually worth the effort.
I wrote a test page that does a bunch of busy work in a method called at page load. This process as I have it now takes around 12 seconds.
If I try to load another page while the first long running page is loading, this second page doing nothing except writing out a hello world, it doesn't load until the first long running page is finished.
Why is this the case? I would think IIS would be able to handle multiple concurrent connections, it seems crazy that one long running page would stop every other page in the application from loading. I must be missing something or not understand how IIS works.
I would think multiple independent requests would be spawned on different threads. Is this only the case if the requests are from different sessions entirely? Are all requests from a single session bound to a single thread?
bd
You need to turn off sessionState... if session is on, any sub-sequent request is waiting on the previous thread to finish in order to continue on that particular session.
So turn session off, it will not need to wait for any previous session.
in web.config put
<system.web>
<sessionState mode="Off" />
</system.web>
Now you should have those request process concurrently.
I just hit this issue too, and for me it turns out it is the session feature of ASP.NET. Basically if you have sessions enabled, each request will return in the order it arrived per user.
Try using 2 different browsers, for me it no longer blocked. Here is a similar question:
IIS 5.1 on XP Classic ASP appears to be in single threaded mode
It's a threading problem. While IIS can handle multiple connections simultaneously, ASP.NET, by default, is configured in single-threaded mode.
This choice by Microsoft was to prevent dummy developers to make common concurrency mistakes. ASP.NET for Mono doesn't show this behaviour and if you access shared resources without prior locking you might be... f... you know ;) by a yellow page of death.
I don't remember the exact procedure, but in the properties of your website you can choose the application pool and also the number of threads. I saw with my eyes that IIS6 sets it to 1 by default.
If you can't find it, tell me and tomorrow I'll take a look on my development server in lab.
Hope to have been of help.
We have an ASP.Net 2.0 web application running in a web farm which is using the ASP.Net State service to store sessions.
We have been having problmes with the service intermittently and have changed a few things such as the machineKey in the machine.config.
My actual question is around the monitoring of the state service service. We have all 4 available performance counters running on the server that hosts the service and as yet we have not seen a single session time out. We have also seen the number of active sessions slowly rise over a period of time, but never become less.
Does the state service recognise when sessions time out? Is there something we should be doing manually?
Edit: We have given up on the state service and gone with SQL server sessions.
To answer the questions below, it seems that sessions go up forever until the service falls over and it is very doubtful that any oen threads are linked to the state server. This is a fairly basic web app at the end of the day.
It seems from the reading I am doing that anumber of other people have experienced similar things, but there seems to be a general lack of common sense and knowledge in any responses flying about.
MS seem to have almost no documentation on this topic.
In ASP.Net session time outs can be configured in web.config and machine.config. The default time out assuming nothing has changed will be 20 mins. The machine.config file can be set to not allow overriding, which means that any changes specified in web.config files will not override these settings.
Have you ensured that the appropriate settings are in place in both machine and web config files?
The state service should drop each session after 20 mins of inactivity assuming the default settings.
At what point are your inactive sessions dropped? I assume they are not exponentially increasing, unless your are restarting the service in order to clear them they must be being dropped at some point.
Do you have something that might be hitting the session and keeping it alive without you knowing? Is there are thread being spawned somehwere that is doing work inteh background and holding on to your session? As far as my expeireince goes the timeout is set int he web config file and it just doesn't it's magic from there.
In my experience we've found out that native state server or even using SQL Server for sessions is a very scary scenario as both have issues.
I think you can explore other products for this to achive the absolute best.
A free option would be Velocity but it is still not released.
And another comprehensive but proven product will be (Very expensive actually) NCache
Take a look and see which looks best for you.
About SQL Server, you server will die very soon if you have enough number of hits coming in (I belive you have some hits already which yielded you to do Web Farm or you do it just for the sake of redundancy)
I am sure this will get modded down, but I have to say it.
If you are having issues with the state server, then there is likely an error somewhere in your web application. Charles' comment above seems like good places to start checking, but somewhere there is a life cycle issue.
Go back over the code and check your assumptions. Take a new computer, visit your website (create a session) and let it sit for an hour. If your session is still alive, then something is wrong. Create a new web application that just has a single page reporting the the age of the current session and try the same thing. You should find that after an hour (default is 20 minutes) the session is no longer valid. Now you have a system that is working as expect and one that is not, both using the same session server, so you can rule that out as the problem, now start going through code/configuration and see where you could be keeping it alive (or preventing the time-out).
Here, by the way, is a 'valid' session config. If you don't have your looking something like this, you have likely found your issue:
<sessionState
mode="StateServer"
stateConnectionString="tcpip=10.1.1.1:55455"
cookieless="false"
timeout="20" />
Also make sure you are not overriding your web.config with your machine.config to have a longer timeout.