Using performance monitoring tool "New Relic" I am seeing occasional (but too many) long delays in the "AcquireRequestState". I am talking about 10, 20 second delays, sometimes minutes.
I know we have not written our own event handlers for this event.
Where do I even begin looking for the cause of these delays? The little information I have found so far on msdn has not been helpful.
My team saw this "AcquireRequestState" delay reported by NewRelic earlier today when one of our ASP.NET applications was having performance problems on a particular page.
The root cause turned out to be a change to a stored procedure that we'd recently deployed to our SQL Server database, which was inadvertently causing that stored procedure to take a very long time to execute. The stored procedure was called as a part of displaying the page that was having the performance issue.
We were able to resolve the issue by identifying and fixing the performance problem with the stored procedure. The "AcquireRequestState" issue from NewRelic turned out to be irrelevant; it was a symptom of the problem, rather than the cause.
This was on an ASP.NET 4.5 application running on Windows Server 2008.
tl;dr: The "AcquireRequestState" delay reported by NewRelic may be a side effect of some other problem that's causing one or more of the pages and/or AJAX requests in your ASP.NET app to take a long time to load.
Try to apply Hotfix Rollup 2828841 on the server.
Issue 6
Symptoms
When you send many concurrent requests that have the same
SessionId to an ASP.NET 4.5 web application, some requests may freeze
at the RequestAcquireState stage unexpectedly.
Resolution
After you apply the hotfix, the hotfix makes sure that the
EndRequest event will always trigger.
This hotfix applies to
Windows 7 Service Pack 1 (SP1),
Windows Server 2008 R2 SP1,
Windows Server 2008 Service Pack 2 (SP2), and
Windows Vista SP2.
I suspect your stored proc change might be highlighting a slightly different problem in terms of session locking, we had roughly the same being reported for a different scenario, I'd seriously like to encourage you to test out the async session provider as mentioned here and making sure you use the concurrent requests per session app setting :
https://stackoverflow.com/a/55331786/7581050
Ultimately any long running process (in your case the stored proc change) is blocking any further requests for that session. Since this "block" is happening in a different part of the IIS pipeline, NewRelic simply records it as "AcquireRequestState"
PS: I know this answer comes quite late but I've finally found something that solved a similar problem for us and I suspect will help many people in the future.
Related
This may very well be a question that is too broad to answer but any ideas would be incredibly beneficial. I have a web site where load times are incredibly slow in one environment but not the other. In general, the time to first byte is around 15 seconds on most pages. It takes this long on every page within the entire application and not only on first load. I have been troubleshooting the issue for several days now and feel completely lost as to the actual cause for the latency.
Now for a long explanation about the issue.
The environment is a Frankenstein monster of different sources where too many people have had their hands in it, from what I can gather. I have carefully taken the time to compare each of the two environments and haven't identified a key difference. There are numerous things at play here, but I can summarize the main components.
It is a .NET web application built using Orchard CMS running within IIS and has a SQL Server backend. A dedicated server hosts the database and the another dedicated server hosts the web application itself, which is pretty standard. The main difference between the environments is the production site is running in Liquid Web and the new development site is running in AWS. Basically, the site will ultimately be migrated to AWS once the latency issues are resolved.
AWS has more than enough resources. In fact, production (Liquid Web) has been running into issues as of late due to the CPU usage being nearly maxed out. There are many more resources in AWS, and neither of the servers appear to be using more than 1% or 2% of their available resources. I verified this.
If the issue is within the database, I'm not really sure where else to look. I used SQL Server Profiler on the database server to analyze traffic and no transactions were taking more than a half second, aside from the Audit Logins/Outs (which from my research is normal behavior). The main database queries execute almost immediately after trying to navigate to a page within the site, not 15 seconds later when the page loads.
I had a thought that the network traffic in AWS application server and the database server could be bottlenecked somewhere. However, resolving the application locally does not improve performance. I thought it could have been an issue with the routing within the domain, such as the way in which DNS is set up, but that does not seem to be the case either... or perhaps it is, and I just haven't figured out the best way to troubleshoot that. Either way, resolving the application on localhost does not improve performance. The page still hangs for 15-20 seconds.
The vRAM usage for the site's application pool and the default app pool certainly does seem on the high end, if that makes a difference.
I have browsed the IIS logs and cannot find anything obvious. Granted, I don't have much experience in IIS and could be missing something. Windows Event logs show me nothing out of the ordinary either. There are some errors in both Liquid Web and AWS in regards to printer drivers not being installed, but those have nothing to do with the application itself.
I am unsure of how to check if it has something to do with the Orchard CMS. Granted, this is just a package/framework that was migrated over into the dev server, directly along with the application itself. I see nothing that would have changed within the environment.
The fact is that the two environments seem identical, yet one is running very slowly based on some factor that I just can't seem to identify.
Thank you!
We're getting more and more complaints from users that our ASP.NET 4.5.2 website is running slowly or just generally "freezing up." Things look fine from our test servers and from our workstations, but we're probably using better workstation hardware and browsers than our customers. We're running ASP.NET 4.5.2, C#, SQL Server.
What are some areas that we should concentrate on for debugging such a nebulous request? Should I be looking at system performance and resources on the application servers? System performance and resources on the SQL server? We're tracking application page load times, and they don't seem to be excessive or much changed from months ago, even though customer complaints have gone up.
What are some best practices for starting our investigation, and where's the low hanging fruit on improving performance overall?
If your page is getting slower "sometimes" during the day, I would suggest first to check the Performance Monitor at your IIS server. This could easily be an issue with the server hitting it's limits (Machine or IIS settings). One way verifying this is by creating a sandbox server and run your application from there for your testers.
After that if you are executing stored procedures, add a monitor function in them to gather some cases and then check if any of them causes the process to freeze or delay.
I must also mention here the possibility of locked tables, so maybe a code review maybe in line. (most time consuming from all the above..)
This should be able to give you a hint where your issue originate.
Good luck
If you suspect some SQL problems, you can try to run a Sql Server Profiler to check what is running at the moment and if there is something that could be "freezing up" your system. This way you can check what is going on when the system is slow.
Reference
I am relatively new to asp.net and IIS things, i have done many apps using other technologies, but never faced such a problem.
We have set up a lab on Azure, with all needed virtual machines, hosting the ASP.NET MVC 5.0 app on a machine (IIS 7), and another machine hosting SQL Server 2014 to host its database, we are using code first.
The issue is that when opening a page that lists records from a table from the database, it takes 15 seconds to do so, only for the first time when opening the website from chrome browser or any other browser, when continue navigating the website it takes no time at all. leaving the website for 10 minutes or so, and coming back, the issue happens again, and it takes 15 seconds again only once.
Digging deeper into the issue and after profiling of the performance, the most of the time taken is spend on the first time the app touches the database, for example, a very simple query:
model.Workflows.ToList();
Apparently connecting to the SQL server is taking the most of the time. 15 seconds for the first time is a lot of time to accept, I have never seen such a delay in any application.
the object graph is small its only 12 tables with very few records, even no records for some tables, for example, workflow table contains only 2 records. the relationships are complicated but not that much.
I have tried to connect without entity framework using SqlConnection class, and it takes 4 seconds for the first time also, it is much better but still there is something wrong.
For those who still facing the same issue, I have tried many posted solutions with no luck at all, including the link provided by raderick and many other resources.
The only thing that worked for me, is by updating the whole Windows Server 2008 r2 (with IIS 7.5) to Windows Server 2012 r2 (with IIS 8.5). It looks like it is something related to IIS performance (maybe my testing lab) and not Entity framework.
There is a very useful option in IIS 8.5 within windows server 2012 r2, look into this links at point 7 (Application Pool Suspension), that did a huge jump in performance in my case.
http://www.infragistics.com/community/blogs/brijmishra/archive/2015/08/21/12-tips-to-increase-the-performance-of-asp-net-application-drastically-part-2.aspx
I know it is not a first option solution but it may help in some cases.
Recently we have upgraded our main product to be an ASP.NET 4.0 Project (upgraded from 2.0). This project is the main source of traffic to the web service, the other forms have not changed in this release. The web service has been running without issue for a couple years now.
Following the release of the 4.0 environment to our main product our web service starting to die after about an hour and would do nothing but timeout until we restarted the worker process.
We have changed it so that the web service was also in the 4.0 Framework. However this did nothing. Other possible attempts we have tried have also failed such as making it a service instead of a web reference, and updating our certificate validation to also use the proper/current framework techniques.
The worst part is that there is no log in the event viewer being created so I have no leads as to what the problem is. We have applied a band-aid solution of recycling the app pool every 20 min (the recompile takes a second as the web service is very small) and it seems to be holding for now, but we would rather fix the problem than rely on this.
So does anyone have any additional idea/suggestions as to where our problem may be coming from? Or anyone experience anything similar?
Both projects exist in the same web farm and all machines are using IIS 6 32 bit.
Thanks!
edit- Some more info, The web service has a couple basic functions:
1 - accepts XML documents loads them into a dataset then updates internal DBs with the information sent - and simply returns true
2 - receives a request for processed data through XML and queries the DB for it builds XML response and sents it out
3 - receives a confirmation that the data requested in step 2 was recieved and deletes it from the DB
4 - hits a function that updates a DB so we can monitor some applications on our clients systems.
First of all, can you reproduce the problem on a dev box (using some load testing platform to simulate production load)?
If so, then it sounds like an issue with your code. You could then take it a step further and profile the code on the dev server to pinpoint the issue.
If that isn't an option, I would fire up Process Explorer and watch the server resources over time. I'm not sure what your service does, but it sounds like it could be spinning up threads and then not cleaning up after itself.
It might help if you posted some sample code so we could see what could've been affected by the change from .NET 2.0 to 4.0.
I would appreciate any advice regarding tools and practices I could use to confirm my recently completed website is performing correctly.
Although I am confident the code is not producing errors and is functionally operating as it should, I have little understanding of how to identify IIS, SQL Server and Windows performance/concurrency issues. For example if the website was briefly hit by a huge deluge of traffic, how would I be aware that event had ever happened and how would I know whether the website coped with it.
The website was written using ASP.NET 2.0 and C# running on Windows 2003 R2 Standard Edition, SQL Server 2005 Workgroup Edition and IIS 6.
Consider using a logging mechanism that also raises alerts, so when a database call takes too long, indicating a high server load, the logger raises a warning. Check out log4net.
Regarding tools and practises, I recommend badboy and jmeter as tools for load testing your site. Badboy is simple and can generate urls that may also be used in jmeter. The latter does a very good job load testing your site. Do tests that run over a long period and use different hardware setups to see how adding more web/app servers affect performance.
Also, check out PerfMon, a tool that lets you monitor a local or remote Windows server regarding contention rate, cpu load and so on.
You can use a load generating tool like WebLoad to capture and then replay (with possible variations through scripting) user interactions with your application's UI with lots of threads and connections.
As mentioned, load generation tools are quite helpful. One thing you can add for the database side is to use SQL Tracing. Setup a test plan with very specific steps, and as you step through your plan, trace the SQL that is running on the server.
This way, you can identify if certain actions are causing unnecessary/duplicate database calls. Also, you may discover very large and non-performant queries being run for very simple actions.
For SQL Server use the sys.dm_exec_requests DMV and check for CPU usage, reads, writes, blocking etc etc
select blocking_session_id,wait_type,*
from sys.dm_exec_requests