We have a .net web application hosted on IIS 7.5.
Earlier this application was running on a 32bit application pool but some time ago we've switched to 64 bit application pool.
Recently users have started to complain that after 1-2 minutes of idling their session is being killed which we have confirmed today.
In the web.config file the session timeout is set to 60 minutes.
Also we have noticed in task manager that the w3wp process for this application consumes about 2-2,4GB of memory so maybe the problem is that the application pool is trying to recycle some memory?
The recycling is set to limited time periods 21:00 and 4:00
What could be the reason for this problems with sessions?
EDIT:
I have inspected some counters and done the basic memory dump analyze but I don't see any problems.
In the dump eeheap analyze I see only generation 2 objects about 10-30MB for every heap and I have 24 of them
Heap 0 (0000000003083a90) generation 0 starts at 0x00000000fff568b8 generation 1 starts at 0x00000000ffa6acf0 generation 2 starts at 0x00000000ff471000 ephemeral segment allocation context: none segment begin allocated size 00000000ff470000 00000000ff471000 00000000ffff8de0 0xb87de0(12090848) Large object heap starts at 0x00000006ff471000 segment begin allocated size 00000006ff470000 00000006ff471000 00000006ff7495c8 0x2d85c8(2983368)
Heap Size: Size: 0xe603a8 (15074216) bytes.
Heap 1 (00000000030889c0) generation 0 starts at 0x000000013fc36ed8 generation 1 starts at 0x000000013f949348 generation 2 starts at 0x000000013f471000 ephemeral segment allocation context: none segment begin allocated size 000000013f470000 000000013f471000 000000014035e7b8 0xeed7b8(15652792) Large object heap starts at 0x0000000703471000 segment begin allocated size 0000000703470000 0000000703471000 00000007035c5d58 0x154d58(1396056) Heap Size: Size: 0x1042510 (17048848) bytes.
EDIT: 2015-08-19 09:00
Those are the counters for 09:00 2015-08-19
What worries me is why the memory in task manager shows 2,5GB when the Bytes in all Heaps shows only about 100MB and why the Private Bytes (216MB) are bigger then Bytes in all Heaps?
The load in this current moment is about 40 users on this server.
EDIT 2015-08-19 14:09
After some time I see that there could be a problem with assemblies.
How can I check this with windbg when I'm on .NET 4.5 where there is no !dda command?
Try copy the running app to a different pool but in this new one disable all assemblies / references that you dont need, to see what is doing that.
Like you said i think that some assembler is crashing your application pool, maybe because maybe isnt support for 64 bits.
Try disabling all references that you dont use, update all, etc.
Related
I have an interesting problem with how Windows and .Net manage memory for Asp.Net applications that I can't explain myself. The problem is that I have a big Asp.Net application that after starts up can take about 1 GB memory according Resource Manager. We tried to test how many instances of the application we can run at the same time on a single machine with 14-16 GB memory.
First test is with an Azure Windows 2016 server with 8 vCPUs, 14 GB RAM, HDD.
After a few instances:
After 30 instances:
As you can see, private byes and working set of some instances reduced a lot. Based on what I read from how memory is managed (aka working set, physical memory, virtual memory, page files...), I can understand how the OS can take physical memory away from some idle processes for the others that are in need. So far so good.
Then we tested the same scenario with another Azure Windows 2016 server with 4 vCPUs, 16 GB RAM, but this one uses SSD.
After about 20 instances, we got OutOfMemoryException:
The key difference I could see is that memory of all those w3wp processes were still high. In other words, they were not reduced as in the test above.
My question is why the behaviors were different? What prevented the second cases from saving memory to page file (my guess!) and thus caused OutOfMemoryException?
Checking pagefile setting showed us that it was stilled enabled in "System managed size" mode but somehow Windows refused to use it for the w3wp processes. We tried to change it to custom size and set it to 20 GB and everything started working again as expected. I must admit that I still don't know why Windows 2016 behaves like that when SSD is used though.
I am running a large scale ERP system on the following server configuration. The application is developed using AngularJS and ASP.NET 4.5
Dell PowerEdge R730 (Quad Core 2.7 Ghz, 32 GB RAM, 5 x 500 GB Hard disk, RAID5 configured) Software: Host OS is VMWare ESXi 6.0 Two VMs run on VMWare ESXi .. one is Windows Server 2012 R2 with 16 GB memory allocated ... this contains IIS 8 server with my application code Another VM is also Windows Server 2012 R2 with SQL Server 2012 and 16 GB memory allocated .... this just contains my application database.
You see, I separated the application server and database server for load balancing purposes.
My application contains a registration module where the load is expected to be very very high (around 10,000 visitors over 10 minutes)
To support this volume of requests, I have done the following in my IIS server -> increase request queue in application pool length to 5000 -> enable output caching for aspx files -> enable static and dynamic compression in IIS server -> set virtual memory limit and private memory limit of each application pool to 0 -> Increase maximum worker process of each application pool to 6
I then used gatling to run load testing on my application. I injected 500 users at once into my registration module.
However, I see that only 40% / 45% of my RAM is being used. Each worker process is using only a maximum amount of 130 MB or so.
And gatling is reporting that around 20% of my requests are getting 403 error, and more than 60% of all HTTP requests have a response time greater than 20 seconds.
A single user makes 380 HTTP requests over a span of around 3 minutes. The total data transfer of a single user is 1.5 MB. I have simulated 500 users like this.
Is there anything missing in my server tuning? I have already tuned my application code to minimize memory leaks, increase timeouts, and so on.
There is a known issue with the newest generation of PowerEdge servers that use the Broadcom Network Chip set. Apparently, the "VM" feature for the network is broken which results in horrible network latency on VMs.
Head to Dell and get the most recent firmware and Windows drivers for the Broadcom.
Head to VMWare Downloads and get the latest Broadcom Driver
As for the worker process settings, for maximum performance, you should consider running the same number of worker processes as there are NUMA nodes, so that there is 1:1 affinity between the worker processes and NUMA nodes. This can be done by setting "Maximum Worker Processes" AppPool setting to 0. In this setting, IIS determines how many NUMA nodes are available on the hardware and starts the same number of worker processes.
I guess the 1 caveat to the answer you received would be if your server isn't NUMA aware/uses symmetric processing, you won't see those IIS options under CPU, but the above poster seems to know a good bit more than I do about the machine. Sorry I don't have enough street cred to add this as a comment. As far as IIS you may also want to make sure your app pool doesn't use default recycle conditions and pick a time like midnight for recycle. If you have root level settings applied the default app pool recycling at 29 hours may also trigger garbage collection against your child pool/causing delays even in concurrent gc where it sounds like you may benefit a bit from Gcserver=true. Pretty tough to assess that though.
Has your sql server been optimized for that type of workload? If your data isn't paramount you could squeeze faster execution times with delayed durability, then assess queries that are returning too much info for async io wait types. In general there's not enough here to really assess for sql optimizations, but if not configured right (size/growth options) you could be hitting a lot of timeouts due to growth, vlf fragmentation, etc.
We have installed BizTalk 2013 R2 and deployed a simple solution.
What we observed is the memory consumed by BizTalk service keeps on growing.
It does not come down even after it has completed the processing.
Please find the details of the tests done.
BizTalk Solution (contains 2 schemas, 1 map and 1 orchestration).
scenario 1
Test File Size: 2 KB
No of files:250
Start memory : 12MB
END memory :122 MB
scenario 2
Test File Size: 2 KB
250 files processed 3 times one after other
start memory 13.2 MB
end memory 160 MB
scenario 3
Test File Size: 2 KB
250 files processed 6 times one after other
start memory 13.2 MB
end memory 215 MB
BizTalk will actually "cache" the assemblies in memory for a while.
This actually means that the next time the process runs it will have less of a start-up time as it is already in memory.
If the process is not called for a while it will unload it from memory, unless you've configured it to stay in memory permanently which is also possible.
That is also the reason you have to restart the BizTalk hosts if you update the assembly in the GAC. This forces it to unload the assembly from memory and will load it only if the process that needs it runs.
So what you need to do is monitor your BizTalk server for a longer period while it is not processing those files, and eventually you will see it release the memory again.
A tool to do this monitoring and detect memory leakage and other issues is Performance Analysis of Logs (PAL) Tool, this will help you log the data from the Performance Counters and then analyse the results against thresholds to detect issues.
Also did you try running any other applications/services just after you performed this test? .Net framework will activate GC if there is memory in demand by other processes and you could probably see it coming down.
The problem is that IIS worker consumes to much memory. After inspecting the w3wp process with VMMAP I noticed that the bigest component of the Private WS is the Managed Heap, the GC memory.
Further, I inspected the w3wp process using Performance Monitoring, and the results were as follows:
# Bytes in All Heaps : 32MB
# Gen 0 Collections : 4
# Gen 1 Collections : 3
# Gen 2 Collections : 2
Gen 0 Heap Size 570MB
Gen 1 Heap Size 5MB
Gen 2 Heap Size 26MB
Active Sessions : 4
Gen 0 heap size is increased with every new session. The peak is when I have 4 active session(~570MB). Than when I have 6 session it decreases to ~250MB and than increases again until the application pool is recycled(~8-9 active sessions).
As I know the Gen 0 heap size has to be very small(comparable with the L2 Cache) and this is the size that triggers the GC to run Gen 0 GCs.
Why is the Gen 0 heap size so big?
I have the following enviroment:
IIS 6.0
The application is Asp.Net WebForms
Application Pool is restricted to 700Mb, and it gets recycled when
I have ~8-9 active sessions, so all session are lost.
.Net Framework v4.0.3
64 bit version of w3wp worker.
I also inspected the application memory using CLR profiler and the
number of Bytes in all heaps are 10-60 mb depending on number of active sessions.
Thank you!
http://msdn.microsoft.com/en-us/library/ee817660.aspx
Use WinDbg or any commercial .NET memory profiler, you should be able to review what are the objects in the heap and whether they should be there.
Common causes are string manipulation without StringBuilder and big objects in session such as DataTable.
Find the exact cause in your case and remedy it.
I am running IIS6 on Windows 2003 Server 32 bit. I have read that IIS6 has a maximum virtual memory limit of 2gb (3gb with the 3gb switch fipped).
What I am unclear on is whether this means all ASP.NET sessions have 2gb between them or 2gb each.
So if I have a session variable storing 200kb and have 10,000 active sessions am I going to be hitting up against this 2gb limit?
In general the advice is to leave these options unticked for ASP.NET applications, it affects how quickly the appPool recycles more information here summary below:
Physical and Virtual memory: This section is for recycling application pools which consume too much memory. Focusing on physical I typically like to limit app pools around 800MB to 1200 MB max on a 32 bit app with very few app pools depending on the number and amount of memory. On a server with 2 GB RAM I'd set it at around 800MB max. On a 4GB of RAM server around 1GB and more if more with a max around 1200. On a 64 bit web front end with 8-16 GB memory I've heard of settings of 2GB of RAM or even allowing it to let it ride, rather than limiting it.
You really need to profile it since these can really grow to process and cache. The greater the amount of memory and the greater the load the higher the worker process will grow. When people ask about configuring the app pool, this is where they are usually asking what the numbers should be. What you are doing here is explicitly limiting the app pool from consuming more memory.
Notice this setting is on the recycle tab, there's a reason for that. When an app pool reaches the max it isn't like the max processor setting. It will cycle the worker process which is like a tiny reboot or similar to an iisreset, but not since sometimes we want this to happen so we can release our memory. You really don't want to cycle more than a couple of times per 24 hour period in an ideal world. I've heard of some trying to cycle right before the morning peak occurs so they have the most amount of memory available, then a cycle right at the end of the day before the backups or crawling begins.
Basically the recommendation is not setting a limit (leave the options unchecked) because once the limit is hit IIS will recycle the application pool causing all active users to be temporarily disconnected from the site. You users will likely receive an HTTP 500 while the application pool recycles and then once it's back there will be a delay while the application pool warms up.
Sessions
For an application of any size do not use InProc (stored in memory) sessions use state server or SQL server to store your sessions. http://msdn.microsoft.com/en-us/library/ms178586.aspx
Conclusion
It really depends on the profile of your application, if your expecting 10,000 active sessions though don't use InProc, don't use IIS6 and don't use a 32 bit server.