My current application is comprises of 3 tier- Web tier - App Tier - Database
While testing with 100 users, we found that App tier's cpu is touching almost 90% where as web server and database server are doing fine.
I am not able to figure out what code is causing high cpu usage. Majorly we have CRUD operation there. We take input in the form of DTO, we transfer them into entities (using Entity framework), add/update/delete into database. In case of Get operation, we fetch data into EF entities, store them in DTO and then send DTO to client.
I have tried to use DebugDiag but could not figure out any useful information.
Following are the server's configuration:
Web Server (Quantity = 1) Processor Intel Xeon CPU X5675 #3.07 GHz 2.19 GHz
Number of Cores (Virtual) 8
RAM 8GB
Operating System Windows Sever 2012 Standard
Processor Type 64 Bit
Softwares Installed NET Framework 4.5
App Server (Quantity = 1) Processor Intel Xeon CPU X5675 #3.07 GHz 3.07 GHz
Number of Cores (Virtual) 8
RAM 8GB
Operating System Windows Sever 2012 Standard
Processor Type 64 Bit
Softwares Installed NET Framework 4.5
DB Server (Quantity =1) Processor Intel Xeon CPU E7-4830v2 # 2.20 GHz 2.19 GHz
Number of Cores (Virtual) 8
RAM 8GB
Operating System Windows Sever 2012 Standard
Processor Type 64 Bit
Softwares Installed Microsoft SQL Sever 2014
There is no better solution than to install an APM tool. With them you'll find the root cause very quickly. AppDynamics or NewRelic are easy, Dynatrace a bit more complex but maybe more powerful.
Else go on shooting in the dark
Windows sysinternal tool Process Explorer(Procexp) is a good tool to find the high CPU process and thread call stack(method calls)
OR
- Collect multiple full user dump using task manager/Procexp on the high CPU process
And collect perfmon log with Thread counter. Perfmon -> Add counter -> Thread under thread select %Processor Time, ID thread, ID process.
From the perfmon you can find the high CPU thread ID. Now you can co-relate the thread ID with debug diag analysis report and find the thread call stack.
Hope this helps.
Thanks,
Parthiban
Related
I have a Windows Server 2012 R2 with 128Gb of memory and INtel Xeon E5 2650 v3.
Any time the memory usage for the App Pool goes close to 2.4Gb (according to Task Manager) I'm starting to get Out of memory exceptions and I have to recycle the app pool.
The Virtual Memory Limit is set to 0 for this app pool.
The overall server memory usage is always under 25%.
How do I increase the app pool memory limit?
How do I make it so that is not running as 32-bit application? Is this happening because I enabled Enable 32-bit applications? I actually need this because I have a managed assembly which is 32-bit only.
What is other solution for this? Don't say check your code, I need for certain cases that memory usage amount.
Thx
32 bit applications can address a maximum of 2 GB. Also see https://msdn.microsoft.com/en-us/library/windows/desktop/aa366778(v=vs.85).aspx
I am running a large scale ERP system on the following server configuration. The application is developed using AngularJS and ASP.NET 4.5
Dell PowerEdge R730 (Quad Core 2.7 Ghz, 32 GB RAM, 5 x 500 GB Hard disk, RAID5 configured) Software: Host OS is VMWare ESXi 6.0 Two VMs run on VMWare ESXi .. one is Windows Server 2012 R2 with 16 GB memory allocated ... this contains IIS 8 server with my application code Another VM is also Windows Server 2012 R2 with SQL Server 2012 and 16 GB memory allocated .... this just contains my application database.
You see, I separated the application server and database server for load balancing purposes.
My application contains a registration module where the load is expected to be very very high (around 10,000 visitors over 10 minutes)
To support this volume of requests, I have done the following in my IIS server -> increase request queue in application pool length to 5000 -> enable output caching for aspx files -> enable static and dynamic compression in IIS server -> set virtual memory limit and private memory limit of each application pool to 0 -> Increase maximum worker process of each application pool to 6
I then used gatling to run load testing on my application. I injected 500 users at once into my registration module.
However, I see that only 40% / 45% of my RAM is being used. Each worker process is using only a maximum amount of 130 MB or so.
And gatling is reporting that around 20% of my requests are getting 403 error, and more than 60% of all HTTP requests have a response time greater than 20 seconds.
A single user makes 380 HTTP requests over a span of around 3 minutes. The total data transfer of a single user is 1.5 MB. I have simulated 500 users like this.
Is there anything missing in my server tuning? I have already tuned my application code to minimize memory leaks, increase timeouts, and so on.
There is a known issue with the newest generation of PowerEdge servers that use the Broadcom Network Chip set. Apparently, the "VM" feature for the network is broken which results in horrible network latency on VMs.
Head to Dell and get the most recent firmware and Windows drivers for the Broadcom.
Head to VMWare Downloads and get the latest Broadcom Driver
As for the worker process settings, for maximum performance, you should consider running the same number of worker processes as there are NUMA nodes, so that there is 1:1 affinity between the worker processes and NUMA nodes. This can be done by setting "Maximum Worker Processes" AppPool setting to 0. In this setting, IIS determines how many NUMA nodes are available on the hardware and starts the same number of worker processes.
I guess the 1 caveat to the answer you received would be if your server isn't NUMA aware/uses symmetric processing, you won't see those IIS options under CPU, but the above poster seems to know a good bit more than I do about the machine. Sorry I don't have enough street cred to add this as a comment. As far as IIS you may also want to make sure your app pool doesn't use default recycle conditions and pick a time like midnight for recycle. If you have root level settings applied the default app pool recycling at 29 hours may also trigger garbage collection against your child pool/causing delays even in concurrent gc where it sounds like you may benefit a bit from Gcserver=true. Pretty tough to assess that though.
Has your sql server been optimized for that type of workload? If your data isn't paramount you could squeeze faster execution times with delayed durability, then assess queries that are returning too much info for async io wait types. In general there's not enough here to really assess for sql optimizations, but if not configured right (size/growth options) you could be hitting a lot of timeouts due to growth, vlf fragmentation, etc.
I have Visual Studio 2015 on x64 machine (Intel i5, 8GB RAM, Windows 10) running a Windows Communication Foundation (WCF) service. The service runs for like 5 hours to process data and it uses the IIS Express web server. When it is running the CPU usage of the IIS is about 13% every time.
One time I noticed that the data was getting processed very fast and I looked at the Task Manager and the CPU usage of the process was about 40%, but I can't make the IIS process go to 40% again. I don't know what happened that increased the CPU usage to 40% that time.
Is there a way to increase the CPU usage of the IIS (Express) so that the computation runs faster and therefore the data gets processed faster? Is there any settings that can be changed to accomplish this. I think the CPU should be above 14% every time. Is there a setting that specifically restricted this CPU usage in the IIS settings?
In IIS 8.5, there is Limit (percent) property per app pool settings which allows % of CPU time consumed by worker thread related to that app-pool
We have a small (for now) Asp.Net MVC 5 website on a dedicated VPS. When I go to the server and fire-up task manager, I see that "SQL Server Windows NT - 64 bit" is using around 80% of CPU and 170MB of RAM and IIS is using 6% CPU and 400MB of RAM. Server Specs are:
CPU 1.90Ghz dual core
Memory 2GB
Windows Server 2012
SQL Server Express 2012
Disk Space: 25GB, 2.35 Free.
The database is not very big. Its backup is less than 10MB.
I have tried to optimize the website as much as I could. I added caching to a lot of controllers and implemented donut caching for quite a lot of controllers. But today, even though there were only 5 users online, our search wouldn't work. I restarted the Windows on the server and it started working but I got the high CPU usage the minute server started. Interestingly when I open the SQL Server Management Studio and try to get the report for top CPU-consuming queries it says that there are no queries currently consuming any CPU!!! But at the same time I can see that SQL server is consuming a lot of CPU. How can I examine what is taking all the CPU? Below is a picture from the server:
I was/am very careful with designing and implementing the website. All the database access is through latest version of Entity Framework. I just wonder if the server's specs are low. Any help would be very much appreciated.
Update:
Here's the result of the sp_who2 stored procedure.
This could happen if the memory set to use is more than the available memory on the box. The default memory setting of 2147483647MB. In our case the AWS box had only 30.5 GB so we changed the setting to 26GB and the CPU usage fell to 40%. You generally want to leave 20% of memory for OS and its operations.
I would agree running SQL Profiler to spot large query durations and large write operations. Try running perfmon and spotting any potential connection leaks (reclaimed connections).
We have a vanilla ASP.NET application (ASP.NET web forms, Entity Framework, SQL Server 2005) without any explicit multithreading from code. It has been deployed on the stagging environment (OS - Windows Server 2008 R2 64 bit, CPU - Intel Xeon E5507 # 2.27 GHz 2.34 GHz, RAM - 7.5 GB). This environment is composed of a web, a database and a reporting server each a separate instance in the cloud (Amazon EC2). When testing for concurrency, observations are as under:
1 user - CPU usage ~25%, Response time 2-4 seconds
2 users - CPU usage 40-50%, Response time 3-6 seconds
4 users - CPU usage 60-80%, Response time 4-8 seconds
8 users - CPU usage 80-100%, Response time 4-10 seconds
My questions are:
Is CPU usage is relative to no. of concurrent users? And the response time can vary to a great extent as seen in the above observations?
As from the above observations, CPU will be maxed out when concurrent user counts is ~10. Shouldn't the CPU handle much more concurrent users seamlessly without a drastic increase in the response time? In ideal scenario, in case of a basic ASP.NET application, how many concurrent users a CPU can handle?
If yes in the above question, what could be the problem here for high CPU/long response time? What ways we should go ahead for effective debugging to find out bottlenecks in code/IIS settings?
PS: IIS settings (i.e. in machine.config) which have been changed:
maxWorkerThreads = 100
MinWorkerThreads = 50
maxIOThreads = 100
minIOThreads = 50
minFreeThreads = 176
maxConnections = 100
The high CPU usage could be caused by a wide variety of things. The easiest way to find out what's going on is to use a profiling tool:
ANTS Memory Profiler:
http://www.red-gate.com/products/dotnet-development/ants-memory-profiler/
ANTS Performance Profiler:
http://www.red-gate.com/products/dotnet-development/ants-performance-profiler/
They're very affordable, but you should be able to work unrestricted with the trial versions. These tools do a fantastic job of identifying bottlenecks and memory leaks.