I had an on-going enterprise development which involved Quartz to do daily/weekly jobs such as
Clean up temporary folder
Send Emails
Update databases
Publish items
and I'm embedding Quartz into an ASP.NET application with always run, no idle timeout, & recycles app pool every day at 4am, & it works perfectly.
I've been searching on Stackoverflow & many people suggested to make it a Windows service, however those answers were a few years old - before IIS 8 became available.
I wonder if my implementations has some cons & will cause side effects that yet to be known? Only 1 ASP.NET application to be run in this machine.
Related
I have a website that is built on top of ASP.NET .NET Core 2.0 and it worked for 8 months without any issues but 2 days ago i got a message from the host provider that my website is consuming 33GB of RAM which made them suspend my account so i asked them to limit my resources according to my Plan which is only 2GB physical and 4GB Virtual, and that made my website pool stops about 6 or more times per day and i have to restart it manually to get the website back,
i have tested they website using Resharper DotMemory Tool and the Visual Studio
Diagnostic Tools (which i'm new for such operation), and i see that the website start with about 300MB of RAM and it keeps raising while using the website, but its top is 1.1GB RAM as here (
Screenshot)
so i'm not sure what should be the solution as the host provider tells me to upgrade to VPS or optimize my website(that i haven't made any Coding changes for about 3 months (Only Data Entry))
We have 4 servers load balanced:
4 cores # 2.6Ghz (E5-2650 v2)
14GB RAM
Windows 2012 R2
High Performance power setting
IIS 8.5
ASP 5.3
EF 6.1
They each have a single application pool with one worker process and a single website. Each server has its own copy of the site (DLLs & views), running on a local disk. We are using IIS virtual directories to point to shares on a clustered file server for log files and common images etc (content only). The application pools are set to not shut down when idle (interval of 0) and we have also disabled the every-1740 minute recycle interval too.
We have New Relic's .NET agent installed on all servers, and looking through our slow transaction log, I can see that many requests are taking 15 seconds or so to complete. Looking into the trace, I can see a common call to System.Web.Compilation.AssemblyBuilder.Compile() and System.Web.Compilation.BuildManager.CompileWebFile().
As far as I know or understand, ASP would compile these views upon first request to them, and cache it (to the temporary ASP files in C:\Windows\Microsoft.Net) and then load from there for subsequent requests.
I'm confused how this is happening so often - when I visit these URLs, the TTFB is about 400ms, and due to constant load I can't see the websites "losing" their cache and needing to compile the views again. These pages are frequently hit - it's an e-commerce store and I can see that it happens often, and on our most popular pages: catalogue (category/brand/gender etc) listings and product details.
I've set the settings against each application pool to log an event when recycling, and there have been no events logged when I'm checking the WAS service in the event viewer. We also have New Relic server installed, and looking over the past 6 hours' worth of data, I can't see any dip in RAM usage on any of the servers - which would indicate the application pool recycling. This has really baffled me!
I'm thinking of moving towards pre-compiling our views as part of our release process - it makes sense really. But it feels like that is working around, or masking an issue which as far as I can see should not be happening. We build our site in Release mode and have <compilation debug="false" /> on all web.config files.
Can anyone think of any causes for this?
It is because of how JIT (Just-In-Time) compilation works.
When you build your application, it is converted into .NET Microsoft Intermediate Language (MSIL) or Intermediate Language (IL).
As your applications is accessed, Common Language Runtime (CLR) converts only executed IL parts of your code into native instructions.
Just-In-Time compilation process converts IL to native machine instructions and it is a part of CLR.
In a simplified terms when you run a .NET application and your program calls a method. JIT Compiler reads IL from metadata and compiles it into native instructions and run it. Next when your program calls the same method, CLR executes native CPU instructions directly. This process adds some overhead for the first method call. You can go with the other option of pre-compiling your application using NGEN, which is usually not recommended because you will loose some optimizations that only JIT can perform due to its awareness of underlying hardware platform. These two articles has more details
http://geekswithblogs.net/ilich/archive/2013/07/09/.net-compilation-part-1.-just-in-time-compiler.aspx and https://msdn.microsoft.com/en-us/library/ms366723.aspx
There are also other things you can try that might help you speed up your application. You can use IIS Application warm up module How to warm up an ASP.NET MVC application on IIS 7.5?, implement distributed caching etc to alleviate some of your application bottlenecks.
I've been meaning to get on here for ages, alas, my recent problem is motivating me, hoping to contribute and not just sponge, but hopefully someone can assist with this issue.
We have two 2003 web servers (looking to upgrade) running IIS6 running multiple ASP.net applications on one web site (via Microsoft NLB).
We have a persistent issue with one application, that is, the white screen of death, the application just hangs, no errors.
Things I have tried / observed:
Taken servers out of NLB
Checked HTTPERR for errors at time of issue, some bad requests and connection abandoned by app pool present
IIS logs show requests coming in
Checked database for locking PID's or performance, everything ok
Checked network connections and links to other API's all good
Checked other websites on server - not working ok
Event app logs no errors found at exact time of issue
Split out app pools, found the problem application
IIS reset and shutting down all IIS process and services gets it working again
So next when application hung, recycling that specific app pool gets it working for a short period
'SOME' parts of the same application will work, even though the main function hangs and gives the white screen of death
Other things to note:
Server has 4GB ram, terrible I know, 4gb page file, 1.7gb total in use
CPU usage always <12%
Resources not 'really' a major factor
Only one w3wp for this app pool (been running like this for years)
After setting the app pool to recycle on an hourly basis and tweaking some app pool settings, the issue seemed to go away for at least six months. Now it's back for whatever reason, no updates, no changes in application or infrastructure.
I've upped the worker process for the app pool to 2, the issue still persists after a period of time
I observed THREE worker process at random intervals when checking not the TWO set up, could be part of the recycle process (one starts up, other shuts down) and I caught it at that moment.
Killing the highest RAM usage W3WP re-instates application instantly when new process starts up for that app
Common RAM usage when site failing appears to be apprx 210mb on one of the applications W3WP
I have now set recycling rules instead of hourly to recycle at a more purposeful 200mb. Why this would be a limiting factor I do not know.
I'm struggling to get to the bottom of it, but it doesn't appear to be network, hardware, NLB, Database, or any other core factors. It is specific to a worker process for this app pool hanging / crashing and not processing requests for a certain part of the site.
I am running an effort to upgrade to 2008 in the next few months (version of ESX does not support 2012 and that upgrade is not in scope). However, I'd really like to work out what the issue is rather than using web deploy to deploy a dodgy installation of a legacy application.
Any advice or help appreciated.
Usually I would look at writing a Windows Service to manage tasks that aren't suited to being hosted in a web application. These types of tasks are usually long running processes or scheduled tasks. Although this is normally the primary approach for these types of tasks, people have looked at ways of running these kinds of background processes in a web application by kicking off a number of threads in the Application_Start event exposed by Global.asax. The problem with this approach has always been that if your IIS worker process dies, then your background thread is killed too (effectively your 'Windows Service' is stopped until the next request is received).
ASP .NET 4.0 offers a solution to this problem. You can now set the StartMode to 'AlwaysRunning' as described in this blog post by Scott Gu. Somewhere in the comments on this post, someone asks a question about the viability of hosting Windows Service type tasks in IIS since the new feature ensures the worker process is always running. Scott mentioned that it would definitely support the scenario. Further to this, the recent introduction of AppFabric means that Microsoft themselves are providing simple hooks for hosting and monitoring WCF and WF services in a web application.
What does this mean for those of us that used to write Windows Services to support our web apps? Should we adopt this model? What are the pitfalls? As far as I can tell, there are a number of benefits to hosting 'Windows Service' processes in a web application, the most useful being the ease of deployment. Furthermore, we can actually start developing simple user interfaces to our services which provide information about what is happening at runtime.
If I had to go this route, I don't think that I would host my 'Windows Service' type functionality in the customer facing web application. I would probably develop a new web application project (much like I would in the Windows Service context) that would host my long running/scheduled task processes. I guess there are few reasons for this.
Security. There may be a different security model for the UI displaying information about the running background processes. I would not want to expose this UI to anyone else but the ops team. Also, the web application may run as a different user which has an elevated set of permissions.
Maintenance. It would be great to be able to deploy changes to the application hosting the background processes without impacting on user's using the front end website.
Performance. Having the application separated from the main site processing user requests means that background threads will not diminish IIS's capability to handle the incoming request queue. Furthermore, the application processing the background tasks could be deployed to a separate server if required.
I would be really interested to hear your thoughts on this approach and whether I should be sticking with Windows Services. I am very tempted to try this new approach.
What does this mean for those of us that used to write Windows Services to support our web apps?
I think this a key scenario where you could be move away from a Windows service to using the continous running web site.
Should we adopt this model?
Standard development answer: Depends ;)
What are the pitfalls?
One issue I can see is the IIS dependency. If you need a service to run on a users machine I would not feel comfortable about asking them to install IIS just to run my service. Here I think the traditional model works better.
Monitoring and tracking are major issues, but as you also point out this is solved by AppFabric. It is even better than what you get from the Window Service. However you have added another dependency which also will require .NET 4.0 and a relatively new version of Windows. I could also be wrong here, but my understanding is that AppFabric is not supported in production on client OS's. Which could bring in additional headaches.
You will lose pause functionality in the continuous web site model too.
Finally IIS killing inactive app-pools isn't the only way an app pool can recycle. Editing a web.config file causes it for instance, which may not be an ideal situation.
the most useful being the ease of deployment.
I also think development is much easier - in the past I have had a console app and a windows service so I can dev/test on my machine using the console app and then change it to a windows service when it goes out. Now dev/test is MUCH easier.
A must read for this is Death to Windows Services...Long Live AppFabric!
What are the pitfalls?
One I found, no shutdown event. You have AppStart when the web site starts (not global.asax because that is HTTP only) but you have no way to handle shutdown which could mean disposing becomes an issue.
I would suggest sticking with a windows service. The issue is with your number 2.
You won't be able to update service part of web site without restarting whole web site.
After deploying a new build (mostly changed DLL's) of an ASP.NET web app the CPU on the server is now jumping to 100% every few seconds and the culprit is lsass.exe. Do you think that the deployment of the asp.net web app to the server and this problem are related? (or a coincidence that it happened at the same time?)
More info:
This is the first time that I've done the build on a Server 2008 x64 machine. Previous the builds were done on a Server 2003 x86 machine. Target is "Any CPU" so should work on either. Deployed to server is Server 2003 x86.
I've searched the web for more info on this and have confirmed that the process is lsass.exe (first character a lower case L and not an upper case i) so ruled out the virus version. Found some docs relating to a Server 2000 bug but doesn't apply here.
I eventually isolated the problem to an ASP forum running "under" that ASP.NET web app. Using the admin page on the forum I took the forum down and then brought it back up again and the problem disappeared. I find this very frustrating because the problem has now gone but I don't know what caused it and as such it could easily return.
I also installed this Microsoft Hotfix and rebooted this server but that didn't work.
Have you checked the System and Application event logs for anything unusual?
Have you updated to use Active Directory role provider? I've seen issues where enumerating groups to do role checking pegs the CPU and really slows down the app. I actually implemented a customized provider that allowed me to specify a particular OU and set of groups that I actually care about to get around this issue.
The xperf tools distributed in the Windows Performance Toolkit will tell you exactly what is usin CPU time or disk bandwith. These tools are free and work on any retail build of WS2008 or Vista. Here is series of posts on the xperf tools from myself.