I want to add quartz scheduling to an ASP.NET application.
It will be used to send queued up emails.
What are the pros and cons of running quartz.net as windows service vs embedded.
My main concern is how Quartz.NET in embedded mode handles variable number of worker processes in IIS.
Here are some things to you can consider while you decide whether you should run embedded or not:
If you are going to be creating jobs ONLY from within the hosting application, then run embedded. Otherwise, run as a service.
If your jobs might need permissions that are different from the permissions that the web app has, run as a service.
If your jobs are long running jobs, or jobs that use a lot memory, run as a service.
If you need to run your jobs in a clustered environment for either performance, scalability or fault tolerance, run as a service.
From the items above you can deduce that my preference is to run it as a service. This is because if you are going to go through the trouble of setting up a job scheduler, this means that you have jobs that need to run on a schedule, or long running jobs. A service is usually the better choice for this type of work.
Quartz.NET can be instantiated on a per-application basis (web farm configuration mandates number of schedulers). You can safely run multiple schedulers if you have your jobs backed in a database and you have Quartz.NET configured in clustered mode (and clocks synced naturally).
The main concern is to the application pool handling prior to IIS 7.5. Without constant checks your application worker can get recycled and your scheduler will be down until someone issues a web request to fire up the application pool again. IIS 7.5 has the new feature to keep application pools running all the time.
Otherwise there should not be a big difference between the two models.
Related
I'm trying to understand exactly how Hangfire will behave if used in a web farm, where each ASP.NET application is configured identically, and there are N instances using the same shared SQL Server database for Hangfire storage.
The documentation just says that distributed locks are used to prevent race conditions, but this is a bit low-level, I need to understand what this means in practice.
Example:
If I have 5 web server instances, and I create a background job with a schedule that will run once a day at 5pm, does this mean that the first instance to obtain a 'lock' on the job will end up running it, and all other instances will ignore the job while it is locked?
I'm assuming that Hangfire will only allow one instance to process a job at a time, but I haven't confirmed it.
What about if I actually wanted to run a job on each server instance at the same time?
If anyone has any practical experience with Hangfire in a web farm, I'm all ears.
These are the basic rules that I've established after more research and testing:
Hangfire will execute a background job on the first Hangfire server that has available capacity based on the number of worker threads on that server
Hangfire will continue to execute background jobs on that server until the worker pool is saturated, at which point it will move to the next available server in the farm, and so on
Hangfire servers are automatically included in a web farm if they use the same Hangfire storage instance, so generally speaking no extra configuration is required.
If you want to run specific background jobs on specific servers or use different workload distribution algorithms, you can use named queues, where a queue is given a name and potentially assigned to a specific server instance, and background jobs must be scheduled to run on that queue as well.
I'm troubleshooting restarts in an ASP.NET application. The application is restarting about 20 times a day. We strongly suspect one part of the application because the restarts began when this particular feature when into production. I've added some logging to those pages using the log4net library, but I'm having trouble interpreting the logs.
When an ASP.NET application is running in an Application Pool, does only a single instance of that application run, or will multiple instances of that application run? I know several worker process threads will be spawned. What is the relationship of the worker process threads to the application(s) running in the App Pool?
I am thinking that I may not be interpreting the results correctly if there are multiple applications logging to the same log. If one is restarted but the other is not, the logs aren't really telling me much about what was happening when the restart occurred.
UPDATE 1
Looking at this page, I find the following information:
An application pool defines a group of one or more worker processes, configured with common settings that serve requests to one or more applications that are assigned to that application pool. Because application pools allow a set of Web applications to share one or more similarly configured worker processes, they provide a convenient way to isolate a set of Web applications from other Web applications on the server computer. Process boundaries separate each worker process; therefore, application problems in one application pool do not affect Web sites or applications in other application pools.
But I am still confused. I know from experience that I can assign two entirely different applications to use the same App Pool. Does that mean that exactly two worker processes will be spawned? Or can there be multiple worker processes spawned for the first app, and multiple worker processes spawned for the second app? If a problem happens in one worker process, can it take down every application running in that App Pool?
UPDATE 2
From this page about using WinDbg, I found this information (emphasis mine):
In IIS 5.x, there is only one Aspnet_wp.exe worker process and one debugger thread. Consequently, only one debugger can be attached to the Aspnet_wp.exe process at a time. This can pose a problem if you're dealing with multiple Web applications on the same machine. In IIS 6.0, you can coerce an AppDomain to run in a separate application pool. (For more information, see "IIS 5.x Process Model" and "IIS 6.0 Process Model" in Chapter 1.) Separate application pools provide multiple W3wp.exe processes. Multiple debugger threads are created in these processes (one in each), allowing you to debug more efficiently.
This sounds to me like each App Pool gets one w3wp.exe process. Am I interpreting that right? And if so, does that still apply in IIS 7.5?
Yes, each application pool is typically a single process1, but can contain multiple threads. You can assign multiple sites to an application pool, and those sites will all run under the same process, however they will run under different "app domains", which are security contexts that separate the code of one site from another, even if they're running on the same app pool.
Two users hitting the site at the same time can run on different threads, meaning they can run concurrently. That means any logging can have values interspersed. You might want to add a session value to your logging so you can sort based on session.
App pool restarts (recycling) are normal, 20 restarts in a day does not seem unusual. They can happen multiple times per day, and IIS controls when app pools are restarted. It does this whenever it feels it needs to clean up the pool.2 Your applications should be written in such a way as to recover gracefully from this (ie, do not keep anything in session that cannot be easily recreated if the app pool restarts).
The app pool can also restart when an unhandled exception occurs in your app. In that case, you want to address the cause of this. Such exceptions are usually logged in the event log.
1 – While you can configure an application pool to have multiple worker processes (this is known as a Web Garden), this is not a typical (nor generally recommended) configuration in my experience.
2 – Note that using IIS Manager you can configure an application to log recycle events to the Windows Event Log. You can also use IIS Manager to set the threshold for when several of the different types of recycle events occur.
Can anyone explain the differences, in IIS, between application pools, worker processes and app domains? Also, how do they work together? I've read a couple of articles but it's still a little confusing.
Does each website created in IIS becomes an application?
Is each application associated with one worker process?
Where do app domains come into the picture?
I try to say them with other words.
In a server you can have many asp.net sites that runs together. Each one site is an app domain.
You must assign to each of them one application pool. Many application domains (sites) can have the same application pool, and because they have the same application pool they run under the same processes, and under the same account - and they have the same settings of the pool. If this pool restarts, then all sites under that pools restarts.
Now each pool can have one or more worker process. Each worker process is a different program that's run your site, have their alone static variables, they different start stop calls etc. Different worker process are not communicate together, and the only way to exchange data is from common files or a common database. If you have more than one worker process and one of them make long time calculations, then the other can take care to handle the internet calls and show content.
When you assign many worker process to a single pool then you make the called web garden and your site is like to be run from more than one computer if a computer is one processing machine.
Each worker process can have many threads.
How the more worker process affect you:
When you have one worker process everything is more simple, among your application all static variables are the same, and you use the lock to synchronize them.
When you assign more than one worker process then you still continue to use the lock for static variables, static variables are not different among the many runs of your site, and if you have some common resource (e.g. the creation of a thumbnail on the disk) then you need to synchronize your worker process with Mutex.
One more note. Its sounds that when you make more worker process then you may have more smooth asynchronous page loads. There is a small issue with the session handler of asp.net that is lock the entire process for a page load - that is good and not good depend if you know it and handle it - or change it.
So let talk about one site only with many worker process. Here you face the issue that you need to synchronize your common resource change with Mutex. But the pages/handlers that use session they are not asynchronous because the session locks them. This is good for start because you avoid to make this synchronization of many points your self.
Some questions on this topic:
Web app blocked while processing another web app on sharing same session
jQuery Ajax calls to web service seem to be synchronous
ASP.NET Server does not process pages asynchronously
Replacing ASP.Net's session entirely
Now this session lock is not affect different sites.
Among different sites the more worked process can help to not the one site block the other with long running process.
Also among different sites the more pools also can help, because each pool have at least one worked process, but remember and see by your self using the process explorer, each working process takes more memory of your computer, and one big server with 16G memory and one SQL server can not have too many different worked process - for example on a server with 100 shared sites, you can not have 100 different pools.
One IIS server may have multiple application pools.
One web application binds to one application pool.
One application pool may have more than one worker process (when Web Garden is enable).
One worker process can have multiple app domains. One app domain lives only in one worker process.
One app domain may have multiple threads. One thread can be shared by different app domains in different time.
The meaning to ASP.NET developers: to make your web site scalable, don't use in-proc session and don't use static class variable lock for synchronization.
Yes, though not every application is a website. You can have an application that is nested under a website.
Yes, every application has to have one worker process (application pool), though one application pool can server several applications. A single web application can be distributed (web garden/farm) meaning that it will run in multiple processes.
Each process will run in its own app domain (every application pool is a separate app domain).
From MSDN.
Create a Web Application:
An application is a grouping of content at the root level of a Web site or a grouping of content in a separate folder under the Web site's root directory.
Application Pools:
An application pool defines a group of one or more worker processes, configured with common settings that serve requests to one or more applications that are assigned to that application pool. Because application pools allow a set of Web applications to share one or more similarly configured worker processes, they provide a convenient way to isolate a set of Web applications from other Web applications on the server computer. Process boundaries separate each worker process; therefore, application problems in one application pool do not affect Web sites or applications in other application pools. Application pools significantly increase both the reliability and manageability of your Web infrastructure.
From the source link:-http://weblogs.asp.net/owscott/archive/2007/09/02/application-vs-appdomain.aspx
An application is an IIS term, but it's one that ASP.NET utilizes.
Essentially it creates a sandbox, or a set of boundaries to separate
different sites, or parts of sites, from the others.
An AppDomain is a .NET term. (In IIS7, AppDomains play a larger role
within IIS, but for the most part it's an ASP.NET term)
The worker process is used to process the request of the web application.
I have a WCF service hosted in IIS which takes a long time (around 5 hours) to execute. the WCF service basically generates some reports using SSRS (SQL server reporting services) and saves them to a locaton on the server. this service was actually stopping after generating few reports, so I disabled the "recycling of worker process", "shut down worker processes after being idle" and "limit kernel request queue" in application pool and that fixed the issue and all the reports were generated regardless of the amount taken to generate them. but I am not sure if this is the right fix for this and I would like to know what is the impact of unchecking these settings in application pool for the WCF service in IIS? and is there any better way to get around this problem?
For any long running process it is much better to do it outside of IIS.
In this case I would have a regular windows service running that monitored a request queue. When a request comes in to generate a report, it would then spin off a thread to perform the generation.
The web service would be responsible for 3 things. First, adding an item to the queue to be handled. Second, checking status on the queue as to whether the report is ready. Third, sending the completed report back to the calling client.
This would allow the client to essentially do a fire and forget on the report request and call back later to check on it's status. Further it would mean that if IIS recycled for whatever reason you are still OK.
For bonus points I would add some error handling code that when the windows service restarted it could restart report jobs that were in the middle of execution. This would make it a bit more robust and allow you to reboot the server at any point.
I have disable also all the automatic shut down process from iis for my application with out any issue. I have monitor the memory limits and of course the program work smoothly with out any issues on memory.
I think that this automatic shut down triggers are designed mostly for process that keep too many web sites together and possible some of them have not good programming. But if you are the master of your iis, and you have check your program that have not memory issue, then is better to not shut it down, or at least control the shut down process with some way.
Ok is better to make long running process outside IIS, but is not so simple to develop it, not so simple to install it, not so simple to check it out.
I have a WCF service installed on IIS7. I noticed that the first call to my service is always very very slow. The subsequent calls are much faster & acceptable.
If there are no calls made to the service for some time, it again goes to sleep mode. After this the next call again takes a long long time.
Any remedies for this problem?
It is because of process management on IIS. When there are no calls for certain period of time IIS release the recourses and stops the process.
This is why you can notice that it is slow for first request and for requests after a long delay. Because while the first request or after long period of silence IIS loads everything from scratch. JIT complier runs and etc...
Also note :
When you are hosting WCF services on IIS, the WCF services enjoy all the features of ASP.NET applications. You have to be aware of these features because they can cause unexpected behavior in the services world. One of the major features is application recycling, including application domain recycling and process recycling. Through the IIS Management Console, you can configure different rules when you want the recycling to happen. You can set certain thresholds on memory, on time, and on the amount of processed requests. When IIS recycles a worker process, all the application domains within the worker process will be recycled as well
If you need automatic starting: The Windows Service Control Manager allows you to set the startup type to automatic, so that as soon as Windows starts, the service will be started, without an interactive logon on the machine. So you can use Windows service as a host.
More details you can check in Hosting and Consuming WCF Services.
There is another approach through which you can make it better. We have some kind of scehduled process which keeps hitting our server like every 5 mins with very light 'fetch' requests to keep all servers "hot" (by loading most of the required dlls etc) so that user experience is far better.
I agree it is not a fool proof way but still is something you can consider apart from increasing the recycling settings in IIS.