Duplicate
This is a close duplicate of Dealing with a longer running process in WCF. Please considering posting your answer to that one instead of this.
Original Question
I'm implementing the business layer of an application that must run some background processes at scheduled times. The business layer is made up of several WCF services all running under the same web application.
The idea is defining a set of 'tasks' that must be run at different times (eg. every 5 minutes, everyday at 23:00, etc). That wouldn't be hard to implement as a windows service, but the problem is, the tasks need access to data caches that are living in the services, so this 'scheduler' must run under the IIS context in order to access that data.
What I'm doing currently is using a custom ServiceHostFactory in one of the WCF services which spawns a child thread and returns. The child thread sleeps and wakes up every X minutes to see if there are scheduled tasks and executes them.
But I'm worried about IIS randomly killing my thread when it recycles the application pool or after some inactive time (eg. no activity on any of the WCF services, which listen for requests from the presentation layer). The thread must run uninterrupted regardless of activity on the services. Is this really possible?
I have found an article by someone doing the same thing, but his solution seems to be pinging the server from the child thread itself regularly. Hopefully there is a better solution.
I have at some point implemented a Windows Service that would load a web page on a regular basis. The purpose of that was was that the site was hosting a Workflow Foundation runtime, and we wanted to ensure that the web application was brought back up after IIS recycling the application pool. Perhaps the same approach can be used in this case; have a service (or Scheduled Task in Windows; even simpler) run every x minutes and load a page that will check for tasks.
Is it a possibility to run either a Windows Service or place applications in the Windows Scheduler to execute methods in the WCF at certain times? Maybe use a BackgroundWorker inside the WCF. Another option would be for WCF to spawn other applications to do the business logic, passing the appropriate data, or pointers to the data in memory(unsafe).
Related
I have a single ASP.NET MVC 4 web application (MvcApplication which extends System.Web.HttpApplication) hosted on IIS 8. My application is running on a separate application pool in integrated pipeline mode. Maximum worker process count for this application pool is set to 1.
I understand, that application instances might get pooled and there might be multiple parallel threads, which will call Begin_Request/End_Request for the same instance of my application (AppDomain). Also I understand that if I set "Maximum worker process count" to more than 1 then there will definitely more than one instance of the application (AppDomain).
What I don't understand is how many calls to Application_Start will be issued and how many instances of my static variables (stored in MvcApplication and also there are some singletons, like NHibernate session factory etc.) will exist at a given moment.
Currently I have some problems with cleaning up .NET MemoryCache. Some users of my website are receiving old values from the cache, thus leading me to think that there might exist more than one instance of my global MemoryCache, even when I set "Maximum worker process count" to 1.
What is the maximal number of instances of my application static variables and singletons which will be run in parallel in a single worker process?
Is there any utility or script which would help me to enumerate all AppDomains on IIS?
I'm not sure if this applies to you, but HttpModule Initialize methods are called more than once because even though there is only one Application instance. One HttpApplication can spawn multiple HttpModule instances, so make sure those initializations are thread safe.
Application_Start and _End will fire only once per site start and stop with your current settings, so that will not be the issue. If you are using the HttpContext.Cache object, you might have the issue mentioned in this answer:
HashTable in HttpContext.Current.Cache
If you want, you can try using WinDbg and get a dump of what is going on in your process but that is a really heavy-handed tactic that probably won't shed as much light as just looking through the code. Not until you look at it a little harder, and are still stumped IMO.
Speaking of server resources (in general) and background processes. Would it be better to use a separate executable and a windows scheduled task or use the timer class and make use of the same resources as you application.
There are a few pros and cons to both methods, but what I'm wondering is this: Would making use of shared resources (thread pools and the like) be better than separate resources? Sure the process would be taking resources from the app, but isn't it technically already doing that either way?
you have given too little context to really understand the whole. how does the timer trigger the activity at certain time if the application is closed or there is nobody connected (logged on)? This kind of stays the same for both ASP.NET and Windows client because IIS takes the application down when nobody is connected for a while.
in my opinion a Windows' scheduled task is way better because you decouple from IIS application pool / application lifecycle and you also separate better and are sure that at that time the call will be executed and the activity started.
When IIS restarts an ASP.Net (2.0) web application, it can either:
Recycle the AppDomain: Unload the AppDomain and load a new AppDomain on the same process (e.g. when HttpRuntime.UnloadAppDomain() is called, when web.config is changed).
Recycle the process: unload the AppDomain and load a new one on a new process (e.g. when invoking Recycle command on an AppPool via inetmgr, or when memory limit is reached).
Due to some internal reasons (trouble with legacy native code we depend upon), we cannot allow the first option. We just can't load the application twice on the same process.
Can IIS be somehow told to never allow worker process reuse?
I've tried preventing it myself by tracking whether an application has already started on a process once using a Mutex, and if so - throwing an exception during Application_Start()); I've also tried terminating the process by calling Environment.Exit() during Application_End(). The problem with both methods is that it causes any requests that arrive during Application_End or Application_Start to fail (unlike a manual process recycle, which fails absolutely no requests because they are redirected to the new process right away).
I believe that "Recycle the AppDomain" comes under preview of ASP.NET runtime and IIS is not really involved anywhere (I am not 100% sure about this in case of integrated pipeline of IIS7). So I don't think that what you want is feasible. But there are couple of workaround that you may consider for your problem:
Ensure that you run start-up code (manipulating legacy code) to run only once - this should be possible via named system semaphores. Once system semaphore is created by app start-up in worker process, it will exists till process is recycled so you can have per process initialization.
If #1 is not possible then consider hosting code manipulating legacy code in a separate process all together - this process can expose relevant functionality via WCF services over named pipes. ASP.NET will consume them to use legacy code.
Couldn't find a way to tell IIS that worker processes are not to be reused. Can't afford fixing the underlying problem that forbids process reuse. Therefore, ended up calling Environment.Exit(0) in Application_End, although it may cause a small number of requests to fail with a connection reset.
Problem: Some 300 candidates make a test using Flex. A test consist of some 100 exercises. After each exercise a .NET service is called to store the result. If a candidate finishes a test, all the data of his/her test is denormalized by Asp.NET. This denormalization can take some cpu and can take 5 to 10 seconds. Now, most of the times, some of the candidates have finished their test earlier than the rest, but still some 200 of them wait until their time is up. At that moment, 200 candidates finish their test and 200 sessions are denormalized at the same time. At this point, server load (cpu) is too high and cause calls to the webserver to go wrong. Now, instead of all these sessions being normalized concurrently, I would like to add them to a queue using MSMQ.
Question:
How do you process the Queue?
Do you start a separate thread in the Application_Start of global.asax that listens to the queue? If there are messages, they are dealt one at the time.
Is it necessary to do this in a separate thread? What if in the global.asax you just call a singleton for instance that starts listening to the queue? In what thread will this singleton run? (what's the thread that calls global.asax)
What are best practices to implement this? Links? Resources? Tutorials? Examples?
I don't like the idea, but could you put an exe on the root of your website, an exe that starts a process listening to the queue...
If you get a message out of the queue, do you remove it when you pull it out or do you remove it if denormalization for this session was successful? If you remove it when you pull it out and something goes wrong...
I could also create my own queue in memory, but restarting the webserver would empty the queue and a lot of sessions would end up not being normalized, so I guess this is really a bad idea.
Is MSMQ a good choice or are there better alternatives?
You could consider using a WCF-Service with MSMQ transport. I used this approach in an application that calculates commissions:
User completes asp.net wizard configuring calculation parameters
Calculation Job is sent to WCF-Service using MSMQ transport
Service transaction is completed as soon as Job entered MSMQ
New transaction scope is created for processing Job instances
One drawback is that the transaction will require MSDTC which will add some overhead when targeting MS SQL Server and even more when dealing with Oracle.
IDesign provides a lot of useful samples and best practices on WCF queueing.
Personally, I use a servicebus for scenario's like that. I know this sounds like an overkill, but I think the .net servicebusses are so good that they require the least amount of code written by you, because it's not easy to create a good scheduler for background processes without disturbing the threads of the application pool the webapp is running in. NServicebus and MassTransit are both good an well enough documented servicebuses for your scenario. With a servicebus, you have a framework that writes to msmq and listens to msmq in several apps connected by the messagequeue. The bus makes it easy for you to create a separate app that runs as a background service and is connected with your web-app by the message queue. When you use topself (included in nservicebus and masstransit), an installer/uninstaller for the seperate apps is automatically generated by the service bus.
Question: Why don't you like the idea of having a separate exe?
How do you process the Queue?
Do you start a separate thread in the Application_Start of global.asax
that listens to the queue? If there are messages, they are dealt one at
the time.
Is it necessary to do this in a separate thread? What if in the
global.asax you just call a singleton for instance that starts listening to
the queue? In what thread will this singleton run? (what's the thread that
calls global.asax)
[skip]
I don't like the idea, but could you put an exe on the root of your website, an exe that > starts a process listening to the queue...
Normally another program processes the queue - not ASP.NET. Either a windows service or an executable that you run under a scheduler (and there's no reason to put it in the root of your website).
If you get a message out of the queue, do you remove it when you pull
it out or do you remove it if denormalization for this session was
successful? If you remove it when you pull it out and something goes
wrong...
For critical work, you perform a transactional read. Items aren't removed from the queue until you commit your read operation, but while the transaction is open, no other process can get the item.
What are best practices to implement this? Links? Resources? Tutorials? Examples?
This tutorial is a good introduction and John Breakwell's blog is excellent and offers a lot of good links (including the ones in his easy-to-find sidebar "MSMQ Documentation").
Recently, the book on threading for Winforms application (Concurrent programming on Windows by Joe Duffy) was released. This book, focused on winforms, is 1000 pages.
What gotchas are there in ASP.NET threading? I'm sure there are plenty of gotchas to be aware of when implementing threading in ASP.NET. What should I be aware of?
Thanks
Since each http request received by IIS is processed separately, on it's own thread anyway, the only issues you should have is if you kick off some long running process from within the scope of a single http request. In that case, I would put such code into a separate referenced dependant assembly, coded like a middle-tier component, with no dependance or coupling to the ASP.Net model at all, and handle whatever concurrency issues arose within that assembly separately, without worrying about the ASP.Net model at all...
Jeff Richter over at Wintellect has a library called PowerThreading. It is very useful if you are developing applications on .NET. => Power Threading Library
Check for his presentations online at various events.
Usually you are encouraged to use the thread pool in .Net because it of the many benefits of having things managed on your behalf.....but NOT in ASP.net.
Since ASP.net is already multi-threaded, it uses the thread pool to serve requests that are mapped to the ASP.net ISAPI filter, and since the thread pool is fixed in size, by using it you are basically taking threads away that are set aside to do the job of handling request.
In small, low-traffic websites, this is not an issue, but in larger, high-traffic websites you end up competing for and consuming threads that the ASP.net process relies on.
If you want to use threading, it is fine to do something like....
Thread thread = new Thread(threadStarter);
thread.IsBackground = true;
thread.Start();
but with a warning: be sure that the IsBackground is set to true because if it isn't the thread exists in the foreground and will likely prevent the IIS worker process from recycling or restarting.
First, are you talking about asynchronous ASP.NET? Or using the ThreadPool/spinning up your own threads?
If you aren't talking about asynchronous ASP.NET, the main question to answer is: what work would you be doing in the other threads and would the work be specific to a request/response cycle, or is it more about processing global tasks in the background?
EDIT
If you need to handle concurrent operations (a better term than multi-threaded IMO) for a given request/response cycle, then use the asynchronous features of ASP.NET. These provide an abstraction over IIS's support for concurrency, allowing the server to process other requests while the current request is waiting for work to complete.
For background processing of global tasks, I would not use ASP.NET at all. You should assume that IIS will recycle your AppPool at a random point in time. You also should not assume that IIS will run your AppPool on any sort of schedule. Any important background processing should be done outside of IIS, either as a scheduled task or a Windows Service. The approach I usually take is to have a Windows Service and a shared work-queue where the web-site can post work items. The queue can be a database table, a reliable message-based queue (MSMQ, etc), files on the file system, etc.
The immediate thing that comes to mind is, why would you "implement threading" in ASP.NET.
You do need to be conscious all the time that ASP.NET is multi-threaded since many requests can be processed simulatenously each in its own thread. So for example use of static fields needs to take threading into account.
However its rare that you would want to spin up a new thread in code yourself.
As far as the usual winforms issues with threading in the UI is concerned these issues are not present in ASP.NET. There is no window based message pump to worry about.
It is possible to create asynchronous pages in ASP.NET. These will perform all steps up to a certain point. These steps will include asynchronously fetching data, for instance. When all the asynchronous tasks have completed, the remainder of the page lifecycle will execute. In the meantime, a worker thread was not tied up waiting for database I/O to complete.
In this model, all extra threads are executing while the request, and the page instance, and all the controls, still exist. You have to be careful when starting your own threads, that, by the time the thread executes, it's possible that the request, page instance, and controls will have been Disposed.
Also, as usual, be certain that multiple threads will actually improve performance. Often, additional threads will make things worse.
The gotchas are pretty much the same as in any multithreaded application.
The classes involved in processing a request (Page, Controls, HttpContext.Current, ...) are specific to that request so don't need any special handling.
Similarly for any classes you instantiate as local variables or fields within these classes, and for access to Session.
But, as usual, you need to synchronize access to shared resources such as:
Static (C#) / Shared(VB.NET) references.
Singletons
External resources such as the file system
... etc...
I've seen threading bugs too often in ASP.NET apps, e.g. a singleton being used by multiple concurrent requests without synchronization, resulting in user A seeing user B's data.