These two may look like they have no correlation but bear with me!
In a previous version of the software I develop/maintain there was a web app sitting on top of a web service. There was a scheduled task that run every hour called one of the web methods to carry out some tasks.
In the new architecture we now have a web application project with two class libraries for the Business Layer and Resource Access Layer.
However I still need the same functionality in this version, and I am currently trying to design a suitable solution.
I was thinking it may be an idea to have the hourly task running on a separate thread of the web application that sleeps for an hour, wakes up and carries out the task, or would it be easier to expose some web methods in a similar way to the old application.
If anyone also has any good examples of ASP.NET threading I would appreciate having a look at them
The problem you could have is that the thread will be running in application pool process which may for various reasons be closed down. For example overnight with no activity the process could legitimately shutdown. This fine, any new request will simply spin a new worker process. However if you require something to run every hour this won't be happening if the pool is shutdown.
Additional the application pool may be a web garden where there are multiple processes. You then need to consider how you ensure you don't have multiple versions of this task.
Hence it would be better to continue with the scheduled task approach with posts a request to the web server to intiate the activity.
Related
client wants an asp.net page that has a button to fire off a database update from an external source with hundreds of records. This process takes a long time. He also wants status update as the process runs, like "processing 10 out of 1000 records". In reading various articles, I'm thinking of putting the database update code in a windows service. I've never worked with windows services before and I can't find many tutorials on how to fire off a windows service and poll it from an asp.net page. My questions are is this the best way to handle this process? And, does anyone have any examples on how they've accomplished this?
There are a few ways to approach this.
You're right in that executing a long-running task within the Web's worker process doesn't usually end well: it ties up resources, the app pool can get recycled, etc. In most of my projects of any complexity, I usually end up with 4 pieces: the database, a DLL with my model, a "Worker" that is a Windows service, and an ASP.NET Web site.
The "Worker" is a Windows service that is always running and uses Quartz.net to execute scheduled tasks using the same model that the Web site uses. These can be all sorts of periodic tasks that seem to crop up when maintaining a Web site of any complexity: VacuumExpiredPickTicketsJob, BackupAndFtpDatabaseJob, SendBackorderReminderEmailsJob, etc.
Writing a Windows service is not difficult in C# (there is a built-in template in Visual Studio, but you pretty much inherit from ServiceBase and you're off to the races), and libraries like TopShelf make it even easier to deploy them.
What is left is triggering the update from the Web site and communicating the results back to the user. This can be as simple or as complicated as you want it to be. If this is something that has to scale up to lots of users, you might use something like MSMQ to queue up update commands to the Windows service, and the Windows service would respond to that queue. I get the impression that that is probably overkill here.
For a handful of users, you could override your service's OnCustomCommand(int command) method to be the trigger. Your Web site would then use ExecuteCommand() of the ServiceController class to get the process started. Your Web site and service would agree on the parameter value that means "do that update thing," let's say 142 (since it has to be a number between 128 and 255 for reasons of history).
As for communicating progress back to the client, it's probably easiest to just have the Web page use a timer and an AJAX call to poll for updated progress data. You can get fancy with new stuff like WebSockets (bleeding edge stuff as I write this) and long polling, but regular polling will simply work for something that doesn't need to scale.
Hope this helps!
In addition to Nicholas' thorough answer, another option is to deploy your back end processes as command line scripts, and schedule them to run through Window's built in task scheduler, which has improved quite a bit in Windows Server 2008+. Or you can use any other host of task scheduler applications.
I find the command line approach to be easier for MIS staff to understand and configure, and to migrate to new servers, versus standard Windows services.
I’m working on a business problem which has to import files which has 1000s of records. Each record has to be registered in a Workflow as individual record which has to go through its own workflow.
WF4 Corporate Purchase Process example has a good solution, as in the first step it create bookmarks for all the required record ids. So the workflow can be resumed with rest of the actions for each individual record/id.
I would like to know how to implement same thing using Workflow services as I could get the benefits of AppFabric for my workflows.
Is there any other solutions to handle batch of records/ids? Otherwise workflow service has to be called 1000s of times just to register every record in a workflow instance which is a not a good solution.
I would like to know how to implement same thing using Workflow services as I could get the benefits of AppFabric for my workflows.
This is pretty straight forward. You're going to have one workflow that reads the file and loops through the results using the looping activities that exist. Then, inside the loop you'll be starting up the workflow that each record needs (the "Service") by calling the endpoint with a Send activity.
Now, as for the workflow that is the Service, you're going to have a Receive activity at the top of the workflow that also has CanCreateInstance set the true. The everything after the Receive is no different than any other workflow. You may consider having a Send activity right after the Receive just to let the caller know that the Service has been started. But that's not a requirement -- the Receive will be required because it forces WF to build the workflow to use the WorkflowServiceHost.
Is there any other solutions to handle batch of records/ids? Otherwise workflow service has to be called 1000s of times just to register every record in a workflow instance which is a not a good solution.
Are you indicating that a for a web server to receive 1000's of requests is not a good solution? Consider the fact that an IIS server can handle roughly 25-50 requests, per instant in time, per core. Now consider the fact that you're loop that's loading the workflows isn't going to average more than maybe 5 in that instant of time but probably more like 1 or 2.
I don't think the web server is going to be your issue. I've started up literally 10,000's of workflows on a server via a loop just like the one you're going to build and it didn't break a sweat.
One way would be to use WCF's MSMQ binding to launch your workflows. Requests can come in normally through HTTP, and WCF would route them to MSMQ and process the load. You can throttle how many workflow instances are used through the MSMQ binding + IIS settings.
Download this word document that describes setting up a workflow application with WCF and MSMQ: http://www.microsoft.com/en-us/download/details.aspx?id=21245
In the spirit of the doing the simplest thing that could work, you can bring the subworkflow in as an activity to the main workflow and use a parallel for each to execute the branch for each input from your file. No extra invoking is required and the tooling supports this out of the box because all workflows are activities. Hosting the main process in a service so you can avoid contention with the rest of your IIS users, real people that they may are, might be a good idea.
I do agree that calling IIS or a WCF service 1000's of times is not a problem though, unless you want to do it in a few seconds!
It is important to remember that one of the good things about workflow is that it has fairly low overhead (compared to other workflow products) so you should be more concerned about what your workflow does than just the idea of launching lots of instances. The idea of batches like your example is very common.
I have a long running process in my MVC application(C#).
Its building data for many reports.
Some of the clients could take several minutes or longer to calculate. Is running the process in a separate thread the best option?
Is there another way to allow the process to run, while allowing the user to still use the rest of the site?
If threading is the best solution, any good sites or stackoverflow threads to look at on how to do this?
When I've had cases like those, I usually would build a service to asynchronously process requests, and return a handle that I could use to check on its status in a database. IMHO, splitting it off as a thread in the web application seems like you'd be trying to shove a square peg into a round hole.
I've used two methods to solve this. If the work is guaranteed to not be TOO long running, I've kicked off a thread to do the work and return immediately to the user. When we couldn't make that guarantee, we used a queue (we happened to use MSMQ) for executing long running tasks. This processing was done on a different server apart from IIS. A benefit of this is that we built in a wait and retry on failure mechanism. So besides it handling long running tasks, we also used it for anything that might fail in a way that was inconvenient to handle in our MVC app. The main example of this is sending an email. Rather than do that in the MVC app we would just toss an email task on the queue. We used the Command Pattern for the task objects placed on the queue. Once we had that mechanism in place, we stopped using the technique of spawning a thread from our MVC code.
My web site stores invoices.
I would like to process them every night at midnight.
I would also like to send out notices nightly at midnight.
But the web is stateless.
I do not have my own server.
Is there a way?
No, you need a Windows Service on a dedicated box, however the box can be virtual and in the cloud.
You do not own your own server, but your website must be hosted somewhere. You may want to see if you can install windows services on to that server. If you can, I would strongly recommend using a service to run any automated tasks. It is generally frowned upon to use IIS to run automated tasks simply because the process shuts down after periods of inactivity... and if IIS isn't running, neither will your automated task.
A similar question was asked a while back and I think the answers are still very relevant to your problem: Best way to run scheduled tasks
And even further, after google searching I found even more info which you might find helpful:
https://blog.stackoverflow.com/2008/07/easy-background-tasks-in-aspnet/
http://www.codeproject.com/KB/aspnet/ASPNETService.aspx
If you absolutely cannot install a service, you could try something like creating a new thread on Application_Start() that checks the time and decides whether or not to run. Then, to keep IIS running during periods of inactivity, you could possibly try spawning a process or something every few minutes to send a request to your server and thus keep it from shutting down.
We have a long running data transfer process that is just an asp.net page that is called and run. It can take up to a couple hours to complete. It seems to work all right but I was just wondering what are some of the more popular ways to handle a long process like this. Do you create an application and run it through windows scheduler, or a web service or custom handler?
In a project for long running tasks in web-application, i made a windows service.
whenever the user has to do the time-consuming task, the IIS would give the task to the service which will return a token(a temporary name for the task) and in the background the service would do the task. At anytime, the user would see the status of his/her task which would be either pending in queue, processing, or completed. The service would do a fixed number of jobs in parallel, and would keep a queue for the next-incoming tasks.
A windows service is the typical solution. You do not want to use a web service or a custom handler as both of those will lie prey to the app pool recycling, which will kill your process.
Windows Workflow Foundation
What I find the most appealing about WF, is that workflows can be designed without much complexity to be persisted in SQL Server, so that if the server reboots in the middle of a process, the workflow can resume.
I use two types of processes depending on the needs of my BAs. For transfer processes that are run on demand and can be scheduled regularly, I typically write a WinForms (this is a personal preference) application that accepts command line parameters so I can schedule the job with params or run it on demand through an interactive window. I've written enough of them over the last few years that I have my own basic generic shell that I use to create new applications of this nature. For processes that must detect events (files appearing in folders, receiving CyberMation calls, or detecting SNMP traps), I prefer to use Windows Services so that they are always available. It's a little trickier simply because you have to be much more cautious of memory usage, leaks, recycling, security, etc. For me, the windows application tends to run faster on long duration jobs than they do when through an IIS process. I don't know if this is because it's attached to an IIS thread or if its memory/security is more limited. I've never investigated it.
I do know that .Net applications provide a lot of flexibility and management over resources, and with some standards and practice, they can be banged out fairly quickly and produce very positive results.