In an ASP.NET MVC4 project, I want to update some data in sql server(2012) on 00:00 everyday.
Maybe I have three choice:
1. writing an windows service which running on the server and execute database updating.
2. writing an sql server Stored Procedure which execute on 00:00 everyday.
3. use Third-party tools，like Quartz.Net, fluent.
Which one is the best choice? Why?
If you're comfortable with T-SQL and the task can be completely entirely within the database, then it makes sense to implement it as a stored proc and schedule it using SQL Server Agent. This reduces the number of external dependencies as everything happens inside SQL Server and there are less points of failure (e.g. it will still run if IIS or your web solution is down).
If your update needs to interact with other resources, such as importing a text file from a known location, etc., then you might also consider implementing it in SSIS (SQL Server Integration Services). This has the advantage of still being SQL hosted, while giving you access to additional functionality not easily achieved with a stored proc.
I would only implement as a batch process in .NET if I felt that the functionality required would be difficult or awkward to implement in a proc or SSIS package. This is especially relevant since SQL Server 2012 allows you to build an SSIS package using .NET type code, but it "lives" in the SSIS package that is registered on the SQL Server.
I wouldn't implement it within your ASP.NET solution at all unless it actually needs a web based user interface for some reason. The fact that you want to execute this process at precisely the same time every night tells me that this does not require human interaction.
Where a process can be fully automated, avoid putting a user interface or anything which can potentially hang or become a single point of failure while waiting for some kind of UI input. Remember, a presentation layer is for human interaction - consider if you need one. Better to implement it as a batch of some sort, which also makes it easier to execute via automation. There are exceptions to this rule, e.g. if you wanted to implement your update as a web API with a REST interface or the like, but as a general rule it holds true.
As a side note, if your aim is to run your process overnight, consider scheduling it in the early hours of the morning (between 3 and 4 am) rather than midnight as this is generally when most people are asleep and your update is least likely to impact the availability of your app and its database, or if your update is long running, run into an edge case like a daylight savings change or conflict with other overnight processes.
Check out Hangfire. Scheduled tasks in ASP.NET. Super easy and reliable.
http://hangfire.io/
We just started using it and I like it.
I used all three ways you write, all have their pros and cons. But I suggest you to use Quartz.Net. It is very easy to impliment and very easy to use.
You can see here a wonderful article by mike on Scheduled tasks in ASP.NET
Related
Few questions:
1. Is SQL server installation needed to run Windows Work Flow?
2. If yes, where does work flow stores (persists) data for a long running process
3. I see that some files are created in .\windows\Microsft.NET\Framework\v4.0\SQL\en\ (some sql scripts to create persistense points)
4. Do we need to run these scripts to manually create database?
5. Can we persist data on file system instead? so that we don't need to install SQL Server?
Thanks
I see one supposed answer already, but "read the docs" answers really aren't good answers, especially in an area so poorly documented as WF, so in case anyone else stumbles across this thread:
(1) SQL Server doesn't have to be installed just to use workflows, but if you want persistence for long running workflows, (2) SQL Server is your easiest way to get it.
(3) and (4) You can let AppFabric do most of the heavy lifting in setting up the persistence database for you.
(5) you could persist on a file system instead of SQL Server but IMHO, from what I've seen in my short time with WF and persistence so far, you'd be crazy to try to implement your own persistence provider like that, especially when just starting out. You can use SQL Server Express to get started. Why reinvent the wheel?
I am writing a web application in ASP.NET 3.5 that takes care of some basic data entry scenarios. There is also a component to the application that needs to continuously poll some data and perform actions based on business logic.
What is the best way to implement the "polling" component? It needs to run and check the data every couple of minutes or so.
I have seen a couple of different options in the past:
The web application starts a background thread that will always run while the web application does. (The implementation I saw started the thread in the Application_Start event.)
Create a windows service that is always running
What are the benefits to either of these options? Are there additional options?
I am leaning toward a windows service because it is separated and can run on a different server (more scalable) as well as there is more control over when it is started/stopped, etc. However, I feel like the compactness of having the "background" logic running in the process of the web application might make the entire solution more understandable.
I'd go for the separate Windows service primarily for the reasons you give:
You can run it on a different server if necessary.
You can start and stop it independently of the web site.
I'd also add that it could well have some impact on the performance of the web site itself - something you want to avoid.
The buzz-word here is "separation of concerns". The web site is concerned with presenting the data to the user, the service with checking the integrity of the data.
You can also update the web site and service independently of each other should you need to.
I was going to suggest that you look at a scheduled task and let Windows control when the process runs, but I re-read your question and noted that you wanted the checks to run every couple of minutes. The overhead of starting the process might be too great in this case - though some experimentation would probably prove this one way or the other.
If you use a scheduled task there's also the possibility that you could start the next check before the current one has finished - something you can code for if you're in complete control.
Why not just use a console app that has no ui? Can do all that the windows service can and is much easier to debug and maintain. I would not do a windows service unless you absolutely have to.
You might find that the SQL Server job scheduler sufficient for what you want.
Console application does not do well in this case. I wrote a TAPI application which has to stay in the background and intercept incoming calls. But it did it only once because the tapi manager got GCed and was never available for the second incoming call.
I have a website that's running on a Windows server and I'd like to add some scheduled background tasks that perform various duties. For example, the client would like users to receive emails that summarize recent activity on the site.
If sending out emails was the only task that needed to be performed, I would probably just set up a scheduled task that ran a script to send out those emails. However, for this particular site, the client would like a variety of different scheduled tasks to take place, some of them always running and some of them only running if certain conditions are met. Right now, they've given me an initial set of things they'd like to see implemented, but I know that in the future there will be more.
What I am wondering is if there's a simple solution for Windows that would allow me to define the tasks that needed to be run and then have one scheduled task that ran daily and executed each of the scheduled tasks that had been defined. Is a batch file the easiest way to do this, or is there some other solution that I could use?
To keep life simple, I would avoid building one big monolithic exe and break the work to do into individual tasks and have a Windows scheduled task for each one. That way you can maintain the codebase more easily and change functionality at a more granular level.
You could, later down the line, build a windows service that dynamically loads plugins for each different task based on a schedule. This may be more re-usable for future projects.
But to be honest if you're on a deadline I'd apply the KISS principle and go with a scheduled task per task.
I would go with a Windows Service right out of the gates. This is going to be the most extensible method for your requirements, creating the service isn't going to add much to your development time, and it will probably save you time not too far down the road.
We use Windows Scheduler Service which launches small console application that just passes parameters to the Web Service.
For example, if user have scheduled reports #388 and #88, scheduled task is created with command line looking like this:
c:\launcher\app.exe report:388 report:88
When scheduler fires, this app just executes web method on web service, for example, InternalService.SendReport(int id).
Usually you already have all required business logic available in your Web application. This approach allows to use it with minimal efforts, so there is no need to create any complex .exe or windows service with pluggable modules, etc.
The problem with doing the operations from the scheduled EXE, rather than from inside a web page, is that the operations may benefit from, or even outright require, resources that the web page would have -- IIS cache and an ORM cache are two things that come to mind. In the case of ORM, making database changes outside the web app context may even be fatal. My preference is to schedule curl.exe to request the web page from localhost.
Use the Windows Scheduled Tasks or create a Windows Service that does the scheduling itself.
I've been using WWF for a while as part of an internal call center application (ASP.NET), and while learning it was a good practice in understanding how a state machine based workflow system should work, I am definitely not in love with WWF itself. In my opinion it is:
Overly complex, especially for use within web apps (all that threaded runtime stuff)
Immature (ever worked with that horrible designer?)
Anemic in its current feature set
Does anyone have a suggestion for a better .NET based workflow framework? Specifically, I am looking for the following features:
State machine based (mapping states to available actions)
A focus on user permissions (controlling who has access to what actions)
The ability to run workflows as timed background tasks (for example, to send out reminders for items that have been sitting in a certain state for x days)
That's really all I need. I don't need to be able to "drag and drop" any activities or visually design the flow. I am perfectly comfortable writing actual code once a particular action is triggered.
You could try Simple State Machine. You would have to implement access control and background timers yourself, but that shouldn't be a big deal. SSM was also built out of frustration with WF. There are some other state machine implementations on Codeplex as well. If one of them doesn't fit he bill out of the box, they are open source and should get you close enough.
I wholeheartedly agree with you about state machines in WF - they aren't testable, are too complicated, the threading model is peculiar and hard to follow, and I'm not sure a visual designer could have been more poorly conceived for designing state machines graphically. I think this may be because the state machine concept feels tacked onto the WF runtime, which was designed for sequential state machines, something WF does a much better job with, in my opinion. The problem is that state machines are really not the same animal as a sequential work flow, and should have been given a first class implementation of their own, because the warping of WF to make it seem to support them turned out to be more or less unsupportable, if not actually unusable.
I would stay away from Drools.Net since it's last SVN commit was in September 2007. Looks nice but it seems a bit too risky to make such a big library part of your project when you know it doesn't get any attention anymore.
Try Drools.NET
Have a look at Workflow Engine. It is a lightweight workflow framework for .NET and Java solutions. It has an HTML5 visual designer, version control, a decent UI and supports a wide range of databases.
Do you have the option to consider BizTalk Server?
I quite enjoyed working with Oracle BPEL Process Manager. It's part of JDeveloper.
http://www.oracle.com/technology/bpel/index.html
http://gemsres.com/story/dec06/313602/jellema-fig1.jpg
You may want to take a look at Jazz - http://jazz.codeplex.com/
Try WF4.5. It was completely redesigned since .NET4.0.
First of all you should look for a engine supporting BPMN. BPMN is a standard in Workflow and Process management and well supported from a lot of projects.
Second you should think about the requirements to thus an engine.
When you look for a BPMN Engine, there are two different approaches:
Task-Orientated
These engines (e.g. JBoss BPM - jbpm) are designed to process an input data by a well defined process model. Each task in the model gives the control to a piece of code - either a standard or an individual implementation. The process ends when the process-token reaches the end of the process model (End-Event). This kind of processing takes milliseconds. The engine can be used for batch jobs or processing data with a complex process orientated flow.
Event-Driven
Human-Centric workflow engines are event driven (e.g. Imixs-Workflow). This is a kind of state machine but offers typically much more functionality. You can start a new processinstance by assigning your business object with the initial task (defined by the start event). Than the workflow engine allows you to trigger events assigned to each task, defined in your model. Each event (Intermediate CatchEvent) triggers the workflow engine to transfer the running processinstance to the next task (state). Until no new event is triggered, the processinstance 'waits' in the current task (state). An approval process is an typical example for this kind of human-centric workflow.
You can find a list of engines here.
Jeff has previously blogged about using the cache to perform "out of band" processing on his websites, however I was wondering what other techniques people are using to process these sorts of tasks?
Years ago, I saw Rob Howard describe a way to use an HttpModule to process tasks in the background. It doesn't seem as slick as using the Cache, but it might be better for certain circumstances.
This blog post has the details, and there are many others that capture the same information if you look around.
Windows Service
You may want to look at how DotNetNuke does it. I know it is written in VB.NET, but I retrofitted the code into C#. I was perusing the source and noticed they had a feature in their admin area to setup scheduled tasks. These tasks get setup thru the admin interface and stored in the database. When the site starts, thru the Global.asax file, they either created another thread to run this service that then runs the scheduled tasks at their scheduled time. I can't remember the exact logic, it's been a while, but it is definitely a good resource on how other people have done out of band processes for Asp.Net applications. This technique still keeps the logic within the Asp.Net application, but it runs out of band in my opinion.
if it's primarily data processing tasks and you're using MSSQL, how about scheduled SSIS tasks?
Scheduled tasks using http://www.codeproject.com/KB/cs/tsnewlib.aspx or schtasks.exe.
Quartz.NET
MSMQ
SQL Server jobs
Windows service
System.Threading.Timer or System.Timers.Timer
System.ComponentModel.BackgroundWorker
Asynchronous calls and callbacks
Scheduled tasks, or cron jobs.
The problem with scheduled tasks or cron jobs is that they don't share memory space with the web server. You could set up a scheduled task that requested pages from the web server, but that might create problems with long running tasks. It would be nice to have some low priority threads running on the actual ASP.Net application stack to do simple utility tasks like cleaning up caches, monitoring resources, and just to deal with general housekeeping.
Simple queue files along with a separate agent. For each type of out of band process write a separate agent .exe which watches a directory for queue files that include whatever data is needed to perform the specified process.
This may seem dirty but in the real world I find it gives a lot of flexibility, you aren't doing a lot of processing in ASP.net process space and you could easily adapt this style to farm processing out to cheap Linux servers running the agent process on Mono for when you start needing more RAM/CPU/disk.
If you are most comfortable with asp.net pages you can write a small app to handle your job and then "ping" the app with an outside service that monitors your web site. This will keep the app alive.