In my project I required to write some background jobs for scheduled processing. I did it using quartz scheduler with spring, but quite often it required me to execute the tasks at random without schedule. So later I pulled out the tasks from the quartz and created web endpoints for them(exposed internally).
To perform the regular scheduled based operation of tasks, I created unix cron jobs that hit the web endpoints using curl command.
My question is, why could this approach not work always. Even in case you don't want to expose web endpoints, you can always execute standalone tasks using unix cron. Is there any particular advantage I gain by using quartz scheduler over unix cron jobs?
You may still opt for using Quartz if:
An event needs to be scheduled as part of the activity that happens within the java application itself. For example, user subscribes to a newsletter.
You have a listener object that needs to be notified when the job completes.
You are using JTA transactions in your scheduled job
You want to keep the history of job executions or load job and trigger definitions from a file or a database
You are running on an application server and require load balancing and failover
You are not running on an UNIX / Linux environment (i.e. you wanted platform independence)
Related
I have a web api that's running 24/7 on LAN(due to data security reasons, it's not connected to the internet). I have to call one method inside it each morning around 07:00. I've read that Timers are not reliable. I do not own the server so I don't have access to the Task Scheduler. How can I achieve this?
Do you think I should tell them that I have to install the software? Would they let me use the Task Scheduler? Is there any way to do this without the Task Scheduler?
There are a couple of options that you should look into.
Native .NET Core solution through IHostedService
Hangfire
Personally, I like that you can configure a dashboard and see what happened during the execution of your scheduled tasks.
As suggested by #Lei Yang in the comments, Quartz.NET
Hangfire is still adding in-memory support. Therefore, you're required to have a SQL Server to store all the information about the job (triggers, states, etc.).
This can have a huge impact in your decision. On the other hand, Quartz.NET does support in-memory store type.
I have been using Netflix Conductor as workflow orchestration before and Apache Airflow is new to me. In Conductor, the execution of workflows works in these steps:
Workflow starts via REST API call
Each separate worker (service) polls its own tasks by constantly calling REST API methods of Conductor
After completing or failing these tasks, each worker calls REST API to change status of workflow
Each of these tasks workers are separate services. They are implemented on different programming languages.
I can't seem to find any examples of how to use these concepts on Apache Airflow. Constantly using BashOperator seems to me very bad solution.
Are there any examples that shows how to use workers, some of them are written not on python, to listen and execute its tasks that are defined in DAGs?
I am working on an asp.net mvc-5 web application, deployed under windows 2012 & iis-8. my asp.net mvc have many CRUD operations which are implemented as action methods inside my asp.net mvc.
But my asp.net mvc web application will be doing scheduled long running network scan process, the network scan will mainly do the following steps:-
Get the list of our servers and vms from our database.
Get the scanning username and password for each server and vm from a third party tool, using Rest API.
Call some powershell scripts to retrieve the servers & vms info such as network info, memory, name, etc.
Update our ERP system with the scan info using Rest API.
Now I did a pilot project using the following approach:-
I define a Model method inside my asp.net mvc to do the above 4 steps.
Then I install hangfire tool which will be calling the scan method on predefined scheduler.
Also I create a View inside my asp.net mvc which allow users to set the hangfire schedule settings (this require to do an IIS reset on the host server for hangfire to get the new settings).
Now I run a test scan for a round 150 servers which took around 40 minutes to complete , and it worked well. The only thing I noted is that if I set the schedule to run on non-business hours (where no activity is made on IIS) then hangfire will not be able to call the job, and once the first request is made the missed jobs will run. I overcome this limitation by defining a windows task which calls IIS each 15 minutes, to keep application pool live, and it worked well...
Now the other approach I am reading about is doing my above is as follow:-
Instead of defining Model method inside asp.net mvc to do the scan, I can create a separate console application to do the scan.
Then inside my asp.net mvc to create a view which allow users to create and schdule a task inside the windows tasks scheduler. I can do so by integrating with the windows task scheduler API.
Where this windows task will be calling the console application.
Now I am not sure which approach is better and why ? now generally speaking long running/background jobs should not run under iis.. But at the same time defining these long running processes as console app and calling these apps inside windows task scheduler will create extra dependencies on my web application. And will add extra effort when moving the application from move server to another (for example from test to live)..
Beside this I read that tools such as hangfire, quartz and other are designed to allow running long running tasks inside IIS and they eliminate the need to create console applications and scheduling these console applications using task scheduler ..
So can anyone advice on this?
In my opinion, if it is possible to solve the scheduling problem on the web application side, there is no need to create a scheduler task or a new console application for triggering purposes. The problem you will probably face when using scheduling task in a web application is generally common as you might see is that: The scheduler works like a charm during debugging of the web application, but not being able to trigger after publishing it to IIS. At this point the problem is generally related to IIS rather than the schedulers Quartz.NET, Hangfire, etc. Although there are lots of articles or solution methods posted on the web, unfortunately only some of them is working properly. In addition to this, most of them require lots of configuration settings on the web and machine configuration.
However, there are also some kind of solutions for such a kind of scheduling problem and I believe in that it is worthy to give a try Keep Alive Service For IIS 6.0/7.5. Just install it on the server to which you publish your application and enjoy. Then your published application will be alive after application pool recycling, IIS/Application restarting, etc. That is also used in our MVC application in order to send notification mails weekly and has been worked for months without any problem. Here are the sample code that I use in our MVC application. For more information please visit Scheduled Tasks In ASP.NET With Quartz.Net and Quartz.NET CronTrigger.
*Global.asax:*
protected void Application_Start()
{
JobScheduler.Start();
}
*EmailJob.cs:*
using Quartz;
public class EmailJob : IJob
{
public void Execute(IJobExecutionContext context)
{
SendEmail();
}
}
*JobScheduler.cs:*
using Quartz;
using Quartz.Impl;
public class JobScheduler
{
public static void Start()
{
IScheduler scheduler = StdSchedulerFactory.GetDefaultScheduler();
scheduler.Start();
IJobDetail job = JobBuilder.Create<EmailJob>().Build();
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity("trigger1", "group1")
.StartNow()
.WithSchedule(CronScheduleBuilder
.WeeklyOnDayAndHourAndMinute(DayOfWeek.Monday, 10, 00)
//.WithMisfireHandlingInstructionDoNothing() //Do not fire if the firing is missed
.WithMisfireHandlingInstructionFireAndProceed() //MISFIRE_INSTRUCTION_FIRE_NOW
.InTimeZone(TimeZoneInfo.FindSystemTimeZoneById("GTB Standard Time")) //(GMT+02:00)
)
.Build();
scheduler.ScheduleJob(job, trigger);
}
}
Also I create a View inside my asp.net mvc which allow users to set the hangfire schedule settings (this require to do an IIS reset on the host server for hangfire to get the new settings).
You're resetting your webserver to update a task's schedule? That doesn't sound healthy. What you might do is keep track of what the scheduled time should be, and on execution, check if the current time is within a certain range of the scheduled time (or has already been executed), otherwise abort the job.
The only thing I noted is that if I set the schedule to run on non-business hours (where no activity is made on IIS) then hangfire will not be able to call the job, and once the first request is made the missed jobs will run. I overcome this limitation by defining a windows task which calls IIS each 15 minutes, to keep application pool live, and it worked well...
Hangfire's documentation has a page about running delayed tasks that mentions what you need to change to accomodate this.
Using Windows' Task Scheduler doesn't seem like a good idea; it's not meant for the execution of ad-hoc, short-lived tasks. You probably need elevation to create tasks, and you'd probably need to define another scheduled task to clean up the mountain of tasks that would exist after a few dozen background jobs have been executed.
You're also correct that using Windows' Task Scheduler would make it more difficult to move your application around.
I have a function that takes about 2-3 minutes to do all of the processing it needs
It only needs to do the calculations once daily. The processing is done inside of a background thread that is started inside Application_Start in Global.asax.cs.
I am worried that the application will kill the method while it is running.
You are doing the process in in Application_Start, are you restarting your application daily ?. A better approach would be to create a separate executable for the task, and schedule it on your server using Windows Scheduler.
For the above scenario, either you can use windows service or console application and schedule it when you want it to run.
But doing it at the start of application is never recommended in this scenario.
Is it possible for user who access my page to run a schedule task in asp.net? any reference?
The page can turn on a flag to run a scheduled process, than a windows service can be setup to monitor that flag (stored in DB or file), when its turned on, run the process.
HTH.
ASP.NET itself can't run a scheduled task, but you can use Powershell or Windows Task Scheduler to queue up and schedule tasks to be run at a later time.
If you want a user to be able to alter them it would probably be easiest to have a task set to call a special ASP.NET page which would then run any logic needed to actually do the tasks scheduled by users.
If you have a SQL database backend, you can always create SQL Agent jobs to run and use your ASP.net application to control them. (When to start, frequency, etc)