Currently I have a task that run every 5 minute.
what I want is to have that task rerun every time it is completed with 1 minute delay.
what I have in mind is to create multiple task, task A and task B. task B will run after task A complete and vice versa. But not sure how to execute that.
I have found a workaround for my situation. what I do is create loop for task A to run followed by task B with delay in between.
Related
I'm still fairly new to using airflow. I have a few dags scheduled to run but they run sequentially. I want them to start running in parallel to each other. So let's say I have a task scheduled to run at 1 pm and another task to run at 2pm. If the first tasks isn't done, by 2pm, I still want the 2pm task to start running and allow the 1pm task to keep going as well.
As explained in https://www.astronomer.io/guides/airflow-executors-explained/, you should use LocalExecutor if you want to run more than a single task at any point-in-time.
I am trying to understand autosys job. Suppose I have Job A that runs every 15 minutes. Suppose for some reason if Job A takes more than 15 minutes, will another instance of it run or it will wait for the job to finish before running another instance?
In my experience, if the previous job run is still running, another instance will not run if the next scheduled time comes. The next time the job runs is when the previous run is finished and the next scheduled time comes.
Another user also experienced this according to this answer.
I did not find any AutoSys documentation that officially confirms what happens in this situation, but I guess the best way to find out is to test it on your AutoSys instance.
I have experienced this first hand and can confirm that there won't be two instances in the mentioned scenario. The job will wait on the previous run to complete and will immediately kick off the next instance if the time condition is met before the previous completes.
But this will be the case only when the job is in running state, if the job is in any other state it will kick off based on the given start_time condition.
I have just started to explore Apache Airflow.
Is there any way to run a job that will look into the running DAGS and move those tasks in those DAGS to new DAG by creating them and adding those tasks in it.
For Example : DAG A has four tasks, 4th one has been waiting from 7 hours to start - Goal is to create new DAG and move that tasks automatically to new DAG.
Scenario : Actually we have around 40 VM, and each job time varies with its own instance. For Example : Task A will take 2 hours today but might take 12 Hours tomorrow in the same DAG. What i need is to move that task to other DAG if the waiting time of any task exceed certain time to run on other VM instantly.
The main benefi is to keep all the task waiting time minimum as possible by building dynamic DAGs
I have three task in one dag.
Task A run first. Task B runs if task A is successful.
I have Task C which has run after Task B but it is not depend up Task B or Task A success or failure.
Task C needs to no matter what happen to task A and B. However, it needs to run after task A and B is completed.
Any idea ?
To have a task run after other tasks are done, but regardless of the outcome of their execution, set the trigger_rule parameter to all_done like so:
my_task = MyOperator(task_id='my_task',
trigger_rule='all_done'
See the trigger rule documentation for more options
My requirement is to create a job in informatica which will run for every 15 min and look for a status column in abc table.If it is “Approved” THEN It will exit and kick off the rest of the jobs.
If the status is not approved it will not do anything and run after 15 min.This process wil continue until we have a approval status.
So, No matter what happens in the above two scenarios,This process will run in every 15 minutes.
I have worked on the same requirement in unix using loops and conditional statments but I am not sure how this can be achieved using informatica.Could you please help me on this.
Regards,
Karthik
I would try adding a scheduler that runs every 15 minutes. The best way that I've found to "loop" sessions in Informatica is:
run the session once, check if it failed using conditional links
if it did fail, run a timer task for an amount of time (a minute, an hour, whatever)
then try to run the same session again by copying and pasting the session up ahead of the timer task, and repeat a few times as necessary.
So if you added a scheduler into the mix, you could set the scheduler to have the workflow run every 15 minutes, and have the timer tasks halt the workflow for 4 or 5 minutes each. Then you could use SESSSTARTTIME function in some pre/post-session task to determine when the scheduler will fire off again and simply abort the workflow before that time.