I have a task in Airflow that uses bash operator which is running continuously.
Is there any way to make the task instance as success before its next scheduled run? I know how to mark it as success in api.
Thank you for your replies
If you just want to call the api automatically to mark the previous dagrun as success, you can use SimpleHttpOperator as the first task of your dag. This operator might call airflow REST API to request to mark the previous dagrun as success.
https://airflow.apache.org/docs/apache-airflow-providers-http/stable/operators.html#simplehttpoperator
Related
I followed the docs and created slack function:
It does work and I get notifications in channel, but get the name of the task and link to log are to another task, and not to the one that gets failed.
It gets the context of the upstream failed task, but not the failed task itself:
I tried with different operators and hooks, but get the same result.
If anyone could help, I would really appreciate it.
Thank you!
The goal of the on_failure_callback argument on the Dag level, is running this callback once when the DagRun fails, so we provide the context of the DagRun which is identical between the task instances, for that we don't care which task instance context we provide (I think we provide the context of the last defined task in the dag regardless its state).
If you want to run the callback on each failed ti, you can remove the on_failure_callback argument from the dag and add it to the default args: default_args=dict(on_failure_callback=task_fail_slack_alert).
we have created a task for sensor operation, but the task name will be dynamically updated. i.e., f"{table_name}_s3_exists". We have a scenario where we have to check a table's location twice, but if the task is still present, we don't have to create the sensor. Is there a way to find whether the task exists or not within the DAG during building ?
The CLI command
airflow tasks list [-h] [-S SUBDIR] [-t] [-v] dag_id
will give you list of all the dags.
https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#list_repeat6
You can also use the REST API to get the same info:
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_tasks
You could try the get_tasks endpoint in the Airflow REST API. The endpoint returns a lot of information for tasks in a given DAG.
What is the difference between max_retries and status_retries when using Airflow BatchOperator? I need to ensure that if a batch job fails, airflow will mark the task (that triggered the batch job) as a failure. Currently, my batch job fails, but the airflow BatchOperator task is marked as a success. I believe that using one or both of these parameters will solve my problem, but I'm not sure what the difference really is between them. Thanks!!
I have a running dag, Can I stop it using REST API ?
There is a method to delete a running dag:
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/delete_dag_run。
But There is not REST API how to stop a running dag?
Thanks!
You can set the running tasks to failed with Set a state of task instances note that this will stop the task from running and set the task status to failed but it will not invoke the on_failure_callable if you set one on your tasks nor it will invoke the on_kill method of the operator (if there is one).
If you want the DAG to not schedule any more tasks you can set it to Pause with Update a DAG
We are currently evaluating airflow for a project. I was wondering if there is a way to stop/start individual dagruns while running a DAG multiple times in parallel. Pause/unpause on dag_id seems to pause/unpause all the dagruns under a dag. Instead we want to pause individual dagruns (or tasks within them). Let me know if this is achievable in airflow.
If its not possible, here are other alternatives I am thinking of, let me know your opinion on these
Change task state. – Change all tasks under a dagrun to Mark Failed or Success. That way that particular dagrun is stopped on its tracks without affecting other dagruns.
Airflow sensor to pull this information from s3 or http or sql or somewhere to pause current dagrun. And have a task to check on s3 everytime if this dagrun needs to be stopped (not other dagruns).
subdags. - Can we pause/unpause subdags. That way for each parallel user's request we want to do we issue a subdag and we can pause userAs subdag without impacting other user’s subdags.
There's nothing "baked" into Airflow to support this but you could (ab)use the state of the DagRun by changing it to "failed" to pause and then back to "running" to resume; you won't be able to blanket unpause but for testing it should be workable.