I'm using SparkSubmitOperator in Airflow to submit job to yarn. When my job is failed, it throw exception in log but Airflow mark this DAG is "success".
How to mark DAG to "failed" in this scenario, because I want alert when DAG "failed" to Telegram.
Please help me... Thank you very much!!
Related
I had created a SCHEDULED DAG in AIRFLOW which used to give email alert when whenever it exceeds a particular time threshold.But now I deleted the DAG py file and also deleted it from the AIRFLOW UI as I don't want it now.Still I am getting email alerts everyday.Can anyone help with this.
Airflow UI comes very handy to check the status of Airflow DAG progress.
Airflow Rest API is another way to check the status to Airflow DAGs but it requires authentication token.
We can get authentication token from Airflow UI, but if Airflow UI is down it would be difficult to get authentication token and Airflow DAG status.
Is there any other way to check/monitor and clear task instance from backend (apart from Rest API and Airflow UI) ?
We have setup status checks on airflow health check endpoint based on this Airflow doc page Checking Airflow Health Status. We have serverless functions running every 5 minutes to check that the status for metabase and scheduler are healthy.
When Airflow is down, you can get alerts routed directly to Slack channel / Email / Opsgenie Alerts through another code block defined in the serverless function.
from airflow.models.dagrun import DagRun
from airflow.utils.state import DagRunState
dag_runs = DagRun.find(dag_id='the_dag_id_you_want_to_check')
last_run = dag_runs[-1]
print('the dag state is -->: ', last_run.state)
We have encountered a scenario recently where someone mistakenly turned off a production dag, and we want to get alert whenever a dag is paused using datadog.
I have checked https://docs.datadoghq.com/integrations/airflow/?tab=host
But have not got any metric for dag to check if it is paused or not.
I can run a custom script in datadog as well.
One of the method is that I exec into postgres pod and get the list of active dags:
select * from dag where is_paused=true;
Or is there any other way I can get the unpaused dag list and also when new dag is added what is the best way to handle it.
I want the alert whenever a unpaused dag is paused.
If you are on Airflow 2 you can use the REST API to query for state of the DAG.
https://airflow.apache.org/docs/apache-airflow/stable/stable-rest-api-ref.html#operation/get_dag
There is "is_paused" field.
And of you are not Airflow 2, you should be. Airflow 1.10 is end-of-life and will not receive any fixes (including critical security fixes) so you should upgrade as soon as you can.
I have a task in Airflow that uses bash operator which is running continuously.
Is there any way to make the task instance as success before its next scheduled run? I know how to mark it as success in api.
Thank you for your replies
If you just want to call the api automatically to mark the previous dagrun as success, you can use SimpleHttpOperator as the first task of your dag. This operator might call airflow REST API to request to mark the previous dagrun as success.
https://airflow.apache.org/docs/apache-airflow-providers-http/stable/operators.html#simplehttpoperator
When I run manually, sometimes DAG is not getting started. It is showing tasks's staus as 'no status' only. Then I am manually making each task as 'success' and restarting the airflow scheduler, webserver.
Can you please suggest/help me to get this solved?.
Thanks,
Samanth