How to check status of long running http task with airflow? - airflow

Question

My usecase is to control lot of scheduled jobs across microservices using airflow. The solution I am trying is to use airflow as a centralized job scheduler and trigger jobs by making http calls. Some of these jobs will run for long time eg. more than 10min or upto 1 hour.
How can I regularly check the status of these jobs from airflow? What if the remote task has finished but airflow does not know about the job success? Can I publish the event for job completion to kafka and make airflow listen on kafka to get status of job?

Answer 1

There are many ways you could do this with Airflow and your microservices. In general, you will want to use a sensor, that's the appropriate Airflow object for something like this. Start by checking out the BaseSensorOperator and about operators. In Airflow, Sensors are used just like Operators (sensors are operators). So you can create a job like this:
http_post_task -> http_sensor_task -> success_task
Where http_post_task will trigger a job, http_sensor_task will check periodically to see if the job is done (e.g. GET request the microservice and check for 200, maybe?), and success_task will execute after the http_sensor_task is successful.
Your http_sensor_task will need to be your own custom sensor. Here is some sudo code that can help you create this sensor (remember sensors are used like operators). Consider the case where you make a request to the microservice and then make another request to check the status of the job (GET request and check 200), you will extend the BaseSensorOperator kind of like this:
from airflow.operators.sensors import BaseSensorOperator
from airflow.utils.decorators import apply_defaults
from time import sleep
import requests
class HTTPSensorOperator(BaseSensorOperator):
"""
Pokes a URL until it returns 200
"""
ui_color = '#000000'
#apply_defaults
def __init__( self, url, *args, **kwargs):
super(HTTPSensorOperator, self).__init__(*args, **kwargs)
self.url = url
def poke(self, context):
"""
GET request url and return True if response is 200, False otherwise
"""
r = requests.post(self.url)
if r.status_code == 200:
return True
else:
return False
def execute(self, context):
"""
Check the url and wait for it to return 200.
"""
started_at = datetime.utcnow()
while not self.poke(context):
if (datetime.utcnow() - started_at).total_seconds() > self.timeout:
if self.soft_fail:
raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
else:
raise AirflowSkipException("Exporting {0}/{1} took to long.".format(self.project, self.instance))
sleep(self.poke_interval)
self.log.info("Success criteria met. Exiting.")
Then use the operator like:
http_sensor_task = HTTPSensorOperator(
task_id="http_sensor_task",
url="http://localhost/check_job?job_id=1",
timeout=3600, # 1 hour
dag=dag
)
So you'll have to decide how your microservices will communicate with Airflow. Just of the top of my head I'm thinking you'll make 1 request to trigger a job and then make subsequent request (maybe ever 10 seconds) to check on a job. Good luck!