Airflow HTTP call from unreliable network - airflow

Question

I need to fetch data from REST API via HTTP get on apache airflow (e.g. to https://something.com/api/data).
The data is come in pages with following structure :
{
"meta" : {
"size" : 50,
"currentPage" : 3,
"totalPage" : 10
},
"data" : [
....
]
}
The problem is, the API provider is not reliable. Sometimes we get 504 gateway timeout. So I have to retry the API call, until current page = total page, and retry if we got 504 Gateway timeout. However, the overall retry process must not exceed 15 minutes.
Is there any way I can achieve this using apache airflow?
Thanks

Answer 1

You could use HTTP Operator from HTTP providers package. Check the examples and guide in those links.
If you don't have it already, start by installing the provider package:
pip install apache-airflow-providers-http
Then you could try it sending requests to https://httpbin.org .To do so, create a connection like:
You could create Tasks using the SimpleHttpOperator:
from datetime import datetime, timedelta
from airflow import DAG
from airflow.providers.http.operators.http import SimpleHttpOperator
with DAG(
'example_http_operator',
default_args={
'retries': 1,
'retry_delay': timedelta(minutes=5),
},
start_date=datetime(2021, 10, 9),
) as dag:
task_get_op = SimpleHttpOperator(
task_id='get_op',
method='GET',
endpoint='get',
data={"param1": "value1", "param2": "value2"},
headers={},
)
By default, under the hood, this operator performs a raise_for_status to the obtained response. So, if the response status_code is not in the range of 1xx or 2xx will raise an exception and the Task will be mark as failed. If you want to customize this behaviour you can provide your own response_check as an argument to the SimpleHttpOperator
:param response_check: A check against the 'requests' response object.
The callable takes the response object as the first positional argumentand optionally any number of keyword arguments available in the context dictionary.
It should return True for 'pass' and False otherwise.
:type response_check: A lambda or defined function.
Finally to handle retries on failures as needed, you could use the following parameters avaiblables in any Operator in Airflow (docs):
retries (int) -- the number of retries that should be performed before failing the task
retry_delay (datetime.timedelta) -- delay between retries
retry_exponential_backoff (bool) -- allow progressive longer waits between retries by using exponential backoff algorithm on retry delay (delay will be converted into seconds)
max_retry_delay (datetime.timedelta) -- maximum delay interval between retries
Finally, to try out how everything works together, perform a request to an endpoint which will answer with an specific error status code:
task_get_op = SimpleHttpOperator(
task_id='get_op',
method='GET',
endpoint='status/400', # response stus code will be 400
data={"param1": "value1", "param2": "value2"},
headers={},
)
Let me know if that works for you!