Airflow - Persist task state/checkpoint information in a long running Task - airflow

Question

I have a task that can run for ~10+ hours, say in a for loop.
I want to store checkpoints after each loop execution, so that if there is some error in the task or if the worker crashes, the retried task can resume from where it left off by retrieving the checkpoint information specific to that task run.
So, the question is, how and where can I store this check point information?
The task logic is below :-
long_running_task:
seqNo = getStoredCheckpointForTask()
do
if(seqNo == null )
seqNo = getFirstSequenceFromSomeSource() //1-2 seconds
doSomething(seqNo); //3-4 seconds
seqNo = getNextSequenceFromSomeSource(oldSeq: seqNo) //1-2 seconds
storeCheckpointForTask (seqNo);
while sequence != null

Answer 1

If your task is running on Airflow workers, you have two options:
You can use an external storage system as checkpoints store (ex S3):
Or using Airflow Metadata (Airflow DB) as checkpoints store by saving an xcom
def my_task_func(**context):
seqNo = context["ti"].xcom_pull(key=f"checkpoint_{context['execution_date']}", default=None)
while True:
if not seqNo:
seqNo = getFirstSequenceFromSomeSource()
doSomething(seqNo)
seqNo = getNextSequenceFromSomeSource(seqNo)
if not seqNo:
break
context["ti"].xcom_push(key=f"checkpoint_{context['execution_date']}", seqNo)
And if your task is running outside Airflow, the first option is still valid, but the second no, and you will have another options depending on how you run your tasks (volumes for docker, PVC for K8S, ...).