Airflow: Get previous task id in the next task - airflow

Question

I have 2 tasks. In the first, python operator computes something and in the second I want to use the output of the python operator in the Http operator. Here is my code:
source_list = ['account', 'sales']
for source_type in source_list:
t2 = PythonOperator(
task_id='compute_next_gather_time_for_' + source_type,
python_callable=compute_next_gather_time,
provide_context=True,
trigger_rule=TriggerRule.ALL_SUCCESS,
op_args=[source_type],
retries=3
)
t3 = SimpleHttpOperator(
task_id='request_' + source_type + '_report',
method='POST',
http_conn_id='abc',
endpoint=endpoint,
data=json.dumps({
"query": {
"start": "{{ task_instance.xcom_pull(task_ids='prev_task_id') }}",
"stop": str(yesterday),
"fields": [
1
]
}
}),
headers={"Content-Type": "application/json", "Authorization": 'abc'},
response_check=lambda response: True if len(response.json()) == 0 else False,
log_response=True,
retries=3
)
Query: I want to pass previous task id in t3 in its data variable. I am not sure how to do that since t2 task id is not constant. It changes with changing source_type. Evidently, when I tried it did not render it.

Answer 1

I was able to get it by doing this:
next(iter(context['task'].upstream_task_ids))

Answer 2

I haven't used Jinja templating in any of my DAGs before, but I have been faced with similar problems where I was needing to retrieve XCOM values from a particular task that has a dynamically generated task_id.
You could define the task_ids in T3 in the same way you defined the task_id in T2. For example:
source_list = ['account', 'sales']
for source_type in source_list:
task_id='compute_next_gather_time_for_' + source_type
t2 = PythonOperator(
task_id=task_id,
python_callable=compute_next_gather_time,
provide_context=True,
trigger_rule=TriggerRule.ALL_SUCCESS,
op_args=[source_type],
retries=3
)
t3 = SimpleHttpOperator(
task_id='request_' + source_type + '_report',
method='POST',
http_conn_id='abc',
endpoint=endpoint,
data=json.dumps({
"query": {
"start": "{{ task_instance.xcom_pull(task_ids=task_id) }}",
"stop": str(yesterday),
"fields": [
1
]
}
}),
headers={"Content-Type": "application/json", "Authorization": 'abc'},
response_check=lambda response: True if len(response.json()) == 0 else False,
log_response=True,
retries=3
)

Answer 3

To elaborate a bit on #cosbor11's answer.
The approach uses the Airflow task object extracted from the key-word arguments supplied by Airflow during a DAG run. These were once referred to as context and there was an argument to PythonOperator provide_context, but that is deprecated now, I believe. The context is always provided now, making available task, task-instance and other objects and attributes.
So, we can pull things like upstream_task_ids from the task object. Use an iterator or just access it as a list.
def my_python_callable(**context):
upstream_id = next(iter(context['task'].upstream_task_ids))
upstream_ids = context['task'].upstream_task_ids
print(f"got upstream task_id from the task object in the Airflow-provided context: {upstream_id} from a list: {upstream_ids}")
with DAG('silly_hats') as dag:
task0 = DummyOperator(task_id = 'my_spoons_too_big')
task1 = PythonOperator(task_id = 'i_am_a_banana', python_callable = my_python_callable, dag = dag)