How to render values from Xcom with MySqlToGoogleCloudStorageOperator - airflow

Question

I have the following code:
import_orders_op = MySqlToGoogleCloudStorageOperator(
task_id='import_orders',
mysql_conn_id='mysql_con',
google_cloud_storage_conn_id='gcp_con',
sql='SELECT * FROM orders where orders_id>{0};'.format(LAST_IMPORTED_ORDER_ID),
bucket=GCS_BUCKET_ID,
filename=file_name,
dag=dag)
I want to change the query to:
sql='SELECT * FROM orders where orders_id>{0} and orders_id<{1};'.format(LAST_IMPORTED_ORDER_ID, ...)
The value for {1} is generated with operator in the task before this one. It's being pushed with XCOM.
How can I read the value here?
It should be something with xcom_pull but what is the proper way to do it ? Can I render this sql parameter inside the operator?
I tried to do this:
import_orders_op = MySqlToGoogleCloudStorageOperator(
task_id='import_orders',
mysql_conn_id='mysql_con',
google_cloud_storage_conn_id='gcp_con',
sql='SELECT * FROM orders where orders_id>{0} and orders_id<{1}'.format(LAST_IMPORTED_ORDER_ID,{{ task_instance.xcom_pull(task_ids=['get_max_order_id'], key='result_status') }}),
bucket=GCS_BUCKET_ID,
filename=file_name,
dag=dag)
It gives:
Broken DAG: name 'task_instance' is not defined

Answer 1

In your dag file you aren't actively in a dagrun context with an existing task instance to use as you have.
You can only pull the value when the operator is running, not while you're setting it up (that latter context is executed in a loop by the scheduler and would be run 1000s of times a day, even if the DAG were weekly or was disabled). But what you wrote is actually really close to something that would have worked, so maybe you already considered this contextual point.
Let's write it as a template:
# YOUR EXAMPLE FORMATTED A BIT MORE 80 COLS SYTLE
…
sql='SELECT * FROM orders where orders_id>{0} and orders_id<{1}'.format(
LAST_IMPORTED_ORDER_ID,
{{ task_instance.xcom_pull(
task_ids=['get_max_order_id'], key='result_status') }}),
…
# SHOULD HAVE BEEN AT LEAST: I hope you can spot the difference.
…
sql='SELECT * FROM orders where orders_id>{0} and orders_id<{1}'.format(
LAST_IMPORTED_ORDER_ID,
"{{ task_instance.xcom_pull("
"task_ids=['get_max_order_id'], key='result_status') }}"),
…
# AND COULD HAVE BEEN MORE CLEARLY READABLE AS:
…
sql='''
SELECT *
FROM orders
WHERE orders_id > {{ params.last_imported_id }}
AND orders_id < {{ ti.xcom_pull('get_max_order_id') }}
''',
params={'last_imported_id': LAST_IMPORTED_ORDER_ID},
…
And I know that you're populating LAST_IMPORTED_ORDER_ID from an Airflow variable. You could not do that in the dag file and instead change {{ params.last_imported_id }} to the {{ var.value.last_imported_order_id }} or whatever you named the Airflow variable you were setting.