Can Airflow Macros be used with the CloudSqlInstanceExportOperator? - airflow

Question

We're using Airflow to schedule daily database exports using the CloudSqlInstanceExportOperator. This doesn't appear to work with Airflow Macros. We are trying to export 1 day of data using the execution date macro or {{ ds }} in the where clause. It's important to use the macro because we want our DAG to backfill.
The sample code is made of of two parts. First we define the export context:
export_body = {
"exportContext": {
"fileType": "csv",
"uri": "gs://"+GCP_BUCKET+'/'data.csv',
"databases":["database"],
"csvExportOptions": {
"selectQuery": """
select * from table
where datetime BETWEEN "{{ ds }} 00:00:00"
AND "{{ ds }} 23:59:59
"""
}
}
}
Next, pass the export context to the task:
cloudsql_export_task = CloudSqlInstanceExportOperator(
project_id=PROJECT_ID,
body = export_body,
instance='instance',
task_id='cloudsql_export_task',
dag=dag)
The task runs and get's marked as a success, however, the Google Cloud Storage file created has no data in it. When we hard code the date, the query works as expected. As a result, we know the problem is being caused by the macro value not being populated.
Any suggestions would be appreciated. Either how to fix this task or an alternative way to achieve the same objective (note: query is large and uses too much memory for MySqlToGoogleCloudStorageOperator to work)

Answer 1

Make sure the operator includes body in template_fields.
You can also use Jinja templating with nested fields, as long as these
nested fields are marked as templated in the structure they belong to:
fields registered in template_fields property will be submitted to
template substitution
More info about templating: https://airflow.readthedocs.io/en/stable/concepts.html#jinja-templating
You can extend the operator like the following
class CloudSqlInstanceExportTemplatedOperator(CloudSqlInstanceExportOperator):
template_fields = CloudSqlInstanceExportOperator.template_fields + ('body',)

Answer 2

shankshera answer is correct however you are using deprecated operator. In the updated version there is no need for the suggested modification.
The CloudSqlInstanceExportOperator was renamed to CloudSQLExportInstanceOperator and moved to providers.
For Airflow <2.0 you will need to install backport providers :
pip install apache-airflow-backport-providers-google
For Airflow >=2.0 you will need to install providers:
pip install apache-airflow-providers-google
The you can import the operator as:
from airflow.providers.google.cloud.operators.cloud_sql import CloudSQLExportInstanceOperator
Since the operator already has body listed in the templated fields you are good to go.