When triggering a dag in airflow, there is a window, with which I am able to parameters to the dag in a json format. This looks like the following:
This json is always empty and I do have to know which parameters I can pass to the dag. Instead I would like to be able to prefill this json, so that when another user tries to trigger the dag he can simply change to values of the json, instead of having to look at the dags code first.
Is there any way to do this in the current version (2.0.0) of airflow?
On Airflow 2.1.0 it is possible to set default arguments as follow:
dag = DAG(dag_id="my_dag",
schedule_interval=None,
default_args={'retries': 3, 'retry_delay': timedelta(seconds=20)},
catchup=False,
tags=['maintenance'],
params={"description": ""} #Set parameters as a dictionary
)
In the thrigger UI it looks like this:
When writing my feature request i actually found a pull request, which is already merged and seems to exactly do as described:
https://github.com/apache/airflow/pull/10839
An improvement of the this feature also seems to be planned. See:
https://github.com/apache/airflow/issues/11054
No, it is currently not supported -- at least for Airflow 2.0.0
Related
I am trying to run a batch predict job task using the Vertex AI Airflow Operator CreateBatchPredictionJobOperator. This requires pulling a model id from XCom which was pushed by a previous custom container training job. However, CreateBatchPredictionJobOperator doesn't seem to render Xcom pulls as expected.
I am running Airflow 2.3.0 on my local machine.
My code looks something like this:
batch_job_task = CreateBatchPredictionJobOperator(
gcp_conn_id="gcp_connection",
task_id="batch_job_task",
job_display_name=JOB_DISPLAY_NAME,
model_name="{{ ti.xcom_pull(key='model_conf')['model_id'] }}",
predictions_format="bigquery",
bigquery_source=BIGQUERY_SOURCE,
region=REGION,
project_id=PROJECT_ID,
machine_type="n1-standard-2",
bigquery_destination_prefix=BIGQUERY_DESTINATION_PREFIX,
This results in a value error when the task runs:
ValueError: Resource {{ ti.xcom_pull(key='model_conf')['model_id'] }} is not a valid resource id.
The expected behaviour would be to pull that variable by key and render it as a string.
I can also confirm that I am able to see the model id (and other info) in XCom by navigating there in the UI. I attempted using the same syntax with xcom_pull with a PythonOperator and it works.
def print_xcom_value(value):
print("VALUE:", value)
print_xcom_value_by_key = PythonOperator(
task_id="print_xcom_value_by_key", python_callable=print_xcom_value,
op_kwargs={"value": "{{ ti.xcom_pull(key='model_conf')['model_id'] }}" },
provide_context=True,
)
> [2022-12-15, 13:11:19 UTC] {logging_mixin.py:115} INFO - VALUE: 3673414612827265024
CreateBatchPredictionJobOperator does not accept provide_context as a variable. I assumed it would render xcom pulls by default since xcom pulls are used in the CreateBatchPredictionJobOperator in an example on the Airflow docs (link here).
Is there any way I can provide context to this Vertex AI Operator to pull from the XCom storage?
Is something wrong with the syntax that I am not seeing? Anything I a misunderstanding in the docs?
UPDATE:
One thing that confuses me is that model_name is a templated field according to the Airflow docs (link here) but the field is not rendering the XCom template.
Did you set render_template_as_native_obj=True in your DAG definition?
What version of apache-airflow-providers-google do you use?
====
From OP:
Your answer was a step in the right direction.
The solution was to upgrade apache-airflow-providers-google to the latest version (at the moment, this is 8.6.0). I'm not able to pinpoint exactly where in the changelog this fix is mentioned.
Setting render_template_as_native_obj=True was not useful for this issue since it rendered the id pulled from XCom as an int, and I found no proper way to convert it to str when passed into CreateBatchPredictionJobOperator in the model_name arg.
We're using Airflow to schedule daily database exports using the CloudSqlInstanceExportOperator. This doesn't appear to work with Airflow Macros. We are trying to export 1 day of data using the execution date macro or {{ ds }} in the where clause. It's important to use the macro because we want our DAG to backfill.
The sample code is made of of two parts. First we define the export context:
export_body = {
"exportContext": {
"fileType": "csv",
"uri": "gs://"+GCP_BUCKET+'/'data.csv',
"databases":["database"],
"csvExportOptions": {
"selectQuery": """
select * from table
where datetime BETWEEN "{{ ds }} 00:00:00"
AND "{{ ds }} 23:59:59
"""
}
}
}
Next, pass the export context to the task:
cloudsql_export_task = CloudSqlInstanceExportOperator(
project_id=PROJECT_ID,
body = export_body,
instance='instance',
task_id='cloudsql_export_task',
dag=dag)
The task runs and get's marked as a success, however, the Google Cloud Storage file created has no data in it. When we hard code the date, the query works as expected. As a result, we know the problem is being caused by the macro value not being populated.
Any suggestions would be appreciated. Either how to fix this task or an alternative way to achieve the same objective (note: query is large and uses too much memory for MySqlToGoogleCloudStorageOperator to work)
Make sure the operator includes body in template_fields.
You can also use Jinja templating with nested fields, as long as these
nested fields are marked as templated in the structure they belong to:
fields registered in template_fields property will be submitted to
template substitution
More info about templating: https://airflow.readthedocs.io/en/stable/concepts.html#jinja-templating
You can extend the operator like the following
class CloudSqlInstanceExportTemplatedOperator(CloudSqlInstanceExportOperator):
template_fields = CloudSqlInstanceExportOperator.template_fields + ('body',)
shankshera answer is correct however you are using deprecated operator. In the updated version there is no need for the suggested modification.
The CloudSqlInstanceExportOperator was renamed to CloudSQLExportInstanceOperator and moved to providers.
For Airflow <2.0 you will need to install backport providers :
pip install apache-airflow-backport-providers-google
For Airflow >=2.0 you will need to install providers:
pip install apache-airflow-providers-google
The you can import the operator as:
from airflow.providers.google.cloud.operators.cloud_sql import CloudSQLExportInstanceOperator
Since the operator already has body listed in the templated fields you are good to go.
I have a DAG with one DataflowTemplateOperator that can deal with different json files. When I trigger the dag I pass some parameters via {{dag_run.conf['param1']}} and works fine.
The issue I have is trying to rename the task_id based on param1.
i.e. task_id="df_operator_read_object_json_file_{{dag_run.conf['param1']}}",
it complains about only alphanumeric characters
or
task_id="df_operator_read_object_json_file_{}".format(dag_run.conf['param1']),
it does not recognise dag_run plus the alpha issue.
The whole idea behind this is that when I see at the dataflow jobs console and job has failed I know who the offender is based on param1. Dataflow Job names are based on task_id like this:
df-operator-read-object-json-file-8b9eecec
and what I need is this:
df-operator-read-object-param1-json-file-8b9eecec
Any ideas if this is possible?
There is no need to generate new operator per file.
DataflowTemplatedJobStartOperator has job_name parameter which is also templated so can be used with Jinja.
I didn't test it but this should work:
from airflow.providers.google.cloud.operators.dataflow import DataflowTemplatedJobStartOperator
op = DataflowTemplatedJobStartOperator(
task_id="df_operator_read_object_json_file",
job_name= "df_operator_read_object_json_file_{{dag_run.conf['param1']}}"
template='gs://dataflow-templates/your_template',
location='europe-west3',
)
According to the documentation, "If xcom_push is True, the last line written to stdout will also be pushed to an XCom when the bash command completes."
However, I need to return a multi-line string that results from a python script being executed from the terminal. I would like to subsequently use this string in an EmailOperator.
So my question is: is it possible to push more than the last line via xcom_push? Ideally, it would be arbitrarily long. I would really appreciate your help, thanks!
EDIT: I have gotten around this problem by using a PythonOperator and calling the script, but I'm still curious if it's possible to push multi-line data to XCom from a BashOperator
As clearly stated in the source code, only the last line of the BashOperator is being pushed if xcom_push = True.
:param xcom_push: If xcom_push is True, the last line written to stdout
will also be pushed to an XCom when the bash command completes.
However, you could easily create a custom operator inheriting from the BashOperator and implement the double xcom_push.
See the plugins doc on how to build custom operators with Airflow plugins.
I need a BigQueryOperator task like the following one: in which I need to save result from a query to a partitioned table. However, the "month_start" need to be derived from the actual DAG execution_date. I couldn't find any documents or examples on how to read the execution_date in my DAG definition script (in Python). Looking forward to some help here.
FYR: I'm with Airflow 1.8.2
t1_invalid_geohash_by_traffic = BigQueryOperator(
task_id='invalid_geohash_by_traffic',
bql='SQL/dangerous-area/InvalidGeohashByTraffic.sql',
params = params,
destination_dataset_table=
'mydataset.mytable${}'.format(month_start), write_disposition='WRITE_TRUNCATE',
bigquery_conn_id=CONNECTION_ID,
use_legacy_sql=False
)
I think I found the answer. Just ran into this blog: https://cloud.google.com/blog/big-data/2017/07/how-to-aggregate-data-for-bigquery-using-apache-airflow