Is there any way to set a variable in Airflow UI to get today.date() or something similar to {{ds}} in the DAG code?
I want to have flexibility to set a hard code date in variable without changing the DAG code for some use cases.
I am getting today date in DAG code right now:
today = datetime.today()
but wanted to get it like this:
today= models.Variable.get('todayVar')
This is a duplicate of stackoverflow post:
Airflow - Get start time of dag run
You can achieve what you want by:
{{ dag_run.start_date }}
In airflow the date you are meaning is also called the 'run_date'
Related
I am trying to run a batch predict job task using the Vertex AI Airflow Operator CreateBatchPredictionJobOperator. This requires pulling a model id from XCom which was pushed by a previous custom container training job. However, CreateBatchPredictionJobOperator doesn't seem to render Xcom pulls as expected.
I am running Airflow 2.3.0 on my local machine.
My code looks something like this:
batch_job_task = CreateBatchPredictionJobOperator(
gcp_conn_id="gcp_connection",
task_id="batch_job_task",
job_display_name=JOB_DISPLAY_NAME,
model_name="{{ ti.xcom_pull(key='model_conf')['model_id'] }}",
predictions_format="bigquery",
bigquery_source=BIGQUERY_SOURCE,
region=REGION,
project_id=PROJECT_ID,
machine_type="n1-standard-2",
bigquery_destination_prefix=BIGQUERY_DESTINATION_PREFIX,
This results in a value error when the task runs:
ValueError: Resource {{ ti.xcom_pull(key='model_conf')['model_id'] }} is not a valid resource id.
The expected behaviour would be to pull that variable by key and render it as a string.
I can also confirm that I am able to see the model id (and other info) in XCom by navigating there in the UI. I attempted using the same syntax with xcom_pull with a PythonOperator and it works.
def print_xcom_value(value):
print("VALUE:", value)
print_xcom_value_by_key = PythonOperator(
task_id="print_xcom_value_by_key", python_callable=print_xcom_value,
op_kwargs={"value": "{{ ti.xcom_pull(key='model_conf')['model_id'] }}" },
provide_context=True,
)
> [2022-12-15, 13:11:19 UTC] {logging_mixin.py:115} INFO - VALUE: 3673414612827265024
CreateBatchPredictionJobOperator does not accept provide_context as a variable. I assumed it would render xcom pulls by default since xcom pulls are used in the CreateBatchPredictionJobOperator in an example on the Airflow docs (link here).
Is there any way I can provide context to this Vertex AI Operator to pull from the XCom storage?
Is something wrong with the syntax that I am not seeing? Anything I a misunderstanding in the docs?
UPDATE:
One thing that confuses me is that model_name is a templated field according to the Airflow docs (link here) but the field is not rendering the XCom template.
Did you set render_template_as_native_obj=True in your DAG definition?
What version of apache-airflow-providers-google do you use?
====
From OP:
Your answer was a step in the right direction.
The solution was to upgrade apache-airflow-providers-google to the latest version (at the moment, this is 8.6.0). I'm not able to pinpoint exactly where in the changelog this fix is mentioned.
Setting render_template_as_native_obj=True was not useful for this issue since it rendered the id pulled from XCom as an int, and I found no proper way to convert it to str when passed into CreateBatchPredictionJobOperator in the model_name arg.
Currently my DAG utilizes the {{ prev_ds }} variable.
I would like to trigger a DAG run manually. However when I trigger manually from the UI with an execution date of '2021-12-14' the {{ prev_ds }} value gets set to '2021-12-14'. Is there a way through the UI or CLI to set that value appropriately?
When manually triggering DAG prev_ds == next_ds == ds ( see docs)
These are the proper values as manual run has no prev and has no next. There is no workaround for this from UI nor CLI.
You can add conditions to your jinja templates to bring diffrent values bases on run type (manual, schedual) but that means code changes.
When triggering a dag in airflow, there is a window, with which I am able to parameters to the dag in a json format. This looks like the following:
This json is always empty and I do have to know which parameters I can pass to the dag. Instead I would like to be able to prefill this json, so that when another user tries to trigger the dag he can simply change to values of the json, instead of having to look at the dags code first.
Is there any way to do this in the current version (2.0.0) of airflow?
On Airflow 2.1.0 it is possible to set default arguments as follow:
dag = DAG(dag_id="my_dag",
schedule_interval=None,
default_args={'retries': 3, 'retry_delay': timedelta(seconds=20)},
catchup=False,
tags=['maintenance'],
params={"description": ""} #Set parameters as a dictionary
)
In the thrigger UI it looks like this:
When writing my feature request i actually found a pull request, which is already merged and seems to exactly do as described:
https://github.com/apache/airflow/pull/10839
An improvement of the this feature also seems to be planned. See:
https://github.com/apache/airflow/issues/11054
No, it is currently not supported -- at least for Airflow 2.0.0
I have a DAG with one DataflowTemplateOperator that can deal with different json files. When I trigger the dag I pass some parameters via {{dag_run.conf['param1']}} and works fine.
The issue I have is trying to rename the task_id based on param1.
i.e. task_id="df_operator_read_object_json_file_{{dag_run.conf['param1']}}",
it complains about only alphanumeric characters
or
task_id="df_operator_read_object_json_file_{}".format(dag_run.conf['param1']),
it does not recognise dag_run plus the alpha issue.
The whole idea behind this is that when I see at the dataflow jobs console and job has failed I know who the offender is based on param1. Dataflow Job names are based on task_id like this:
df-operator-read-object-json-file-8b9eecec
and what I need is this:
df-operator-read-object-param1-json-file-8b9eecec
Any ideas if this is possible?
There is no need to generate new operator per file.
DataflowTemplatedJobStartOperator has job_name parameter which is also templated so can be used with Jinja.
I didn't test it but this should work:
from airflow.providers.google.cloud.operators.dataflow import DataflowTemplatedJobStartOperator
op = DataflowTemplatedJobStartOperator(
task_id="df_operator_read_object_json_file",
job_name= "df_operator_read_object_json_file_{{dag_run.conf['param1']}}"
template='gs://dataflow-templates/your_template',
location='europe-west3',
)
I need a BigQueryOperator task like the following one: in which I need to save result from a query to a partitioned table. However, the "month_start" need to be derived from the actual DAG execution_date. I couldn't find any documents or examples on how to read the execution_date in my DAG definition script (in Python). Looking forward to some help here.
FYR: I'm with Airflow 1.8.2
t1_invalid_geohash_by_traffic = BigQueryOperator(
task_id='invalid_geohash_by_traffic',
bql='SQL/dangerous-area/InvalidGeohashByTraffic.sql',
params = params,
destination_dataset_table=
'mydataset.mytable${}'.format(month_start), write_disposition='WRITE_TRUNCATE',
bigquery_conn_id=CONNECTION_ID,
use_legacy_sql=False
)
I think I found the answer. Just ran into this blog: https://cloud.google.com/blog/big-data/2017/07/how-to-aggregate-data-for-bigquery-using-apache-airflow