Airflow - skip DAG if sensor failed - airflow

Question

I have process where I'm waiting for a file every week, but this file is timestamped in its name for the mesurment's date. So I know I'm going to have something this week and the name can be 2020-05-25*.csv up to 2020-05-31*.csv.
The only way I find out to start my processes with airflow is to run a sensor at start #daily and using the executing date to find is there is a file.
The thing is, since I don't know which day the file will be uploaded I will have 6 fails sensors, so 6 failed DAGs, and 1 succeeded.
SFTP Sensors part exemple :
with DAG(
"geometrie-sftp-to-safe",
default_args=default_args,
schedule_interval="#daily",
catchup=True,
) as dag:
starting_sensor = DummyOperator(
task_id="starting_sensor"
)
sensor_sftp_A = SFTPSensor(
task_id="sensor_sftp_A",
path="/input/geometrie/prod/Track_Geometry-{{ ds_nodash }}_A.csv",
sftp_conn_id="ssh_ftp_landing",
poke_interval=60,
soft_fail=True,
mode="reschedule"
)
Second With GCSSensor
with DAG(
"geometrie-preprocessing",
default_args=default_args,
schedule_interval="#daily",
catchup=True
) as dag:
# File A
sensor_gcs_A = GoogleCloudStorageObjectSensor(
task_id="gcs-sensor_A",
bucket="lisea-mesea-sea-cloud-safe",
object="geometrie/original/track_geometry_{{ ds_nodash }}_A.csv",
google_cloud_conn_id="gcp_conn",
poke_interval=50
)
That's why I would like the DAGs to be set as skipped, if and only if the sensor have fail. If it's something else I would like a real fail.

Answer 1

Airflow has multiple sensors which senses the directory to check for the defined file. The schedule_interval as None will work to your use case as you want the DAG to trigger only when the file is received(considering that the file can be received anytime within the week).
The below example for GCSSensor will sense the bucket for the particular type of file and will print the filename.I am pretty sure that SFTP sensor should work the same way.
dag = DAG(
dag_id='sensing-bucket',
schedule_interval=None,
default_args=args)
def new_file_detection(**context):
value = context['ti'].xcom_pull(task_ids='list_Files')
print('value is : '+str(value))
File_sensor = GoogleCloudStoragePrefixSensor(
task_id='gcs_polling',
bucket='lisea-mesea-sea-cloud-safe',
prefix='geometrie/original/track_geometry_',
dag=dag
)
GCS_File_list = GoogleCloudStorageListOperator(
task_id='list_Files',
bucket='lisea-mesea-sea-cloud-safe',
prefix='geometrie/original/track_geometry_',
delimiter='.csv',
google_cloud_storage_conn_id='google_cloud_default',
dag=dag
)
File_detection = PythonOperator(
task_id='print_detected_filename',
provide_context=True,
python_callable=new_file_detection,
dag=dag
)
File_sensor >> GCS_File_list >> File_detection