ZenML Dag didn't show up in Airflow UI - airflow

Question

I was trying out ZenML, it said it can convert my .py pipeline into airflow DAG.
I followed every step here: https://docs.zenml.io/guides/low-level-api/chapter-7, all succeeded
My pipeline runs well locally, but why can't see this DAG created on airflow UI? The UI is totally empty....
The problem seems that, ZenML will copy my .py pipeline wrote in ZenML way, and expect it can run in airflow... In my case this won't work. Does anyone know how can I let ZenML run my code through airflow successfully?
Here's my ZenML .py code:
import pandas as pd
import numpy as np
import os
import lightgbm as lgb
from sklearn.model_selection import StratifiedKFold
from sklearn.metrics import balanced_accuracy_score
from zenml.pipelines import pipeline
from zenml.steps import step
from zenml.steps.step_output import Output
from zenml.steps.base_step_config import BaseStepConfig
class pipeline_config(BaseStepConfig):
"""
Params used in the pipeline
"""
label: str = 'species'
#step
def split_data(config: pipeline_config) -> Output(
X=pd.DataFrame, y=pd.DataFrame
):
path_to_csv = os.path.join('~/airflow/data', 'leaf.csv')
df = pd.read_csv(path_to_csv)
label = config.label
y = df[[label]]
X = df.drop(label, axis=1)
return X, y
#step
def train_evaltor(
config: pipeline_config,
X: pd.DataFrame,
y: pd.DataFrame
) -> float:
y = y[config.label]
folds = StratifiedKFold(n_splits=5, shuffle=True, random_state=10)
lgbm = lgb.LGBMClassifier(objective='multiclass', random_state=10)
metrics_lst = []
for train_idx, val_idx in folds.split(X, y):
X_train, y_train = X.iloc[train_idx], y.iloc[train_idx]
X_val, y_val = X.iloc[val_idx], y.iloc[val_idx]
lgbm.fit(X_train, y_train)
y_pred = lgbm.predict(X_val)
cv_balanced_accuracy = balanced_accuracy_score(y_val, y_pred)
metrics_lst.append(cv_balanced_accuracy)
avg_performance = np.mean(metrics_lst)
print(f"Avg Performance: {avg_performance}")
return avg_performance
#pipeline
def super_mini_pipeline(
data_spliter,
train_evaltor
):
X, y = data_spliter()
train_evaltor(X=X, y=y)
# run the pipeline
pipeline = super_mini_pipeline(data_spliter=split_data(),
train_evaltor=train_evaltor())
pipeline.run()

Answer 1

Okay so it worked! See picture below:
The reason is that the words airflow and dag have to be in the airflow if safe_mode is on (which it is by default). This is airflow specific logic can be in the airflow codebase.
So all I did was change the last few lines:
# run the pipeline airflow
pipeline = super_mini_pipeline(data_spliter=split_data(),
train_evaltor=train_evaltor())
DAG = pipeline.run()
You can also change the airflow.cfg file and turn safe mode off:
In $HOME/.config/zenml/airflow_root/<UUID>/airflow.cfg
# When discovering DAGs, ignore any files that don't contain the strings ``DAG`` and ``airflow``.
dag_discovery_safe_mode = False
Edit:
There might be another reason: Airflow DAG discovery also relies on the DAG being the globals() so maybe we needed to catch it with DAG = pipeline.run(). So in any case, it works!