Airflow systemd scheduler not working - airflow

Question

I got the webserver to work properly, and my airflow-scheduler.service file starts the scheduler and it finds my dags etc. However, tasks are not running:
I see an error message about /bin/sh
ERROR - failed to execute task Command 'exec bash -c run'
I have my sysconfig file:
#!/bin/bash
PATH=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow
AIRFLOW_CONFIG=/mnt/var/airflow/airflow.cfg
AIRFLOW_HOME=/mnt/var/airflow
And my airflow-scheduler.service file:
#!/bin/bash
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service
Wants=postgresql.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=airflow
Group=airflow
Type=simple
ExecStart=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow scheduler
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
Here is the journalctl record which shows the bash error I am getting:
[2017-10-30 18:36:13,764] {base_executor.py:50} INFO - Adding to queue: airflow run user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airflow/dags/bin/user_p
Oct 30 18:36:13 airflow[4742]: [2017-10-30 18:36:13,765] {jobs.py:1443} INFO - Heartbeating the executor
Oct 30 18:36:13 airflow[4742]: [2017-10-30 18:36:13,783] {local_executor.py:45} INFO - LocalWorker running airflow run user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airflow/dags/bin/us
Oct 30 18:36:13 airflow[4742]: /bin/sh: 1: exec: bash: not found
Oct 30 18:36:13 airflow[4742]: [2017-10-30 18:36:13,865] {local_executor.py:52} **ERROR - failed to execute task Command 'exec bash -c 'airflow run** user_presence_raw_etl transform_raw_user_presence 2017-10-30T14:00:00 --local -sd /mnt/var/airf
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,786] {jobs.py:1407} INFO - Heartbeating the process manager
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,786] {dag_processing.py:559} INFO - Processor for /mnt/var/airflow/dags/bin/prod/hourly_agent_dag.py finished
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,789] {dag_processing.py:627} INFO - Started a process (PID: 5425) to generate tasks for /mnt/var/airflow/dags/bin/prod/daily_agent_email_dag.py - logging into /mnt/var/airflow/l
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,831] {jobs.py:1000} INFO - No tasks to send to the executor
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,832] {jobs.py:1443} INFO - Heartbeating the executor
Oct 30 18:36:14 airflow[4742]: [2017-10-30 18:36:14,833] {jobs.py:1195} INFO - Executor reports user_presence_raw_etl.transform_raw_user_presence execution_date=2017-10-30 14:00:00 as failed

Answer 1

Looks like your sysconfig file is clobbering the $PATH environment variable, hence the error:
Oct 30 18:36:13 airflow[4742]: /bin/sh: 1: exec: bash: not found
Try setting that line to the following:
PATH=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow:$PATH

Answer 2

It looks like you installed airflow as a user other than airflow. I have a airflow-scheduler.service file that specifies the user that I used to install airflow as, in my case, mghen.
[Unit]
Description=Airflow scheduler daemon
After=network.target postgresql.service mysql.service redis.service rabbitmq-server.service
Wants=postgresql.service mysql.service redis.service rabbitmq-server.service
[Service]
EnvironmentFile=/etc/sysconfig/airflow
User=mghen
Group=mghen
Type=simple
ExecStart=/usr/local/bin/airflow scheduler
Restart=always
RestartSec=5s
[Install]
WantedBy=multi-user.target
I fought with a similar issue but didn't see this same error message. Anyway, maybe changing the User and Group might help. Since you specify an absolute path when you start airflow (ExecStart=/opt/anaconda/anaconda3/envs/airflow_env/bin/airflow scheduler) it makes it look like it's starting fine as the airflow user but really that airflow user can't actually run airflow the program because another user installed it. Idk how to fix the PATH I just changed User and Group in my airflow-*.service files.
Alternatively, perhaps you can install airflow as the airflow user so it's available in the airflow user's PATH.

Answer 3

Workers processes spawned by LocalExecutor cannot run airflow since it's not available in PATH. You only configure it for sysconfig. In example provided in airflow repository, airflow-scheduler.service they expect airflow to be executable for a user.
My suggestion is to create virtualenv for airflow, install airflow there and then activate it for each shell where you want to run something related to airflow e.g. workers, scheduler or webserver.