Airflow webserver start only in debug mode - airflow

Question

Airflow webserver start only in debug mode
airflow webserver -p 8051
[2016-10-18 18:41:13,816] {__init__.py:36} INFO - Using executor CeleryExecutor
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-10-18 18:41:14,081] {models.py:154} INFO - Filling up the DagBag from /home/user/some_dir/airflow/dags
Running the Gunicorn server with 4 syncworkers on host 0.0.0.0 and port 8051 with a timeout of 120...
['gunicorn', '-w 4', '-k sync', '-t 120', '-b 0.0.0.0:8051', '-n airflow-webserver', '-p /home/user/some_dir/airflow/airflow-webserver.pid', 'airflow.www.app:cached_app()']
Traceback (most recent call last):
File "./airflow", line 15, in <module>
args.func(args)
File "/home/user/userenv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 426, in webserver
'gunicorn', run_args
File "/home/user/userenv/lib/python2.7/os.py", line 344, in execvp
_execvpe(file, args)
File "/home/user/userenv/lib/python2.7/os.py", line 380, in _execvpe
func(fullname, *argrest)
OSError: [Errno 2] No such file or directory
But airflow webserver -p 8051 -d works
[2016-10-18 18:45:45,750] {__init__.py:36} INFO - Using executor CeleryExecutor
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-10-18 18:45:46,019] {models.py:154} INFO - Filling up the DagBag from /home/user/some_dir/airflow/dags
Starting the web server on port 8051 and host 0.0.0.0.
[2016-10-18 18:45:46,138] {_internal.py:87} INFO - * Running on http://0.0.0.0:8051/ (Press CTRL+C to quit)
[2016-10-18 18:45:46,139] {_internal.py:87} INFO - * Restarting with stat
[2016-10-18 18:45:46,417] {__init__.py:36} INFO - Using executor CeleryExecutor
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
[2016-10-18 18:45:46,682] {models.py:154} INFO - Filling up the DagBag from /home/user/some_dir/airflow/dags
Starting the web server on port 8051 and host 0.0.0.0.
[2016-10-18 18:45:46,796] {_internal.py:87} WARNING - * Debugger is active!
[2016-10-18 18:45:46,850] {_internal.py:87} INFO - * Debugger pin code: 231-950-074
Also I can't see airflow-webserver.pid file in $AIRFLOW_HOME
But in debug mode I can't run airflow worker
airflow worker
[2016-10-18 18:48:28,541] {__init__.py:36} INFO - Using executor CeleryExecutor
Traceback (most recent call last):
File "./airflow", line 15, in <module>
args.func(args)
File "/home/user/userenv/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 519, in worker
sp = subprocess.Popen(['airflow', 'serve_logs'], env=env)
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Answer 1

After reading through the source code of cli.py (see the webserver and setup_locations functions). I would guess that you do not have access to create the files that will be setup when the gunicorn process starts. Ensure that you have write access to AIRFLOW_HOME. If you don't either change the directory value you have exported or change the permissions on the directory such that Airflow can write there.