DAG started from airflow scheduler not finding values in airflow.cfg - airflow

Question

I'm taking advantage of the fact that Airflow v1.7.1.3 provides access to airflow.cfg to place some configuration values there rather than embedded in the code. We added the following as the first lines of the airflow.cfg file:
[foo]
bar = foo
bar
In the foobarDAG.py class representing the DAG, I do the following:
from airflow.configuration import conf
…
def fooBar():
pass
foobarList = conf['foo']['bar'].split('\n')
foobarOperator = PythonOperator(
task_id='fooBar',
provide_context=True,
python_callable=fooBar,
op_args=[foobarList],
dag=dag)
Testing this manually from the Python prompt is easy:
>>> from foobarDAG import foobarList
…
>>> foobarList
['foo', 'bar']
That's just what I would expect from the information in airflow.cfg, above.
We've also performed a test on the DAG directly:
airflow test foobarDAG fooBar 10-19-2016
That doesn't report any problems.
The problem crops up when we try to use the scheduler to schedule that one DAG:
airflow scheduler -d foobarDAG >& foobar_log.txt
In the web UI, we see the following at the top of the "DAGS" section:
Broken DAG: [/path/to/…/foobarDAG.py] 'foo'
And in foobar_log.txt, here is the error message:
[2016-10-19 14:56:09,028] {models.py:250} ERROR - Failed to import: /path/to/foobarDAG.py
Traceback (most recent call last):
File "/path/to/airflow/models.py", line 247, in process_file
m = imp.load_source(mod_name, filepath)
File "/path/to/anaconda3/envs/foobarenv/lib/python3.5/imp.py", line 172, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 693, in _load
File "<frozen importlib._bootstrap>", line 673, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 662, in exec_module
File "<frozen importlib._bootstrap>", line 222, in _call_with_frames_removed
File "/path/to/foobarDAG.py", line 67, in <module>
foobarList = conf['foo']['bar'].split('\n')
File "/path/to/anaconda3/envs/foobarenv/lib/python3.5/configparser.py", line 956, in __getitem__
raise KeyError(key)
KeyError: 'foo'
So oddly it appears that the scheduler isn't retrieving the ['foo'] section from airflow.cfg and providing it to the DAG. Any idea why?

Answer 1

It turns out that everything was working properly, but the scheduler hadn't been restarted. The scheduler was apparently still using the old airflow.cfg which did not have the added section.