Airflow backfill command FERNET_KEY configuration missing - airflow

Question

Having an issue doing a backfill. When I run this in the command-line
airflow backfill my_dag -s 2021-01-01 -e 2021-01-12
the dag is triggered and begins running. The first task (a simple python script with no external dependencies) completes but in the second task, I receive this error about fernet key. The dag is just calling a MsSqlOperator.
cryptography.fernet.InvalidToken
If I trigger the dag manually in the UI, all the steps succeed.
If I trigger one execution in the dag in the CLI, the dag succeeds.
airflow dags trigger -e '2021-01-19T04:00:00' my_dag
The fernet key is in the config file and we've already run resetdb and re-created the connections. Same issue exists, backfill command doesn't work but other methods do.
Also tried using the --local flag (not sure what this does) but it doesn't work either.
Any ideas how to troubleshoot?
Running Airflow 1.10.14 on-prem with LocalExecutor. Edit: Issue exists when using .15 too.
Backfill Doc for Airflow 1.10.14
Logs:
Note, it says something about variable missing but that's misleading since it works if I manually run it from the UI. Key line is
ERROR - Can't decrypt _val for key=xyz_users_overlap_import, FERNET_KEY configuration missing
Full log:
INFO - Job 4303: Subtask download_user_files
[2021-05-17 23:41:29,327] {{logging_mixin.py:120}} INFO - Running <TaskInstance: xyz_users_overlap_import.download_user_files 2021-01-22T15:10:00+00:00 [running]> on host da2m.mycorp.corp
[2021-05-17 23:41:29,360] {{variable.py:58}} ERROR - Can't decrypt _val for key=xyz_users_overlap_import, FERNET_KEY configuration missing
[2021-05-17 23:41:29,361] {{taskinstance.py:1150}} ERROR - 'Variable xyz_users_overlap_import does not exist'
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 965, in _run_raw_task
self.render_templates(context=context)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1424, in render_templates
self.task.render_template_fields(context)
File "/usr/local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 719, in render_template_fields
self._do_render_template_fields(self, self.template_fields, context, jinja_env, set())
File "/usr/local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 726, in _do_render_template_fields
rendered_content = self.render_template(content, context, jinja_env, seen_oids)
File "/usr/local/lib/python3.8/site-packages/airflow/models/baseoperator.py", line 755, in render_template
return jinja_env.from_string(content).render(**context)
File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/usr/local/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "<template>", line 1, in top-level template code
File "/usr/local/lib/python3.8/site-packages/jinja2/environment.py", line 471, in getattr
return getattr(obj, attribute)
File "/usr/local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1335, in __getattr__
self.var = Variable.get(item, deserialize_json=True)
File "/usr/local/lib/python3.8/site-packages/airflow/models/variable.py", line 117, in get
raise KeyError('Variable {} does not exist'.format(key))
KeyError: 'Variable xyz_users_overlap_import does not exist'

Answer 1

Have you tried by generating a new FERNET_KEY and set up as env variable (in this way you will overwrite the value from the airflow.cfg file):
export AIRFLOW__CORE__FERNET_KEY=your_fernet_key
The code to generate a new one is:
from cryptography.fernet import Fernet
fernet_key= Fernet.generate_key()
print(fernet_key.decode())
Do not forget to install before the package https://pypi.org/project/cryptography/ (pip install cryptography)