The accepted answer to this question states that
"...the gs://my-bucket/dags folder is available in the scheduler, web server, and workers at /home/airflow/gcs/dags."
(which is supported by the newer docs)
So I wrote a bash operator like this:
t1 = bash.BashOperator(
task_id='my_test',
bash_command="touch /home/airflow/gcs/data/test.txt",
)
I thought by prefacing my file creation with the path specified in the answer it would write to the data folder in my cloud composer environment's associated storage account. Simiarly, touch test.txt also ran successfully but didn't actually create a file anywhere I can see it (I assume it's written to the worker's temp storage which is then deleted when the worker is shut down following execution of the DAG). I can't seem to persist any data from simple commands run through a DAG? Is it even possible to simply write out some files from a bash script running in Cloud Composer? Thank you in advance.
Bizarrely, I needed to add a space at the end of the string containing the Bash command.
t1 = bash.BashOperator(
task_id='my_test',
bash_command="touch /home/airflow/gcs/data/test.txt ",
)
The frustrating thing was the error said the path didn't exist so I went down a rabbit-hole mapping the directories of the Airflow worker until I was absolutely certain it did - then I found a similar issue here. Although I didn't get the 'Jinja Template not Found Error' I should have got according to this note.
I have 2 files inside dags directory - dag_1.py and dag_2.py
dag_1.py creates a static DAG and dag_2.py creates dynamic DAGs based on external json files at some location.
The static DAG (created by dag_1.py) contains a task at a later stage which generates some of these input json files for dag_2.py and dynamic DAGs are created in this manner.
This used to work fine with Airflow 1.x versions where DAG Serialization was not used. But with Airflow 2.0 DAG Serialization has become mandatory. Sometimes, I get the following exception in the Scheduler when dynamic DAGs are spawned -
[2021-01-02 06:17:39,493] {scheduler_job.py:1293} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
Traceback (most recent call last):
File "/global/packages/python/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1275, in _execute
self._run_scheduler_loop()
File "/global/packages/python/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1377, in _run_scheduler_loop
num_queued_tis = self._do_scheduling(session)
File "/global/packages/python/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1474, in _do_scheduling
self._create_dag_runs(query.all(), session)
File "/global/packages/python/lib/python3.7/site-packages/airflow/jobs/scheduler_job.py", line 1557, in _create_dag_runs
dag = self.dagbag.get_dag(dag_model.dag_id, session=session)
File "/global/packages/python/lib/python3.7/site-packages/airflow/utils/session.py", line 62, in wrapper
return func(*args, **kwargs)
File "/global/packages/python/lib/python3.7/site-packages/airflow/models/dagbag.py", line 171, in get_dag
self._add_dag_from_db(dag_id=dag_id, session=session)
File "/global/packages/python/lib/python3.7/site-packages/airflow/models/dagbag.py", line 227, in _add_dag_from_db
raise SerializedDagNotFound(f"DAG '{dag_id}' not found in serialized_dag table")
airflow.exceptions.SerializedDagNotFound: DAG 'dynamic_dag_1' not found in serialized_dag table
After this the scheduler gets terminated which is expected.
When I check the table manually after this error, I am able to see the DAG entry in it.
This issue is not reproducible all the time. What can be the probable cause for this? Is there any Airflow configuration which I should try tweaking?
We had the same issue after updating in the following order:
1.10.12 -> 1.10.14
1.10.14 -> 2.0.0
I've followed their guide through, and we had no issues until at some random point after a few hours scheduler started crashing complaining about random DAGs not being found in the database.
Our deployment procedure involves clearing out /opt/airflow/dags folder and doing a clean install every time (we store dags and supporting code in python packages)
So every now and then on 1.10.x version we had cases when scheduler parsed an empty folder and wiped serialized dags from the database, but it always was able to restore the picture on next parse
Apparently in 2.0, as a part of the effort to make scheduler HA, they fully separated DAG processor and scheduler. Which leads to a race condition:
if scheduler job hits a database before DAG processor has updated serialized_dag table values, it finds nothing and crashes
if luck is on your side, the above will not happen and you won't see this exception
In order to get rid of this problem, I disabled scheduling of all DAGs by updating is_paused in the database, restarted the scheduler and once it generated serialized dags, turned all dags back ON
I fixed this issue in https://github.com/apache/airflow/pull/13893 which will be released as part for Airflow 2.0.1.
Will release Airflow 2.0.1 next week (8 Feb 2021 - most likely).
Not enough rep to comment so having to leave an answer, but:
is this a clean 2.0 install or an upgrade of your old 1.10.x instance? and
are you recycling the names?
I literally just hit this problem (I found this question googling to see who else was in the same boat).
In my case, it's an upgraded existing 1.10.x install, and although the dags were generated dynamically, the names were recycled. I was getting errors clicking on the dag in the GUI and it was killing the scheduler.
Turns Out(TM), deleting the dags entirely using the 'trashcan' button in the GUI overview and letting them regenerate fixed it (as in, the problem immediately went away and hasn't recurred in the last 30 minutes).
At a guess, it smells to me like maybe some aspect of the dynamic dags wasn't properly migrated in the db upgrade step, and wiping them out and letting them fully regenerate fixed the problem. Obviously, you lose all your history etc, but (in my case at least) that's not necessarily a big deal.
Selected answer didn't work for me(after bashing my head for few hours).
Heres what works:
Just go to the backend database(postgresql) and delete all the records regarding logs, task instances, faild task and ... but dont delete the main tables(if you cant tell the diffrence, just remove the tables i mentioned)
then go and do airflow db init
seems like old data about obsolete and deleted dags and tasks can really turn airflow into a mess. delete the mess, get it working.
I am very new to Airflow, I have set-up everything according to what are stated on their website. However I find it very confusing to figure out my dag folder location. NOTE: I configure **airflow.cfg (/airflow/dags) within this folder has two files.
/airflow/dags
---dag1.py
---dag2.py
But when I try to do airflow list_dags, it still shows the dags inside example_dags folder on
usr/local/lib/python2.7/dist_packages/airflow/example_dags
How can I see the path when I do airflow list_dags and to change it ? Any helps is appreciated.
There is an airflow.cfg value under the [core] section called load_examples that you can set to false to exclude the example DAGs. I think that should clean up the output you’re seeing from list_dags.
I have recently started working on Airflow scheduler. And I have been observing that it is creating multiple log files for each job that is scheduled. May I know how to restrict it to just one file. I have checked airflow.cfg file and I weren't able to find any argument which is related to the number of copies of log files.
Is there any way to reload the jobs without having to restart the server?
In your airflow.cfg, you've these two configurations to control this behavior:
# after how much time a new DAGs should be picked up from the filesystem
min_file_process_interval = 0
dag_dir_list_interval = 60
You might have to reload the web-server, scheduler and workers for your new configuration to take effect.
I had the same question, and didn't see this answer yet. I was able to do it from the command line with the following:
python -c "from airflow.models import DagBag; d = DagBag();"
When the webserver is running, it refreshes dags every 30 seconds or so by default, but this will refresh them in between if necessary.
Dags should be reloaded when you update the associated python file.
If they are not, first try to manually refresh them in UI by clicking the button that looks like a recycle symbol:
If that doesn't work, delete all the .pyc files in the dags folder.
Usually though, when I save the python file the dag gets updated within a few moments.
I'm pretty new to airflow, but I had initially used sample code, which got picked up right away and then edited it to call my own code.
This was ultimately giving an error, but I only found this out once I had deleted the DAG with the example code, on the airflow webserver's UI (the trash button):
Once deleted, it showed me the error which was preventing it from loading the new dag.