How to resolve Airflow could not allocate space for object - airflow

Question

Airflow web page shows:
"The scheduler does not appear to be running. Last heartbeat was received 6 hours ago.
The DAGs list may not update, and new tasks will not be scheduled"
Airflow is inoperable. It appears I ran out of disk space. I've manually cleared log folder and now have disk space. When I run "airflow scheduler" I get error messages below. I do not know how to resolve.
airflow scheduler
[2023-02-10 21:10:54,079] {cli_action_loggers.py:105} WARNING - Failed to log action with (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Could not allocate space for object 'dbo.log'.'PK__log__3213E83F7F1F073F' in database 'airflow' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup. (1105) (SQLExecDirectW)")
[SQL: INSERT INTO log (dttm, dag_id, task_id, event, execution_date, owner, extra) OUTPUT inserted.id VALUES (?, ?, ?, ?, ?, ?, ?)]
[parameters: (datetime.datetime(2023, 2, 10, 21, 10, 54, 51696, tzinfo=Timezone('UTC')), None, None, 'cli_scheduler', None, 'root', '{"host_name": "plappnx-1", "full_command": "[\'/usr/local/bin/airflow\', \'scheduler\']"}')]
(Background on this error at: http://sqlalche.me/e/14/f405)
sqlalchemy.exc.ProgrammingError: (pyodbc.ProgrammingError) ('42000', "[42000] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Could not allocate space for object 'dbo.job'.'PK__job__3213E83F7D216A15' in database 'airflow' because the 'PRIMARY' filegroup is full. Create disk space by deleting unneeded files, dropping objects in the filegroup, adding additional files to the filegroup, or setting autogrowth on for existing files in the filegroup. (1105) (SQLExecDirectW)")
[SQL: INSERT INTO job (dag_id, state, job_type, start_date, end_date, latest_heartbeat, executor_class, hostname, unixname) OUTPUT inserted.id VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)]
[parameters: (None, <TaskInstanceState.RUNNING: 'running'>, 'SchedulerJob', datetime.datetime(2023, 2, 10, 21, 10, 54, 981528, tzinfo=Timezone('UTC')), None, datetime.datetime(2023, 2, 10, 21, 10, 54, 981540, tzinfo=Timezone('UTC')), 'SequentialExecutor', 'plappnx-1', 'root')]

Answer 1

The problem is not related to Airflow, neither the disk space, it's a DB problem, where you have added MAXSIZE when you created your DB, and the DB log (not Airflow log) has already reached this limit.
You can delete some of the DB log files to unblock you Airflow workload, but you need a persistent solution like increasing the MAXSIZE or setting it to unlimited.
Here is a blog which explain the problem and propose some solutions.