MWAA Airflow "Executor reports task instance <TaskInstance: ... [queued]> finished (success) although the task says its queued." Causing task failure - airflow

Question

Running AWS Airflow Managed (2.2.2) service with Celery, multiple schedulers, and noticing a lot of tasks that use Sensors (time-based reschedule sensors) seem be marked as failed even though some of them continue to run until completion.
The Sensor Operator is fairly simple API-checker that polls API every 5 min until it receives a certain response.
We have an on_failure_handler written which captures all of these as they happen and sends slack notification.
Generally seeing the following sequence of logs in CloudWatch (no worker logs for some reason available):
{
"#timestamp": "2023-01-19 10:57:39.371",
"#message": "[\u001b[34m2023-01-19 10:57:38,924\u001b[0m] {{\u001b[34mscheduler_job.py:\u001b[0m563}} INFO\u001b[0m - TaskInstance Finished: dag_id=custom_dag, task_id=task_xyz, run_id=run_123, run_start_date=2023-01-19 10:50:17.473702+00:00, run_end_date=2023-01-19 10:57:31.030716+00:00, run_duration=433.557014, state=failed, executor_state=success, try_number=1, max_tries=0, job_id=5942262, pool=default_pool, queue=airflow-celery-f742a6b6-c8cc-4a29-81a0-eb81cc44e889, priority_weight=7, operator=CustomSensorOperator\u001b[0m\n",
"#logStream": "scheduler_console_ip-10-1-2-43.ec2.internal_1673666911.1179407.log"
},
{
"#timestamp": "2023-01-19 10:57:39.122",
"#message": "[\u001b[34m2023-01-19 10:57:38,863\u001b[0m] {{\u001b[34mscheduler_job.py:\u001b[0m510}} INFO\u001b[0m - Executor reports execution of custom_dag.task_xyz run_id=run_123 exited with status success for try_number 1\u001b[0m\n",
"#logStream": "scheduler_console_ip-10-1-2-43.ec2.internal_1673666911.1179407.log"
},
{
"#timestamp": "2023-01-19 10:57:32.718",
"#message": "[2023-01-19 10:57:32,718] {{taskinstance.py:1029}} INFO - Dependencies not met for <TaskInstance: custom_dag.task_xyz run_123 [failed]>, dependency 'Task Instance State' FAILED: Task is in the 'failed' state which is not a valid state for execution. The task must be cleared in order to be run.",
"#logStream": "custom_dag/task_xyz/2023-01-19T07_41_27+00_00/1.log"
},
{
# THIS CAUSES TASK INSTANCE TO FAIL EVEN THROUGH ITS STILL RUNNING
"#timestamp": "2023-01-19 10:57:31.730",
"#message": "[\u001b[34m2023-01-19 10:57:30,661\u001b[0m] {{\u001b[34mscheduler_job.py:\u001b[0m572}} ERROR\u001b[0m - Executor reports task instance <TaskInstance: custom_dag.task_xyz run_123 [queued]> finished (success) although the task says its queued. (Info: None) Was the task killed externally?\u001b[0m\n",
"#logStream": "scheduler_console_ip-10-1-3-8.ec2.internal_1674030934.1519198.log"
},
{
"#timestamp": "2023-01-19 10:57:31.719",
"#message": "[\u001b[34m2023-01-19 10:57:30,661\u001b[0m] {{\u001b[34mscheduler_job.py:\u001b[0m563}} INFO\u001b[0m - TaskInstance Finished: dag_id=custom_dag, task_id=task_xyz, run_id=run_123, run_start_date=2023-01-19 10:50:17.473702+00:00, run_end_date=2023-01-19 10:50:18.224642+00:00, run_duration=0.75094, state=queued, executor_state=success, try_number=1, max_tries=0, job_id=5942262, pool=default_pool, queue=airflow-celery-f742a6b6-c8cc-4a29-81a0-eb81cc44e889, priority_weight=7, operator=CustomSensorOperator\u001b[0m\n",
"#logStream": "scheduler_console_ip-10-1-3-8.ec2.internal_1674030934.1519198.log"
},
{
"#timestamp": "2023-01-19 10:57:30.740",
"#message": "[\u001b[34m2023-01-19 10:57:30,580\u001b[0m] {{\u001b[34mscheduler_job.py:\u001b[0m510}} INFO\u001b[0m - Executor reports execution of custom_dag.task_xyz run_id=run_123 exited with status success for try_number 1\u001b[0m\n",
"#logStream": "scheduler_console_ip-10-1-3-8.ec2.internal_1674030934.1519198.log"
},
{
"#timestamp": "2023-01-19 10:57:27.854",
"#message": "[\u001b[34m2023-01-19 10:57:27,612\u001b[0m] {{\u001b[34mscheduler_job.py:\u001b[0m538}} INFO\u001b[0m - Setting external_id for <TaskInstance: custom_dag.task_xyz run_123 [queued]> to c244ff79-0e92-4c3d-8c89-0440af154848\u001b[0m\n",
"#logStream": "scheduler_console_ip-10-1-2-43.ec2.internal_1673666911.1179407.log"
},
{
"#timestamp": "2023-01-19 10:57:27.544",
"#message": "[\u001b[34m2023-01-19 10:57:27,465\u001b[0m] {{\u001b[34mscheduler_job.py:\u001b[0m510}} INFO\u001b[0m - Executor reports execution of custom_dag.task_xyz run_id=run_123 exited with status queued for try_number 1\u001b[0m\n",
"#logStream": "scheduler_console_ip-10-1-2-43.ec2.internal_1673666911.1179407.log"
},
{
# THIS IS THE SUSPECIOUS SETTING OF THE TASK TO STATE SUCCESS
"#timestamp": "2023-01-19 10:57:26.361",
"#message": "\t<TaskInstance: custom_dag.task_xyz run_123 [queued]> in state SUCCESS\n",
"#logStream": "scheduler_console_ip-10-1-3-8.ec2.internal_1674030934.1519198.log"
},
{
"#timestamp": "2023-01-19 10:57:26.054",
"#message": "[\u001b[34m2023-01-19 10:57:25,349\u001b[0m] {{\u001b[34mbase_executor.py:\u001b[0m82}} INFO\u001b[0m - Adding to queue: ['airflow', 'tasks', 'run', 'custom_dag', 'task_xyz', 'run_123', '--local', '--subdir', 'DAGS_FOLDER/custom_dag.py']\u001b[0m\n",
"#logStream": "scheduler_console_ip-10-1-2-43.ec2.internal_1673666911.1179407.log"
},
{
"#timestamp": "2023-01-19 10:57:26.042",
"#message": "[\u001b[34m2023-01-19 10:57:25,349\u001b[0m] {{\u001b[34mscheduler_job.py:\u001b[0m450}} INFO\u001b[0m - Sending TaskInstanceKey(dag_id='custom_dag', task_id='task_xyz', run_id='run_123', try_number=1) to executor with priority 7 and queue airflow-celery-f742a6b6-c8cc-4a29-81a0-eb81cc44e889\u001b[0m\n",
"#logStream": "scheduler_console_ip-10-1-2-43.ec2.internal_1673666911.1179407.log"
},
{
"#timestamp": "2023-01-19 10:57:25.796",
"#message": "\t<TaskInstance: custom_dag.task_xyz run_123 [scheduled]>\n",
"#logStream": "scheduler_console_ip-10-1-2-43.ec2.internal_1673666911.1179407.log"
},
{
"#timestamp": "2023-01-19 10:57:25.542",
"#message": "[2023-01-19 10:57:25,516: INFO/ForkPoolWorker-3] Executing command in Celery: ['airflow', 'tasks', 'run', 'custom_dag', 'task_xyz', 'run_123', '--local', '--subdir', 'DAGS_FOLDER/custom_dag.py']\n",
"#logStream": "worker_console_ip-10-1-3-191.ec2.internal_1674110783.438406.log"
},
{
"#timestamp": "2023-01-19 10:57:25.425",
"#message": "\t<TaskInstance: custom_dag.task_xyz run_123 [scheduled]>\n",
"#logStream": "scheduler_console_ip-10-1-2-43.ec2.internal_1673666911.1179407.log"
},
There are a couple of times when the task is set to error, handler triggered, but the scheduler continues to check the task instance until it actually completes successfully, but the failure state is persisted in the database.
I suspect that the one scheduler message that marks the task as [queued] in state SUCCESS is what is causing the error, but I can't identify why this is happening, or the root cause. All the tasks are manually triggered through an API and are not scheduled - so I do not think the pickling and backfill solutions proposed would help.
I am not entirely sure if this is a scheduler interval issue; but if it is, what could be causing it?
Based on the logStream it almost seems as though two different schedules are competing to set task state.
Checked logs to try and find root cause, but nothing indicative. Have not yet restarted scheduler since MWAA requires taking down all of Airflow for 20-30 min which is not ideal at the moment.