Airflow task instance stuck in retry mode - airflow

Question

We are running airflow version 1.9 in celery executor mode. Our task instances are stuck in retry mode. When the job fails, the task instance retries. After that it tries to run the task and then fall back to new retry time.
First state:
Task is not ready for retry yet but will be retried automatically. Current date is 2018-08-28T03:46:53.101483 and task will be retried at 2018-08-28T03:47:25.463271.
After some time:
All dependencies are met but the task instance is not running. In most cases this just means that the task will probably be scheduled soon unless:
- The scheduler is down or under heavy load
If this task instance does not start soon please contact your Airflow administrator for assistance
After some time: it again went into retry mode.
Task is not ready for retry yet but will be retried automatically. Current date is 2018-08-28T03:51:48.322424 and task will be retried at 2018-08-28T03:52:57.893430.
This is happening for all dags. We created a test dag and tried to get logs for both scheduler and worker logs.
from datetime import *
from airflow import DAG
from airflow.operators.python_operator import PythonOperator
from airflow.operators.bash_operator import BashOperator
default_args = {
'owner': 'Pramiti',
'depends_on_past': False,
'retries': 3,
'retry_delay': timedelta(minutes=1)
}
dag = DAG('airflow-examples.test_failed_dag_v2', description='Failed DAG',
schedule_interval='*/10 * * * *',
start_date=datetime(2018, 9, 7), default_args=default_args)
b = BashOperator(
task_id="ls_command",
bash_command="mdr",
dag=dag
)
Task Logs
Scheduler Logs
Worker Logs