Airflow duplicating logs - airflow

Question

I'm using Airflow 1.10.4 and I'm facing a problem when I use logging in my custom modules.
My code be like:
import logging
log = logging.getLogger(__name__)
log.info('hello there')
And the output be like:
[2020-01-22 09:44:29,954] {{logging_mixin.py:95}} INFO - [[34m2020-01-22 09:44:29,954[0m] {{[34mbase_hook.py:[0m84}} INFO[0m - hello there.
The time, file name and logging level are duplicated in every line of my log.
How can I implement the log of Airflow to avoid this? (I used self.log in my operators and there is no problem.)

Answer 1

Check the log_format value in your airflow.cfg file.
my default config :
# Log format
log_format = [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s
simple_log_format = %%(asctime)s %%(levelname)s - %%(message)s

Answer 2

Since the default logging format of Airflow is [%%(asctime)s] {%%(filename)s:%%(lineno)d} %%(levelname)s - %%(message)s, you'll always have duplicates in the log, because e.g. the datetime is once shown in the asctime and once in the message itself. So in order to fix that, you need to change the logging format of Airflow to show only the message.
If you're using docker-compose.yaml you need to set it there like this:
AIRFLOW__LOGGING__LOG_FORMAT: '%(message)s'
Otherwise you have to set it in the airflow.cfg as #AliNadi mentioned:
log_format = %%(message)s
Note: I had an additional issue - for me the message was also duplicated. The reason for that was that in the python code I was calling logging.basicConfig() and then additionally configuring the logger like this logger = logging.getLogger(__name__), logger.setLevel(logging.DEBUG) ... The solution was to remove the logging.basicConfig() and leave my custom logging configuration.
Reference: here