↓Skip to main content
  1. Posts/

Configure Python logging with dictConfig

··503 words·3 mins·
Python Logging
Table of Contents

Apart from directly configuring the Python logging in the code itself. We can also configuring the logging using fileConfig or dictConfig.

dictConfig in logging
#

The dictConfig is more powerful than fileConfig and is the recommended way to configure logging.

import json
import logging

with open("path/to/logging_conf.json") as f:
    conf = json.load(f)

logging.config.dictConfig(conf)

The content of logging_conf.json:

{
    "version": 1,
    "disable_existing_loggers": True,
    "formatters": {
        "standard":{
            "format": "%(asctime)s [%(levelname)s] [%(name)s:%(lineno)d]: %(message)s"
        }
    },
    "handlers":
    {
        "console":{
            "level": "DEBUG",
            "formatter": "standard",
            "class": "logging.StreamHandler",
            "stream": "ext://sys.stdout"
        }
    },
    "loggers":{
        "requests":{
            "level": "WARNING"
        },
        "my_module":{
            "level": "INFO"
        }
    },
    "root": {
        "level": "DEBUG",
        "handlers": ["console"]
    }
}

Note that the ext://sys.stdout is a special syntax to refer to internal Python object. More info can be found in https://docs.python.org/3/library/logging.config.html#access-to-external-objects.

You can also write the logging configuration in YAML format (easier to write) and read it using the yaml package.

version: 1
disable_existing_loggers: true
formatters:
    standard:
        format: '%(asctime)s [%(levelname)s] [%(name)s:%(lineno)d]: %(message)s'
handlers:
    console:
        level: DEBUG
        formatter: standard
        class: logging.StreamHandler
        stream: ext://sys.stdout
loggers:
    requests:
        level: WARNING
    my_module:
        level: INFO
root:
    level: DEBUG
    handlers: ['console']

disable_existing_loggers
#

Note that you should be very careful with disable_existing_loggers. If this option is set to True, any logger that is defined before calling logging.config.dictConfig() will be disabled. This may cause subtle logging issues that are hard to debug.

For example, suppose you have a module my_module.py:

import logging

logger = logging.getLogger(__name__)

# ....

def some_func():
    logger.info("some message from my_module")
    # ... other statements

In your entry point to this project, you have something like this:

import logging
import my_module

logging.config.dictConfig("/path/to/config/json")

my_module.some_func()

The logging message from my_module will not be printed. You need to set disable_existing_loggers to False, or you need to configure the logger for my_module.py explicitly under the loggers keys in the JSON config file.

Under the key loggers, you can configure the behavior of different loggers.

Disable logging for 3rd party package/modules
#

To disable logging from 3rd party packages/modules, e.g., “urllib3”, we can configure their logging level in the configuration file like this:

loggers:
    urllib3:
        level: WARNING

Or you can also use Python code to disable the logging for a module completely:

logging.getLogger("urllib3").disabled = True

Check existing logger config?
#

Sometimes, we way to check the configuration for a logger to debug the logging issue. To check the existing loggers, we can use logging.root.manager.loggerDict. The keys are the module names and the value are the respecitve loggers.

import logging
current_logger = logging.root.manager.loggerDict['__main__']
print(current_logger.level, current_logger.disabled)

After you get the loggers, you can check the handlers associated with the loggers. Further, you can also check the formatter that is used by a handler. In this way, you can get an idea of the configuration for the different loggers.

Note that logging.root.manager.loggerDict does not contain the root logger. To get the root logger, you can call logging.getLogger() with empty argument:

root_logger = logging.getLogger()
print(root_logger.handlers[0].formatter._fmt)

References
#

Related

Make Python logging Work in GCP
·570 words·3 mins
Python Logging GCP
Why Are Some Logging Messages Missing in Python?
··487 words·3 mins
Python Logging
Duplicate Logging Messages in Python
··265 words·2 mins
Python Logging