Logging setup for Pytest

When you do code testing using pytest, pytest can do sophisticated changes to your logging. This is a very powerful feature, but carries some subtlety.

log_cli and log_cli_level

#

Suppose that we have the following test case:

import logging
logger = logging.getLogger()

def test_something():
    logger.debug("Debug message")
    logger.info("Info message")
    logger.warning("Warn message")
    logger.error("Error message")

    assert True

[log_cli][pytest-option-log-cli] can control whether to enable live log on cli. Note that this option does not have a command line equivalent (i.e., it can only be set in config file) We can use option log_cli_level (or --log-cli-level on cli) to control the logging printed on command line.

If we have the following pytest config (use pytest.ini):

[pytest]
log_cli = true
log_cli_level = WARN

If you run pytest, you will see that only WARN and Error logging is shown on cli. If you have log_cli_level = INFO, you see INFO, WARN and ERROR logging. So under the hood, it seems log_cli_level will also set your root logger level¹.

log_cli_level vs log_level

#

There is another option log_level that controls the log level of root logger. Based on experiments, log_level has priority over log_cli_level in setting the root logger. This is easy to verify. We change the test case a bit:

import logging
logger = logging.getLogger()

def test_something():
    foo = 123
    logger.debug("Debug message")
    logger.info("Info message %", foo)
    logger.warning("Warn message")
    logger.error("Error message")

    assert True

The INFO logging is ill-formatted and will error out if it is run. We use the following pytest config:

[pytest]
log_cli = true
log_cli_level = WARN
log_level = INFO

If you run pytest, you will see format errors:

ValueError: incomplete format

You will not see error if you use this config (logging is only initialized starting with WARN level):

[pytest]
log_cli = true
log_cli_level = INFO
log_level = WARN

caplog fixture

#

There is another level of complication. If you use caplog.set_level() fixture inside test, it can also change the logging level. Now we change our test case to the following:

import logging
logger = logging.getLogger()

def test_something(caplog):
    caplog.set_level(logging.WARN)

    foo = 123
    logger.debug("Debug message")
    logger.info("Info message %", foo)
    logger.warning("Warn message")
    logger.error("Error message")

    assert True

On the command line, if you run pytest --log-cli-level=INFO, the test is still going to pass: caplog.set_level has silently override the logging level you set via --log-cli-level. The effect of caplog.set_level make is only in this test only.