Logging basics in Python

Add logs in Python processes is key to understand what happens in the code during its execution. In modern architectures, logs are extracted from processes and centralized in log sink like Azure Monitor. Logging is the first step for observability and monitoring.

Python comes with the native logging library, to collect log during processes execution.

Initialisation

In the code, the logger must be initialized. The following code is placed at the beginning of the file code or in one of the first cells of the notebook.

Python file:

import logging

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s # %(message)s')
if len(log.root.handlers) > 0:
    log.root.handlers[0].setFormatter(formatter)

log = logging.getLogger('file name or __name__')
log.setLevel(logging.DEBUG)

Notebook:

import logging

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(name)s # %(message)s')
if len(log.root.handlers) > 0:
    log.root.handlers[0].setFormatter(formatter)

log = logging.getLogger(notebookutils.runtime.context['currentNotebookName'])
log.setLevel(logging.DEBUG)

💡 Note: the notebookutils.runtime.context['currentNotebookName'] instruction is used to get the notebook name in Microsoft Fabric. This instruction is preferable to the __name__ instruction.

Note 2: the log level can be different depending on the environment. For instance, in a production environment, WARNING level or ERROR level is sufficient. In test environments, the level can be set to INFO. The log level can be set as a parameter of a notebook, and changed depending on the environment.

Use

To log data, use the following samples.

log.debug('test debug')
log.info('test info')
log.warning('test warn')
log.error('test error')
log.fatal('test fatal')

💡 The logs should contain contextual information, but no confidential or personal information.

Error logging

Logging errors is crucial to understand the abnormal behaviors in the code, even in production:

try:
    # code
except Exception as error:
    errorMessage = f'Error during the process: {error}'
    log.error(errorMessage, exc_info = sys.exc_info())
    raise Exception(errorMessage)