The World of Python Programming

[Complete Guide to Python’s logging Module] From Basic Configuration to Advanced Usage

Practical Guide to Python
佐川 直弘 | Naohiro Sagawa
By 佐川 直弘 | Naohiro Sagawa (運営者)
※記事内に広告を含みます。Amazonアソシエイトとして適格販売により収入を得ています。
目次

1. What is Python’s logging Module?

The logging module in Python is a standard tool for recording program behavior and error information, which is useful for debugging and monitoring during operations. Unlike using print statements, the logging module provides more features, allowing fine control over log levels, output destinations, and formatting. This helps developers efficiently track program anomalies and system status.

Types of Log Levels and Their Uses

  • DEBUG: Detailed debugging information, mainly used during development.
  • INFO: General operational messages, used to confirm normal behavior.
  • WARNING: Minor issues or warnings that do not affect program execution but indicate potential problems.
  • ERROR: Error messages for situations where some functions fail to work properly.
  • CRITICAL: Severe errors that prevent the entire program from continuing execution.

By using log levels appropriately, you can improve the quality of the information obtained from logs, making debugging and monitoring more efficient.

2. Basic Usage of the logging Module

Let’s explore the basic methods for outputting logs using the logging module.

import logging

# Configure log level and format
logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')

# Output logs at different levels
logging.debug('Debug Info: Detailed diagnostic information')
logging.info('Info Message: Confirming normal operation')
logging.warning('Warning Message: A condition that requires attention')
logging.error('Error Message: An issue has occurred')
logging.critical('Critical Message: System failure')

With logging.basicConfig(), you can set the log output destination (default is standard output), log level, and format. In the example above, since level=logging.DEBUG is set, all logs at DEBUG level and above will be output.

3. Customizing Log Output Destination and Format

You can change the default log output destination and customize the log format. For example, to output logs to a file, use FileHandler.

Writing Logs to a File with Custom Format

import logging

# Configure file handler
file_handler = logging.FileHandler('app.log')
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
file_handler.setFormatter(formatter)

# Configure logger
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logger.addHandler(file_handler)

# Output logs
logger.debug('Debug info written to file')
logger.info('Info message written to file')

In the code above, FileHandler is used to output logs to app.log. The log messages are formatted according to the settings specified in Formatter.

4. Log File Rotation

For long-term operations, log rotation is essential to prevent log files from growing too large. The RotatingFileHandler allows you to control the log file size and the number of backup files.

Example Using RotatingFileHandler

import logging
from logging.handlers import RotatingFileHandler

# Configure rotating file handler
handler = RotatingFileHandler('app.log', maxBytes=5000, backupCount=3)
formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)

# Configure logger
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
logger.addHandler(handler)

# Output logs
for i in range(100):
    logger.debug(f'Rotation test {i}')

In this code, maxBytes is set to 5000 bytes to limit the maximum log file size, and backupCount is set to 3 to retain up to three backup files. When the log file exceeds the specified size, a new file is created, and older files are rotated.

RUNTEQ（ランテック）｜超実戦型エンジニア育成スクール

5. Writing Logs to Different Files Based on Log Levels

To improve log readability and facilitate analysis, logs can be separated into different files based on their severity levels. This can be achieved using custom filters and multiple FileHandler instances.

Example of Writing Logs to Different Files by Log Level

import logging

class ErrorFilter(logging.Filter):
    def filter(self, record):
        return record.levelno == logging.ERROR

# Configure file handler for ERROR level logs
error_handler = logging.FileHandler('error.log')
error_handler.setLevel(logging.ERROR)
error_handler.addFilter(ErrorFilter())
error_handler.setFormatter(formatter)

# Configure logger
logger.addHandler(error_handler)

# Output logs
logger.error('ERROR level log is written to a separate file')

In this example, the ErrorFilter class is created to filter only ERROR level logs, which are then written to error.log. This approach helps separate error logs from other types of logs for easier debugging and monitoring.

6. Best Practices and Considerations

To effectively utilize logging, it is important to follow best practices and be mindful of potential issues.

Using Log Levels Appropriately

  • During development, use the DEBUG level to record detailed information, but switch to INFO or WARNING in production to log only essential information.
  • Excessive logging can lead to performance degradation, so only record necessary information.

Security and Privacy in Logging

  • Avoid logging sensitive information such as personal data. If necessary, implement masking techniques to protect user privacy.
  • Ensure proper file permissions are set for log files to prevent unauthorized access.

7. Conclusion

The logging module is an essential tool for efficiently recording program behavior, aiding in debugging and system monitoring. By properly configuring log levels, output destinations, formatting, and rotation, you can gain deeper insights into your program’s operation, leading to early issue detection and resolution. Follow best practices to implement effective log management in your applications.