Python Logger Configuration: Logging to File and stdout Simultaneously

Dual Output Configuration with Python Logging Module

Python's logging module is a powerful tool for handling application logs, offering flexible configuration options to meet various logging requirements. In practical development, it's often necessary to output log messages to both files and standard output for real-time monitoring and long-term storage. This article delves into how to achieve this functionality effectively.

Basic Usage of StreamHandler

StreamHandler is a handler in the logging module designed to output logs to streams such as standard output or standard error. By default, StreamHandler outputs to standard error (stderr). Here's a basic usage example:

import logging
logging.getLogger().addHandler(logging.StreamHandler())

This code retrieves the root logger and adds a StreamHandler, causing all logs output through this logger to appear on the console.

Specifying Output to Standard Output

To direct log output to standard output (stdout) instead of standard error, explicitly specify this when creating the StreamHandler:

import logging
import sys

logging.getLogger().addHandler(logging.StreamHandler(sys.stdout))

By passing sys.stdout as a parameter, StreamHandler directs all log messages to the standard output stream.

Complete Dual Output Configuration

In real-world applications, it's common to configure both file output and standard output while maintaining consistent log formatting. Here's a complete configuration example:

import logging
import sys

# Create formatter
logFormatter = logging.Formatter("%(asctime)s [%(threadName)-12.12s] [%(levelname)-5.5s]  %(message)s")

# Get root logger
rootLogger = logging.getLogger()
rootLogger.setLevel(logging.DEBUG)

# Configure file handler
fileHandler = logging.FileHandler("application.log")
fileHandler.setFormatter(logFormatter)
rootLogger.addHandler(fileHandler)

# Configure console handler
consoleHandler = logging.StreamHandler(sys.stdout)
consoleHandler.setFormatter(logFormatter)
rootLogger.addHandler(consoleHandler)

This configuration produces formatted log output such as: 2012-12-05 16:58:26,618 [MainThread ] [INFO ] my message

Simplified Configuration with basicConfig

Starting from Python 3.3, the logging.basicConfig() function supports the handlers parameter, allowing for more concise configuration of multiple handlers:

import logging

logging.basicConfig(
    level=logging.INFO,
    format="%(asctime)s [%(levelname)s] %(message)s",
    handlers=[
        logging.FileHandler("debug.log"),
        logging.StreamHandler(sys.stdout)
    ]
)

This approach consolidates all configuration into a single function call, resulting in cleaner and more readable code.

Advanced Configuration: Rotating File Handler

For long-running applications, using RotatingFileHandler prevents log files from becoming too large:

import logging
import logging.handlers
import sys

LOG_FILE = "/var/log/application.log"
LOG_LEVEL = logging.DEBUG
LOG_MAX_BYTES = 10485760  # 10MB
LOG_BACKUP_COUNT = 10

def setup_logger():
    logger = logging.getLogger()
    logger.setLevel(LOG_LEVEL)
    
    # Create formatter
    formatter = logging.Formatter(
        "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
        "%d-%m-%Y %H:%M:%S"
    )
    
    # Standard output handler
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setFormatter(formatter)
    logger.addHandler(console_handler)
    
    # File handler (with rotation)
    file_handler = logging.handlers.RotatingFileHandler(
        LOG_FILE,
        maxBytes=LOG_MAX_BYTES,
        backupCount=LOG_BACKUP_COUNT
    )
    file_handler.setFormatter(formatter)
    logger.addHandler(file_handler)
    
    return logger

Practical Considerations

When configuring a logging system, several important aspects require attention:

Log Level Management: Setting appropriate log levels prevents outputting unnecessary debug information. Typically, use INFO level in production environments and DEBUG level in development environments.

Formatter Consistency: Ensure all handlers use the same formatter to maintain consistent log formats across different output channels, facilitating subsequent analysis.

Exception Handling: Incorporate proper exception handling during log initialization to prevent application crashes due to logging configuration failures.

Performance Considerations: For high-performance applications, consider using asynchronous log handlers or adjusting log levels appropriately to minimize the performance impact of I/O operations.

Common Issues and Solutions

In practical usage, you might encounter situations where logs don't output. This is typically due to the following reasons:

Log Level Set Too High: If the logger's level is set to WARNING, DEBUG and INFO level messages won't be output.

Handler Configuration Conflicts: If other code has already configured the logging system, you might need to clear existing handlers first:

logging.root.handlers = []
logging.basicConfig(
    level=logging.INFO,
    handlers=[logging.FileHandler("debug.log"), logging.StreamHandler()]
)

Formatting Issues: Ensure the format string is correct and all referenced LogRecord attributes exist.

Best Practices Summary

Based on years of practical experience, we've compiled the following best practices:

Configure the logging system as early as possible during application startup, avoiding dynamic configuration changes during runtime. Use a unified formatter to ensure consistent formats across all output channels. Set different log levels based on the runtime environment (development, testing, production). For file output, consider using rotating file handlers to prevent disk space exhaustion. Add appropriate log records in critical business logic to facilitate problem troubleshooting and system monitoring.

By properly configuring Python's logging module, you can build a comprehensive logging system that meets both real-time monitoring needs and long-term storage requirements, providing strong support for application stability and problem resolution.