Docker Container Log Management: A Comprehensive Guide to Solving Disk Space Exhaustion

Problem Background and Challenges

In containerized deployment environments, Docker container log management is a common yet often overlooked issue. By default, Docker uses the json-file log driver, storing container output in the /var/lib/docker/containers/CONTAINER_ID/CONTAINER_ID-json.log file. While this design is straightforward, it has a significant drawback: log files continue to grow until they exhaust the host machine's disk space. Many users resort to manually deleting these log files, which not only increases operational burden but may also lead to loss of important log data.

Core Solution: Log Rotation Mechanism

Docker version 1.8 introduced the log rotation feature, which is the fundamental solution to log space issues. By adding the --log-opt parameter when starting a container, you can control the maximum size of log files. For example, using --log-opt max-size=50m limits individual log files to no more than 50MB. When a file reaches this limit, Docker automatically creates a new log file, enabling automatic rotation.

This functionality is implemented based on Docker's log driver architecture. The json-file driver now supports multiple configuration options, where max-size specifies the maximum bytes for a single log file, using units like k, m, and g for kilobytes, megabytes, and gigabytes respectively. For instance:

docker run --log-opt max-size=10m --log-opt max-file=3 my-image

In this example, max-file=3 indicates that Docker will retain up to 3 log files. When the first file reaches 10MB, Docker renames it to CONTAINER_ID-json.log.1 and creates a new CONTAINER_ID-json.log file. When the number of files exceeds 3, the oldest log file is automatically deleted.

docker-compose Configuration Method

For users managing multi-container applications with docker-compose, log options can be configured in the docker-compose.yml file. Note that this method is only applicable to docker-compose version 2 and above. A configuration example is as follows:

version: '2'
services:
  webapp:
    image: nginx:latest
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "5"

This configuration approach allows individual log parameters to be set for each service, providing finer control. In actual deployments, different max-size and max-file values can be set based on the log generation rate and importance of various services.

Global Configuration and Advanced Options

In addition to configuring log parameters for individual containers or services, global default values can be set by modifying the Docker daemon configuration file. Add the following content to the /etc/docker/daemon.json file:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

After making changes, the Docker daemon needs to be reloaded. In systemd-based systems, use the systemctl reload docker command. This configuration affects all newly created containers, providing a consistent log management strategy for the entire Docker environment.

Log Driver Selection and Comparison

Docker supports multiple log drivers, each with its characteristics and suitable scenarios. Besides the default json-file driver, the local driver is also a noteworthy option:

{
  "log-driver": "local",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

The local driver stores logs in protobuf format, offering better space efficiency compared to JSON format. This means that under the same file size limits, the local driver can store more log content. However, the drawback of this format is that external log processing tools may not be able to parse it directly, requiring additional conversion steps.

It is important to note that the docker logs command only supports three log drivers: json-file, local, and journald. If other drivers (such as syslog or fluentd) are chosen, the docker logs command cannot be used to view container logs.

Implementation Recommendations and Best Practices

In actual deployments, a hierarchical log management strategy is recommended:

1. Development Environment: Larger max-size values (e.g., 100MB) and more max-file counts (e.g., 10 files) can be used to retain sufficient debugging information.
2. Production Environment: Smaller max-size values (e.g., 10-50MB) and fewer max-file counts (e.g., 3-5 files) should be set to control disk usage.
3. Critical Services: For services generating important audit logs, it is advisable to combine Docker log rotation with external log collection systems (such as ELK Stack) to ensure long-term log preservation and analysis.

Additionally, regular monitoring of disk usage in the /var/lib/docker/containers/ directory is necessary. The following command can be used to check the total size of log files:

du -sh /var/lib/docker/containers/*/*.log

Conclusion

Docker's log rotation feature provides an effective solution for container log management. By appropriately configuring max-size and max-file parameters, the issue of unlimited log file growth leading to disk space problems can be avoided. Whether through command-line parameters, docker-compose configurations, or global daemon.json settings, flexible log management strategies can be implemented. When selecting log drivers, a balance between storage efficiency and tool compatibility must be considered, making appropriate choices based on actual needs. As container technology continues to evolve, best practices for log management are also constantly advancing. Staying informed about Docker's new features will help build more stable and efficient containerized environments.