Keywords: Docker | Docker-compose | Exit Code 137 | Container Orchestration | Signal Handling
Abstract: This paper provides an in-depth analysis of the root causes behind Docker-compose exit code 137, focusing on graceful shutdown issues arising from inter-container dependencies in non-OOM scenarios. By examining the mechanisms of --exit-code-from and --abort-on-container-exit flags, and analyzing a concrete case where database containers return SIGKILL signals due to forced termination, it offers practical solutions including increasing timeout periods and optimizing container shutdown sequences. The article also elaborates on signal handling mechanisms and best practices for container lifecycle management.
Problem Background and Phenomenon Analysis
When executing the command docker-compose up --build --exit-code-from combined, although the target container combined's internal script explicitly terminates with exit code 0, the entire docker-compose command returns exit code 137. This phenomenon is particularly common in containerized testing environments and requires a deep understanding of Docker's signal handling mechanisms and inter-container dependencies.
Essential Meaning of Exit Code 137
In Unix/Linux systems, exit code 137 carries specific signal implications: 137 = 128 + 9, where 9 represents the SIGKILL signal. This indicates the container was forcibly terminated rather than exiting normally. Common SIGKILL triggers include:
- Intervention by OOM Killer during system memory shortage
- Container timeout when unresponsive to stop commands
- Cascading effects from forced termination of dependent containers
Docker-compose Working Mechanism Analysis
The --exit-code-from parameter implicitly includes the behavior of --abort-on-container-exit. When the specified container exits, docker-compose automatically stops all associated containers. This mechanism is particularly useful in testing scenarios but requires careful attention to graceful shutdown coordination between containers.
Specific Case Analysis
In the user's scenario, the b-combined container exits normally (exit code 0) after completing tests, but triggers the shutdown process for the entire compose project. At this point, the b-db (MongoDB database) container needs to complete graceful shutdown within a limited time frame.
The critical issue lies in MongoDB's shutdown process potentially requiring significant time to handle unfinished connections and write operations. When docker-compose's default timeout period (typically 10 seconds) expires and the database hasn't fully shut down, the Docker daemon sends a SIGKILL signal to forcibly terminate the container, resulting in exit code 137.
Solutions and Best Practices
Increase Stop Timeout: Use docker-compose up --timeout 600 to provide sufficient time window for container shutdown, ensuring stateful services like databases can complete their graceful shutdown processes.
Optimize Container Shutdown Sequence: Properly configure depends_on relationships in docker-compose.yml to ensure dependent services have adequate cleanup time before main service termination.
Improve Application Signal Handling: Implement comprehensive SIGTERM signal handling logic at the application level to ensure rapid resource release and connection closure upon receiving stop signals.
Monitoring and Debugging Techniques: Obtain detailed shutdown process logs through docker-compose --log-level DEBUG, combined with docker inspect to analyze actual container exit status and timeline.
In-depth Analysis of Signal Handling Mechanism
Docker's shutdown process follows standard signal delivery mechanisms: first sending SIGTERM to allow applications graceful shutdown, and if not completed within specified timeout, sending SIGKILL for forced termination. Understanding this mechanism is crucial for diagnosing and resolving exit code issues.
For database applications, it's recommended to configure appropriate shutdown timeout parameters in Dockerfile or startup scripts, such as MongoDB's shutdownTimeoutSecs, ensuring completion of all necessary operations within docker-compose's stop timeout range.
Conclusion and Recommendations
Exit code 137 issues often stem from improper management of inter-container dependencies rather than simple memory insufficiency. Through reasonable timeout configuration, optimized shutdown sequences, and improved signal handling, docker-compose can return accurate exit codes across various scenarios, providing a reliable foundation for continuous integration and automated testing.