Docker Container Debugging: How to Start a Stopped Container with a Different Command

Problem Context and Challenges

In Docker containerized application development, situations frequently arise where containers fail to start normally due to crashes in their default commands. Under these circumstances, the conventional docker exec command becomes unusable since it only functions with running containers. Users require methods to initiate stopped containers with different commands (such as shells) to inspect internal container states.

Core Solution: Committing Container State and Specifying New Entry Points

Using the docker commit command to save the filesystem state of a stopped container as a new image, followed by docker run --entrypoint to designate a new entry point command, represents the most reliable and recommended approach.

Detailed Operational Steps

First, identify the target container's ID:

docker ps -a

This command lists all containers (including stopped ones), from which the target CONTAINER_ID can be obtained.

Next, commit the container state to create a new image:

docker commit $CONTAINER_ID user/test_image

The docker commit command preserves the container's current filesystem state as a new Docker image. Here, user/test_image serves as a customizable image name that can be modified as needed.

Finally, launch a container using the new image with a shell as the entry point:

docker run -ti --entrypoint=sh user/test_image

The --entrypoint parameter overrides the image's default entry point, while the -ti options ensure the container runs in interactive mode with an allocated pseudo-terminal, enabling direct shell operations.

Technical Principle Analysis

The lifecycle of Docker containers is intrinsically linked to their main processes. When the main process terminates, the container enters a stopped state. At this point, the container's filesystem layers remain intact, but the process environment is destroyed. docker commit effectively creates a new image layer containing a snapshot of the filesystem at the time the container stopped. By specifying a new entry point, we essentially launch a fresh container instance based on this snapshot.

The primary advantage of this method lies in its preservation of the container's filesystem state, including any files created or modified during container operation. This proves particularly valuable for debugging application crashes by examining the post-failure environment.

Considerations and Limitations

While the described method effectively addresses filesystem state inspection, several important considerations apply:

• Environment Variables Not Inherited: Newly started containers do not inherit environment variable configurations from the original container and require reconfiguration
• Network Configuration Reset: Network-related settings such as port mappings and network modes need to be respecified
• Explicit Volume Mounting Required: Any data volumes must be remounted in the docker run command
• Process State Loss: Dynamic information including running process states and memory contents cannot be recovered

Alternative Approach Comparison

Beyond the primary solution, alternative methods exist with respective limitations:

Direct Container Configuration Modification: Altering the startup command by editing the Path parameter in /var/lib/docker/containers/<container_id>/config.json. This approach requires direct access to the Docker host filesystem and necessitates Docker service restart, potentially affecting other running containers.

Using Data Volume Containers: For scenarios requiring persistent data, dedicated data volume containers can be created, with data accessed via --volumes-from in temporary containers. This method suits data management better than debugging.

Practical Application Scenarios

This technique proves particularly useful in the following contexts:

• Application Crash Debugging: Inspecting logs and configuration files when applications crash due to configuration errors or dependency issues
• Data Recovery: Extracting critical data files from crashed containers
• Environment Verification: Validating filesystem layouts and permission settings within containers
• CI/CD Pipeline Debugging: Diagnosing build failure causes in continuous integration environments

Best Practice Recommendations

For more effective utilization of this debugging technique, consider:

• Incorporating appropriate log outputs for critical containers during development to reduce dependency on interactive debugging
• Utilizing the HEALTHCHECK instruction in Dockerfiles for automated container health monitoring
• Implementing comprehensive monitoring and log collection solutions in production environments
• Pre-building base images with common debugging tools for frequently debugged scenarios

Mastering this container debugging methodology enables developers to more efficiently diagnose and resolve various issues in Docker environments, enhancing the reliability and maintainability of containerized applications.