Automatic Restart of Unhealthy Docker Containers Based on Healthcheck: Current State, Solutions, and Implementation

Current State of Docker Healthcheck Mechanisms and Restart Policies

In Docker containerized deployment practices, the healthcheck mechanism serves as a critical function for ensuring application availability. When internal applications within containers encounter anomalies, Docker can detect issues through configured healthcheck commands and mark containers as unhealthy. However, in Docker 17.09.0-ce and subsequent versions, a notable limitation exists: once a container is marked unhealthy, the Docker engine does not provide direct automatic restart functionality.

Official Feature Planning and Design Philosophy

According to discussions within the Docker development community, the feature for automatically restarting unhealthy containers was initially proposed in PR #22719 but was temporarily removed in subsequent deliberations. The development team concluded that this functionality should be implemented as an enhancement to the RestartPolicy in future versions. This design decision reflects the Docker team's cautious approach to container lifecycle management—they prefer decoupling health checks from restart policies, allowing users to configure them flexibly based on specific scenarios.

Workaround Using the autoheal Container

In current Docker versions, the most mature solution involves utilizing third-party monitoring tools. Among these, the willfarrell/autoheal container offers an elegant workaround. This container monitors the health status of all containers in real-time by mounting the Docker daemon's Unix socket (/var/run/docker.sock).

A configuration example demonstrates a typical Docker Compose deployment approach:

version: '2'
services:
  autoheal:
    restart: always
    image: willfarrell/autoheal
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Key aspects of this configuration include: setting restart: always to ensure high availability of the monitoring container itself; specifying the monitoring scope via the environment variable AUTOHEAL_CONTAINER_LABEL=all; and mounting the Docker socket to grant container management permissions. After deployment, executing docker-compose up -d initiates the automatic health monitoring system.

Alternative Approach with Custom HEALTHCHECK Commands

Another technical strategy involves implementing self-termination mechanisms within containers. Through carefully designed HEALTHCHECK commands, containers can proactively terminate their processes upon detecting application unavailability, thereby triggering Docker's restart policies.

The following example combines curl health checks with process termination:

HEALTHCHECK --interval=5m --timeout=2m --start-period=45s \
   CMD curl -f --retry 6 --max-time 5 --retry-delay 10 --retry-max-time 60 "http://localhost:8080/health" || bash -c 'kill -s 15 -1 && (sleep 10; kill -s 9 -1)'

The technical principles behind this command warrant detailed analysis: The curl command incorporates built-in retry logic via the --retry 6 parameter. When HTTP requests fail consecutively, the termination command following the || operator executes. kill -s 15 -1 sends a SIGTERM signal to all processes within the container, allowing applications to shut down gracefully. After a 10-second wait, kill -s 9 -1 sends a SIGKILL signal to forcibly terminate any remaining processes. When the container's PID 1 process terminates, the container itself exits, at which point the configured always or unless-stopped restart policy takes effect.

Technical Implementation Details and Considerations

When implementing the above solutions, several technical details require particular attention. First, the behavioral differences of the kill command across various shell environments: in bash, the -1 parameter directs signals to all processes with PIDs greater than 1, ensuring the container's main process is correctly terminated. Second, restart policies must be configured as always or unless-stopped; other policies will not automatically restart containers after they exit.

From a system architecture perspective, the autoheal approach represents an external monitoring model, offering advantages such as decoupling and minimal impact on internal application container logic. Conversely, the custom HEALTHCHECK approach embodies an internal self-management model, benefiting from more direct responses and eliminating the need for additional monitoring components. The choice between these solutions should be based on specific deployment environments and operational requirements.

Future Outlook and Best Practice Recommendations

As container orchestration technologies evolve, the integration of health checks and automatic restarts will become more refined. Currently, while Docker's native functionality lacks direct support, it is advisable for production environments to adopt validated third-party solutions and thoroughly test the reliability of custom HEALTHCHECK commands in testing environments. Regardless of the chosen approach, establishing comprehensive monitoring and alerting mechanisms is essential to promptly detect and address container health anomalies.