Elasticsearch Disk Watermark Mechanism: Principles, Troubleshooting and Configuration Optimization

Problem Phenomenon and Log Analysis

In an Elasticsearch 1.4.4 development environment, repeated warnings appear after startup: [cluster.routing.allocation.decider] low disk watermark [15%] exceeded on [node_id] free: 58.6gb[12.6%], replicas will not be assigned to this node. The user checks disk space and finds 56GB available, which seems contradictory but is actually caused by Elasticsearch's disk watermark mechanism.

Disk Watermark Mechanism Principles

Elasticsearch controls shard allocation through three watermark parameters to prevent node crashes due to insufficient disk space:

cluster.routing.allocation.disk.watermark.low: Default 85% (i.e., used space ratio). When a node's disk usage exceeds this threshold, Elasticsearch stops allocating replica shards to that node. Note: This only affects replica shards; primary shards of newly created indices can still be allocated.

cluster.routing.allocation.disk.watermark.high: Default 90%. When exceeded, the system attempts to migrate existing shards from the node to others, affecting all shard types.

cluster.routing.allocation.disk.watermark.flood_stage: Default 95%. At this level, Elasticsearch enforces a read-only block (index.blocks.read_only_allow_delete) on all indices with shards on the node, serving as the last defense against data loss.

Configuration Methods Explained

Watermarks support two setting methods, but they cannot be mixed:

Percentage/Ratio Values: Represent the proportion of used disk space. For example, 0.85 means 85% used. This is the default method but may cause unreasonable restrictions on large-capacity disks (e.g., 85% of 5TB equals 4.25TB used).

Absolute Byte Values: Represent remaining available space. For example, 30gb means the corresponding watermark triggers when available space falls below 30GB. This method is more suitable for large-capacity disk environments.

Key difference: Percentage values are calculated based on used space, while byte values are based on free space. This means the logic is reversed—for byte values, the low watermark should be set higher (e.g., 30gb) and the high watermark lower (e.g., 20gb).

Configuration Examples and Practice

Configure in elasticsearch.yml:

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 5gb
cluster.routing.allocation.disk.watermark.low: 30gb
cluster.routing.allocation.disk.watermark.high: 20gb

Important note: When using byte values, the flood_stage parameter must also be set, otherwise the configuration may not take effect. This is because Elasticsearch requires a complete set of byte-value configurations to ensure consistency.

Temporary Solutions and Considerations

Temporarily disable disk thresholds via API:

curl -XPUT localhost:9200/_cluster/settings -d '{
    "transient" : {
        "cluster.routing.allocation.disk.threshold_enabled" : false
    }
}'

However, this method is only suitable for emergencies or testing environments. Disabling is not recommended in production as it removes important disk protection mechanisms. A better approach is adjusting watermark values based on actual disk capacity.

Troubleshooting Process

When encountering disk watermark warnings, follow these steps:

1. Check actual disk usage: df -h (Linux/Mac) or equivalent system commands
2. Verify Elasticsearch's configured watermark values and units (percentage or bytes)
3. Compare actual free space with watermark thresholds
4. Choose appropriate configuration based on disk capacity: percentage for small disks, byte values for large disks
5. Restart Elasticsearch service after configuration changes

By properly configuring disk watermarks, you can ensure data safety while fully utilizing disk space, avoiding unnecessary shard migrations or allocation restrictions due to improper threshold settings.