Diagnosis and Solution for Nginx Upstream Prematurely Closed Connection Error

Problem Background and Error Analysis

In web service architectures based on Nginx and Node.js, gateway timeout errors frequently occur when processing large file updates or long-running computation requests. The specific error message from the logs shows: upstream prematurely closed connection while reading response header from upstream. This error indicates that the connection was actively closed by the upstream server while Nginx was reading the response header.

Root Cause Diagnosis

According to the best answer analysis, the core issue of this error lies in the upstream server (Node.js application) actively terminating connections during time-consuming request processing. When handling large data updates (such as MBTiles file updates) that take 3-4 minutes, the Node.js server may prematurely close the connection with Nginx for various reasons.

Common reasons for upstream server connection closure include:

• Node.js application process timeout or crash
• Child process execution timeout (such as shell scripts executed via spawn)
• Process termination due to insufficient memory
• Application-level timeout settings being too short

Nginx Configuration Optimization

Although the root cause is in the upstream server, optimizing Nginx configuration can mitigate this problem:

Timeout Parameter Adjustment

Increase proxy timeout settings to provide sufficient processing time for the upstream server:

location / {
    proxy_read_timeout 300s;
    proxy_connect_timeout 75s;
    proxy_send_timeout 300s;
    proxy_pass http://127.0.0.1:7777;
}

proxy_read_timeout controls how long Nginx waits for responses from the upstream server and should be set to a sufficiently large value for long-running operations. proxy_connect_timeout ensures adequate time during the connection establishment phase.

Buffer Optimization Configuration

Optimize buffer settings to improve large file transfer efficiency:

proxy_buffers 8 64k;
proxy_buffer_size 128k;
large_client_header_buffers 8 32k;

These settings ensure Nginx has sufficient buffers to handle large response headers and bodies.

Upstream Server Optimization Recommendations

Addressing the root cause requires optimizing the Node.js application:

Process Management and Monitoring

Add process status monitoring and error handling in the Node.js code:

command.on('close', function(code) {
    if (code === 0) {
        logger.info("updating mbtiles successful for " + earthquake);
        tilelive_reload_and_switch_source(earthquake);
        res.send("Completed updating!");
    } else {
        logger.error("Error occurred while updating " + earthquake);
        res.status(500);
        res.send("Error occurred while updating " + earthquake);
    }
});

Adding Timeout Control

Set reasonable execution timeouts at the application level to prevent unlimited process execution:

const timeout = setTimeout(() => {
    command.kill();
    res.status(504);
    res.send("Request timeout");
}, 300000); // 5-minute timeout

Comprehensive Solution Approach

Resolving the upstream prematurely closed connection error requires coordinated optimization of both Nginx and upstream servers:

1. First, diagnose upstream server stability to ensure the Node.js application can run stably for extended periods
2. Adjust Nginx timeout parameters to provide adequate processing time for upstream servers
3. Optimize buffer settings to improve large file transfer efficiency
4. Implement comprehensive error handling and timeout control in upstream applications
5. Establish monitoring mechanisms to promptly detect and handle connection anomalies

Through this layered optimization approach, connection interruption issues during large file processing can be effectively resolved, enhancing system stability and reliability.