NGINX Proxy Loop and File Descriptor Exhaustion: Analyzing worker_connections in Kibana Deployment

NGINX Event Handling Mechanism and Connection Management

NGINX employs an event-driven asynchronous architecture for client connection processing, where the worker_connections parameter defines the maximum number of simultaneous connections each worker process can handle. Typically set to 768 in default configurations, this value suffices for most low-concurrency scenarios. However, when applications require handling substantial concurrent requests, this limit can become a performance bottleneck.

Proxy Loop: The Root Cause of Configuration Error

In the presented case, the user attempted to access a Kibana application deployed at /var/www/kibana-3.1.2 through NGINX proxy. The critical issue in the configuration lies in the location /kibana-3.1.2 block using the proxy_pass http://127.0.0.1; directive. Since NGINX typically listens on port 80 by default, this configuration creates a proxy loop: NGINX receives requests for /kibana-3.1.2 and forwards them to localhost port 80, which is being monitored by the same NGINX instance, resulting in infinite request forwarding.

File Descriptor Exhaustion and System Limits

The direct consequence of a proxy loop is rapid consumption of file descriptors. Each proxy connection requires a file descriptor, and as looped requests continuously create new connections, the system quickly reaches the file descriptor limit set by ulimit. This explains the accept4() failed (24: Too many open files) warnings in the error logs. Simply increasing the worker_connections value (such as to 20000 as suggested in some answers) cannot resolve this issue and may exacerbate file descriptor contention through more connection attempts.

Kibana Application Characteristics and Correct Configuration

Kibana is a pure frontend JavaScript application containing no server-side execution code. Therefore, using proxy_pass for reverse proxying represents unnecessary complexity. The correct configuration should leverage NGINX's static file serving capabilities:

root /var/www/;
location /kibana-3.1.2 {
    try_files $uri $uri/ =404;
}

This configuration sets /var/www/ as the document root and gracefully handles file requests through the try_files directive. When requesting /kibana-3.1.2, NGINX directly serves static files from the /var/www/kibana-3.1.2 directory, completely avoiding proxy overhead and loop risks.

Production Environment Security Considerations

While static file serving addresses basic access issues, security hardening should be considered for public deployments. Recommended measures include:

• Implementing HTTP basic authentication or integrating enterprise single sign-on systems
• Configuring appropriate CORS policies instead of using permissive Access-Control-Allow-Origin *
• Protecting backend Elasticsearch services through proxy_pass to restrict direct access
• Setting rate limiting and request filtering rules

System-Level Optimization Recommendations

For high-concurrency production environments, beyond application configuration, system parameters should be adjusted:

# Temporarily adjust file descriptor limit
ulimit -n 65536

# Permanent modification (in /etc/security/limits.conf)
* soft nofile 65536
* hard nofile 65536

Additionally, set worker_processes and worker_connections appropriately based on server resources and expected load, typically recommending worker_connections values not exceeding 80% of the system's file descriptor limit.

Diagnostic and Troubleshooting Methodology

When encountering connection-related errors, a systematic diagnostic process includes:

1. Examining the complete context of NGINX error logs rather than isolated error messages
2. Monitoring actual connection counts using netstat -an | grep :80 | wc -l
3. Viewing system-level socket statistics via ss -s
4. Verifying whether proxy targets conflict with NGINX listening ports
5. Gradually simplifying configurations, starting testing from minimal viable configurations

Conclusion

NGINX worker_connections errors often represent not simple numerical adjustment issues but manifestations of deeper configuration logic flaws. In static web application deployment scenarios, direct file serving should be prioritized over unnecessary proxy forwarding. Proper architectural understanding combined with system-level optimization is essential for building stable and efficient web service environments.