Comprehensive Technical Solutions for Logging All Request and Response Headers in Nginx

Introduction

In web server management and debugging, comprehensive logging of HTTP headers is crucial for monitoring, security auditing, and troubleshooting. Nginx, as a widely used reverse proxy server, does not natively support logging all headers in its standard configuration. Based on community Q&A data and technical documentation, this article systematically explores multiple implementation approaches, with particular focus on the njs module as the officially recommended method.

Core Challenges and Technical Background

Nginx's standard logging module only supports recording specific headers through $http_* variables, which cannot meet the requirement of logging all headers. Particularly in reverse proxy scenarios, there is a need to capture both client request headers and upstream server response headers simultaneously. Answer 2 clearly states that native Nginx lacks this capability and requires extension modules.

Detailed njs Module Solution

njs is Nginx's official JavaScript scripting language extension, available since version 1.9.15 and pre-installed in official Docker images. Compared to third-party Lua modules, njs offers better compatibility and maintainability.

Installation and Basic Configuration

First, load the njs module in nginx.conf:

load_module modules/ngx_http_js_module.so;

njs provides four key header objects: headersIn (request headers as key-value pairs), headersOut (response headers as key-value pairs), rawHeadersIn (raw request headers as arrays), and rawHeadersOut (raw response headers as arrays). Among these, headersIn and headersOut automatically merge duplicate headers.

Access Log Implementation

Create a JavaScript module file /etc/nginx/headers.js:

function headers_json(r) {
  return JSON.stringify(r.headersIn)
}

export default {headers_json};

Import and use it in nginx configuration:

http {
  js_import headers.js;
  js_set $headers_json headers.headers_json;
  
  log_format main escape=json '$remote_addr $headers_json';
  access_log /var/log/nginx/access.log main;
}

Using the escape=json parameter ensures proper JSON escaping. For binary-safe requirements, Base64 encoding can be applied in JavaScript:

function headers_json_base64(r) {
  return JSON.stringify(r.headersIn).toString('base64')
}

Error Log Implementation

Log headers in specific locations via the js_header_filter directive:

function headers_json_log(r) {
    return ngx.log(ngx.WARN, JSON.stringify(r.headersIn))
}

Configure location block:

location /log/ {
  js_header_filter headers.headers_json_log;
  proxy_pass http://backend;
}

Error logs do not undergo escaping, outputting raw JSON directly, which is suitable for debugging scenarios.

Comparison of Alternative Solutions

Lua Module Solution

As described in Answers 3 and 5, OpenResty's Lua module provides similar functionality:

header_filter_by_lua_block {
  local h = ngx.req.get_headers()
  for k, v in pairs(h) do
    ngx.log(ngx.ERR, "Got header "..k..": "..tostring(v)..";")
  end
}

Or set variables via set_by_lua_block for log format usage. The Lua solution is powerful but requires additional OpenResty installation.

Mirror Directive Solution

The mirror module (mentioned in Answer 1) can mirror requests to independent services:

mirror /mirror;
mirror_request_body off;

location /mirror {
    internal;
    proxy_pass http://127.0.0.1:6677;
    proxy_set_header X-Original-URI $request_uri;
}

Combined with nc -kl 6677 > headers.log, this can log request headers. This solution only applies to request headers and generates timeout error logs.

Debug Log Solution

The solution proposed in Answer 4, by setting error_log /var/log/nginx/error.log debug; and using grep "http header", can extract request headers. This method is simple but generates extensive debug information, suitable only for temporary use.

Technical Selection Recommendations

Considering the characteristics of each solution:

1. Production Environment Recommendation: njs module solution, balancing functionality, performance, and maintainability
2. Complex Processing Needs: Lua solution suitable for scenarios requiring complex logic processing
3. Simple Request Monitoring: mirror directive suitable for lightweight solutions needing only request header logging
4. Temporary Debugging: debug log solution suitable for quick verification

All solutions must consider privacy compliance issues, avoiding logging sensitive personal information (as reminded in Answer 5 regarding GDPR compliance requirements).

Performance and Security Considerations

Header logging increases CPU and I/O overhead. Recommendations include:

• Limiting logging frequency or implementing sampling in production environments
• Using asynchronous log writing to avoid blocking
• Sanitizing sensitive headers (e.g., Authorization, Cookie)
• Regularly cleaning log files to prevent disk space exhaustion

Conclusion

Although Nginx does not natively support comprehensive header logging, this functionality can be effectively implemented through extension modules such as njs and Lua. njs, as the official solution, offers balanced usability, performance, and compatibility, making it the preferred choice for most scenarios. Actual deployment should select appropriate solutions based on specific requirements, performance impact, and compliance needs, while establishing corresponding monitoring and maintenance mechanisms.