Technical Analysis of Handling Restricted HTTP Headers in System.Net.WebRequest

Root Cause of HTTP Header Setting Exceptions

When programming with System.Net.WebRequest, developers often encounter the following exception:

This header must be modified using the appropriate property

This exception typically occurs when attempting to add certain specific HTTP headers via the Headers.Add() method. For example:

webRequest.Headers.Add(HttpRequestHeader.Referer, "http://stackoverflow.com");

The above code throws an exception because Referer is a restricted header and cannot be directly added through the conventional Add() method.

Definition and List of Restricted Headers

The WebHeaderCollection class is accessed through the WebRequest.Headers or WebResponse.Headers properties. The system considers some common headers as restricted; these are either exposed directly by the API (such as Content-Type) or protected by the system and cannot be directly modified. Restricted headers include:

• Accept
• Connection
• Content-Length
• Content-Type
• Date
• Expect
• Host
• If-Modified-Since
• Range
• Referer
• Transfer-Encoding
• User-Agent
• Proxy-Connection

These headers are restricted because they have special significance in the HTTP protocol or may be used internally by the system. Modifying them directly via the Headers.Add() method could compromise protocol consistency or security.

Solution 1: Using Explicit Properties of HttpWebRequest

For restricted headers, the most direct solution is to cast the WebRequest object to HttpWebRequest and use its provided explicit properties for setting. For example:

HttpWebRequest httpWebReq = (HttpWebRequest)WebRequest.Create("http://example.com");
httpWebReq.Referer = "http://stackoverflow.com";
httpWebReq.UserAgent = "Mozilla/5.0";

This method is only applicable to headers that have corresponding properties defined in the HttpWebRequest class. While it covers most common use cases, other approaches may be needed for non-standard or custom headers.

Solution 2: Pre-check and Conditional Addition

Another more general method is to use the WebHeaderCollection.IsRestricted() method to check before adding a header. This method takes the header name as a parameter and returns a boolean indicating whether the header is restricted. For example:

string headerKey = "Referer";
string headerValue = "http://stackoverflow.com";

if (!WebHeaderCollection.IsRestricted(headerKey))
{
    webRequest.Headers.Add(headerKey, headerValue);
}
else
{
    // Use alternative methods for restricted headers
    HttpWebRequest httpReq = (HttpWebRequest)webRequest;
    httpReq.Referer = headerValue;
}

This approach offers greater flexibility, allowing developers to dynamically handle different types of headers at runtime. It is particularly useful for complex scenarios involving mixed headers, including standard, restricted, and custom headers.

Alternative Approach with WebClient

In addition to WebRequest, the WebClient class also provides HTTP request functionality, but its header handling mechanism differs slightly. For example:

WebClient client = new WebClient();
client.Headers.Add("referer", "http://stackoverflow.com");
client.Headers.Add("user-agent", "Mozilla/5.0");

It is important to note that WebClient may have more lenient handling of restricted headers in some cases, but this depends on the specific implementation and .NET version. For critical applications, it is advisable to still adhere to the principles of handling restricted headers.

Best Practices and Summary

When handling HTTP headers, the following best practices should be followed:

1. Always prioritize using explicit properties of HttpWebRequest to set restricted headers.
2. Use WebHeaderCollection.IsRestricted() for pre-checks when dynamically adding headers.
3. Avoid directly modifying system-protected headers unless their impact is clearly understood.
4. For custom headers, ensure their names do not conflict with standard or restricted headers.

By understanding the restricted mechanism of WebHeaderCollection, developers can achieve finer control over HTTP requests, avoid common exceptions, and write more robust network code. In practical development, selecting the appropriate solution based on specific needs is key to achieving efficient network communication.