Technical Guide to Capturing and Parsing HTTP Traffic with tcpdump

Technical Background of HTTP Traffic Capture

Capturing and analyzing HTTP traffic is a common requirement in network debugging and security analysis. tcpdump, as a powerful command-line network analysis tool, can effectively monitor packet transmissions on network interfaces. However, HTTP data captured using basic commands often mixes TCP header information with unparsed application-layer data, resulting in poor readability.

Analysis of Basic Command Issues

The initial command used by the user, sudo tcpdump -A -s 1492 dst port 80, while capable of capturing TCP traffic targeting port 80, exhibits several clear problems: first, TCP header information is intermingled with application-layer data; second, HTTP message bodies are presented in raw byte form without parsing; finally, there is no clear distinction between request and response headers. Similar issues encountered by users in reference articles further confirm this dilemma—output results show recognizable text like HEAD mixed with gibberish.

Precise Filtering for HTTP GET Requests

For capturing HTTP GET requests, filtering based on TCP payload content can be employed. The GET method in TCP payload is represented by ASCII codes 0x47, 0x45, 0x54, 0x20 (corresponding to "GET "). By calculating the TCP header length and offsetting to the payload start position, precise matching can be achieved:

sudo tcpdump -s 0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420'

Here, -s 0 ensures capturing the full packet, and tcp[12:1] & 0xf0 extracts the high 4 bits of the TCP header length field, which after right-shifting by 2 bits gives the header byte length, thus locating the payload start position.

Specialized Capture for HTTP POST Requests

For POST requests, the method identifier is 0x50, 0x4f, 0x53, 0x54 (corresponding to "POST"). Combined with destination port filtering, the command is:

sudo tcpdump -s 0 -A 'tcp dst port 80 and (tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x504f5354)'

This filtering method effectively excludes non-HTTP traffic, focusing on capturing POST method requests.

Comprehensive HTTP Traffic Monitoring Solution

To capture both request headers, response headers, and message bodies simultaneously, more complex filtering logic is required. The key idea is to identify TCP segments that contain actual data (i.e., packets with non-zero payload length):

tcpdump -A -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

This expression calculates the IP packet total length minus the IP header length and TCP header length; a non-zero result indicates the presence of application-layer data. With the -X option, both hexadecimal and ASCII formats can be displayed:

tcpdump -X -s 0 'tcp port 80 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

In-Depth Technical Principles

The core of these filtering commands lies in a deep understanding of the TCP/IP protocol stack. The 12th byte of the TCP header contains the data offset field (4 bits), indicating the header length (in 4-byte units). Through (tcp[12:1] & 0xf0) >> 2, the actual header byte length is calculated, accurately locating the start of application-layer data. Similarly, the 0th byte of the IP header contains version and header length information, used to compute the starting point of the effective payload in the IP packet.

Practical Recommendations and Considerations

In practical use, it is advisable to adjust filtering conditions based on the specific network environment. For encrypted HTTPS traffic, these methods can only capture encrypted data, requiring additional decryption steps. Moreover, considering performance impacts, using the -s 0 parameter in production environments should be done cautiously, or packet count can be limited via the -c option. Regularly consulting the man tcpdump documentation and online resources like Wireshark's string-matching capture filter generator helps in mastering more advanced filtering techniques.