Keywords: Python | logging | requests | urllib3 | HTTPConnection
Abstract: A comprehensive guide to log all requests, including URLs and parameters, in the Python Requests module by leveraging the logging module and HTTPConnection debug level for debugging purposes such as OAuth, with complete code examples and explanations.
Introduction
When debugging OAuth activities or other network operations in Python, it is often necessary to log all HTTP requests made by the requests module, including URLs and parameters. However, tools like ngrep are insufficient for HTTPS connections. This article, based on the best answer from the Q&A data, provides a comprehensive solution using Python's built-in logging module.
Enabling Basic Logging for Requests
The underlying urllib3 library, which requests uses, logs new connections and URLs through the logging module. To enable this, simply configure the logging system to output debug messages:
import logging
logging.basicConfig(level=logging.DEBUG)
import requests
r = requests.get('http://httpbin.org/get?foo=bar&baz=python')
This will output messages such as:
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): httpbin.org:80
DEBUG:urllib3.connectionpool:http://httpbin.org:80 "GET /get?foo=bar&baz=python HTTP/1.1" 200 366
Capturing HTTP Connection Debug Information
To get more detailed logs, including request and response headers, enable debug logging at the http.client.HTTPConnection level by setting HTTPConnection.debuglevel = 1. However, this outputs to print statements. To redirect this to the logging framework, use a patch function:
import logging
import http.client
httpclient_logger = logging.getLogger("http.client")
def httpclient_logging_patch(level=logging.DEBUG):
def httpclient_log(*args):
httpclient_logger.log(level, " ".join(args))
http.client.print = httpclient_log
http.client.HTTPConnection.debuglevel = 1
httpclient_logging_patch()
r = requests.get('http://httpbin.org/get?foo=bar&baz=python')
This will log detailed send and reply messages to the logger, which can be captured by logging.basicConfig().
Complete Example Code
Here is a complete example that combines both logging levels for maximum information:
import logging
import http.client
import requests
# Enable basic logging
logging.basicConfig(level=logging.DEBUG)
# Patch http.client to use logging
httpclient_logger = logging.getLogger("http.client")
def httpclient_log(*args):
httpclient_logger.debug(" ".join(args))
http.client.print = httpclient_log
http.client.HTTPConnection.debuglevel = 1
# Make a request
response = requests.get('http://httpbin.org/get')
print(f"Response status: {response.status_code}")
Conclusion
By leveraging Python's logging module and setting HTTPConnection.debuglevel, developers can efficiently log all requests from the requests module, aiding in debugging complex network interactions like OAuth. This method ensures that even HTTPS connections are logged without the need for external tools.