Complete Guide to Proxy Configuration in Python Requests Module

Basic Proxy Configuration Syntax

In the Python Requests module, the proxies parameter accepts a dictionary object that maps protocol names to the complete URL of proxy servers. The correct syntax format is: {"protocol": "scheme://ip:port"}, where protocols can include http, https, or ftp.

Here is a complete basic configuration example:

import requests

http_proxy = "http://10.10.1.10:3128"
https_proxy = "https://10.10.1.11:1080"
ftp_proxy = "ftp://10.10.1.10:3128"

proxies = {
    "http": http_proxy,
    "https": https_proxy,
    "ftp": ftp_proxy
}

response = requests.get("https://example.com", proxies=proxies)

Environment Variable Configuration

In addition to directly specifying proxy configuration in code, the Requests module also supports automatic proxy detection through environment variables. On Linux and macOS systems, use the following commands to set environment variables:

export HTTP_PROXY="http://10.10.1.10:3128"
export HTTPS_PROXY="https://10.10.1.11:1080"
export FTP_PROXY="ftp://10.10.1.10:3128"

On Windows systems, the corresponding setup commands are:

set HTTP_PROXY=http://10.10.1.10:3128
set HTTPS_PROXY=https://10.10.1.11:1080
set FTP_PROXY=ftp://10.10.1.10:3128

After setting environment variables, all requests made through the Requests module will automatically use the corresponding proxy servers without explicitly specifying the proxies parameter in each request.

Session-Level Proxy Configuration

For scenarios requiring multiple uses of the same proxy configuration, Session objects can be used to persist proxy settings. This approach not only improves code maintainability but also leverages connection pooling for performance optimization.

import requests

# Create session object
session = requests.Session()

# Configure session-level proxies
session.proxies.update({
    "http": "http://10.10.1.10:3128",
    "https": "https://10.10.1.11:1080"
})

# All requests made through this session will automatically use the configured proxies
response1 = session.get("https://api.example.com/data1")
response2 = session.get("https://api.example.com/data2")

Authenticated Proxy Configuration

For proxy servers requiring authentication, username and password information can be included in the proxy URL. The basic syntax format is: scheme://username:password@host:port.

import requests

# Authenticated proxy configuration
proxies = {
    "http": "http://user:pass@10.10.1.10:3128",
    "https": "https://user:pass@10.10.1.11:1080"
}

response = requests.get("https://secure.example.com", proxies=proxies)

If passwords contain special characters (such as @, :, %, etc.), URL encoding is required to avoid parsing errors.

SOCKS Proxy Support

The Requests module also supports SOCKS protocol proxies, but requires installation of additional dependencies. First install SOCKS support:

pip install "requests[socks]"

After installation, SOCKS proxies can be configured:

import requests

proxies = {
    "http": "socks5://user:pass@host:port",
    "https": "socks5://user:pass@host:port"
}

response = requests.get("https://example.com", proxies=proxies)

When using the socks5 scheme, DNS resolution occurs on the client side; if DNS resolution through the proxy is required, the socks5h scheme should be used.

Best Practices for Proxy Configuration

In practical applications, it is recommended to follow these best practices: sensitive information such as usernames and passwords should not be hardcoded in source code but managed through environment variables or secure configuration management systems. For production environments, consider implementing proxy rotation mechanisms to avoid overloading or blocking of individual proxy servers. Additionally, properly handle proxy connection failures by implementing appropriate retry logic and failover mechanisms.

Through proper configuration and use of proxies, the reliability and security of network requests can be significantly improved, particularly in application scenarios requiring bypassing geographical restrictions or protecting client real IP addresses.