Deep Analysis and Solutions for Python requests SSL Certificate Verification Failure

Fundamentals of SSL/TLS Certificate Verification Mechanism

In HTTPS communication, SSL/TLS certificate verification is a critical component ensuring secure communication. When a client (such as Python's requests library) initiates an HTTPS request to a server, it performs a complete certificate verification process. This includes validating certificate authenticity, checking certificate chain completeness, confirming whether certificates are issued by trusted Certificate Authorities (CAs), and more. Python's requests library relies on the OpenSSL library for SSL/TLS functionality, so its certificate verification behavior follows OpenSSL specifications.

Analysis of Incomplete Certificate Chain Issues

In the provided case, accessing https://hcaidcs.phe.org.uk results in [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777) error. Analysis through SSLLabs report reveals that this server has an "incomplete certificate chain" issue. This means the server does not send the complete intermediate certificate chain during the handshake process, preventing the client from building a complete trust path from leaf certificate to root certificate.

A complete certificate chain typically includes: leaf certificate (server certificate) → intermediate CA certificate → root CA certificate. When the server only sends the leaf certificate, the client needs to obtain intermediate certificates independently to construct a complete verification chain. OpenSSL requires that trust stores contain only CA certificates (including intermediate and root CAs), and cannot directly use server certificates as trust anchors.

Building Custom Trust Store Solutions

To address incomplete certificate chain issues, manually building a trust store file containing missing certificates is necessary. Specific steps include:

1. Obtain PEM format content for missing intermediate CA certificates. In this case, DigiCert SHA2 High Assurance Server CA certificate is required.
2. Obtain PEM format content for root CA certificates. DigiCert High Assurance EV Root CA certificate is needed in this case.
3. Create my_trust_store.pem file and merge the above certificate contents.
4. Specify custom trust store in requests:

import requests
response = requests.get("https://hcaidcs.phe.org.uk/WebPages/GeneralHomePage.aspx", 
                        verify='my_trust_store.pem')

The core principle of this method is extending the client's trust store to include all necessary CA certificates for verification. Importantly, server certificates (leaf certificates) should not be added to the trust store, as OpenSSL only accepts CA certificates as trust anchors.

Common Errors and Considerations

Developers often make the following mistakes when handling such issues:

• Directly adding server certificates to trust stores, violating OpenSSL's trust model design principles.
• Using incomplete certificate chain files lacking necessary intermediate certificates.
• Incorrect PEM file formatting leading to certificate parsing failures.

The correct approach ensures trust store files contain only CA certificates with proper formatting. Certificate chains can be verified using OpenSSL commands:

openssl verify -CAfile my_trust_store.pem server_cert.pem

Security and Best Practices

While custom trust stores can resolve certificate verification failures, security implications must be carefully considered:

1. Use this method only when fully trusting the target server.
2. Regularly update certificates in trust stores to ensure they are not expired or revoked.
3. In production environments, prioritize contacting server administrators to fix certificate chain configurations.
4. For internal systems or testing environments, consider using verify=False parameter, but recognize this completely disables certificate verification with man-in-the-middle attack risks.

Long-term solutions involve encouraging server-side fixes to certificate chain configurations. This not only resolves current client verification issues but ensures all clients can establish secure connections normally.

Technical Implementation Details

Python's requests library implements SSL/TLS functionality through urllib3 and underlying OpenSSL library. When specifying the verify parameter, requests passes the file path to OpenSSL as a trust store. OpenSSL uses certificates in this store to verify the server-provided certificate chain.

Verification includes: checking certificate signature validity, verifying certificate chain completeness, confirming certificates are not expired, checking hostname matching, etc. When any verification step fails, OpenSSL throws corresponding errors, which requests library encapsulates as SSLError exceptions.

For more complex scenarios requiring dynamic management of multiple trust stores or handling Certificate Revocation Lists (CRLs), consider advanced SSL context configuration:

import ssl
import requests

context = ssl.create_default_context()
context.load_verify_locations(cafile='my_trust_store.pem')
response = requests.get(url, verify=context)

This approach provides finer-grained control but requires deeper SSL/TLS knowledge.