Resolving UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in Python

Problem Background and Error Analysis

During Python data processing, encoding-related errors frequently occur when reading CSV files. A typical error message such as UnicodeDecodeError: 'utf-8' codec can't decode byte 0x96 in position 35: invalid start byte indicates that the actual file encoding does not match the specified UTF-8 encoding.

In-depth Analysis of Encoding Mechanisms

UTF-8 encoding uses variable-length bytes to represent characters, where single-byte characters start with 0, and multi-byte characters have specific starting patterns. The byte 0x96 is represented in binary as 10010110, which does not conform to any valid starting byte pattern in UTF-8 encoding. Valid UTF-8 starting byte patterns include: single-byte (0xxxxxxx), double-byte (110xxxxx), triple-byte (1110xxxx), and quadruple-byte (11110xxx), while the pattern 10010110 of 0x96 can only appear as a continuation byte in multi-byte sequences.

Encoding Characteristics in Windows Environment

In Windows operating systems, many legacy systems and applications default to ANSI encoding, particularly cp1252 (also known as Windows-1252) encoding. This encoding is a superset of ISO-8859-1, containing additional characters such as curly quotes, dashes, etc. The byte 0x96 corresponds to the EN DASH character (–) in cp1252 encoding, which is a common punctuation mark.

Solution Implementation

Based on the understanding of encoding mechanisms, we can resolve this issue by specifying the correct encoding format. Below is the complete solution code:

import pandas as pd

# Solution 1: Using cp1252 encoding (recommended for Windows systems)
try:
    data_frame = pd.read_csv("C:/Users/Admin/Desktop/Python/Past.csv", encoding='cp1252')
    print("File read successfully using cp1252 encoding")
except UnicodeDecodeError as e:
    print(f"cp1252 encoding failed: {e}")

# Solution 2: Using ISO-8859-1 encoding as an alternative
try:
    data_frame = pd.read_csv("C:/Users/Admin/Desktop/Python/Past.csv", encoding='ISO-8859-1')
    print("File read successfully using ISO-8859-1 encoding")
except UnicodeDecodeError as e:
    print(f"ISO-8859-1 encoding failed: {e}")

# Solution 3: Automatic encoding detection (requires chardet library)
try:
    import chardet
    with open("C:/Users/Admin/Desktop/Python/Past.csv", 'rb') as file:
        raw_data = file.read()
        encoding_result = chardet.detect(raw_data)
        detected_encoding = encoding_result['encoding']
    
    data_frame = pd.read_csv("C:/Users/Admin/Desktop/Python/Past.csv", encoding=detected_encoding)
    print(f"File read successfully, automatically detected encoding: {detected_encoding}")
except Exception as e:
    print(f"Automatic encoding detection failed: {e}")

Encoding Detection and Verification Methods

To ensure the accuracy of encoding selection, the following verification steps are recommended:

def validate_encoding(file_path, encoding):
    """Verify if the specified encoding can correctly decode the file"""
    try:
        with open(file_path, 'r', encoding=encoding) as file:
            content = file.read()
        return True, f"Encoding {encoding} validation successful"
    except UnicodeDecodeError as e:
        return False, f"Encoding {encoding} validation failed: {e}"

# Test common encoding formats
test_encodings = ['utf-8', 'cp1252', 'ISO-8859-1', 'latin1']
for encoding in test_encodings:
    success, message = validate_encoding("C:/Users/Admin/Desktop/Python/Past.csv", encoding)
    print(message)

Related Case Analysis and Extensions

Referencing other technical discussions, similar encoding issues frequently appear in different scenarios. For example, in text corpus processing, the file 2020_Article_CancerSurgeryAndCOVID19.txt also encountered the same 0x96 byte decoding error, further demonstrating the prevalence of encoding issues in Windows environments.

Best Practice Recommendations

1. When processing text files in Windows environments, prioritize trying cp1252 encoding
2. For projects requiring high cross-platform compatibility, recommend uniformly using UTF-8 encoding
3. Specify encoding format explicitly during file creation to avoid ambiguity in subsequent processing
4. Use encoding detection tools to assist in determining the encoding format of unknown files

Technical Principle Extensions

Deeply understanding the nature of encoding issues requires mastering the difference between character sets and encoding. Character sets define the collection of characters (such as Unicode), while encoding rules define how these characters are converted into byte sequences. UTF-8, as an implementation of Unicode, has the advantage of backward compatibility with ASCII, but requires correct multi-byte sequences when processing non-ASCII characters.