Keywords: UTF-8 encoding | character encoding | Unicode
Abstract: This article explores the working principles of UTF-8 encoding, explaining how it supports over a million characters through variable-length encoding of 1 to 4 bytes. It details the encoding structure, including single-byte ASCII compatibility, bit patterns for multi-byte sequences, and the correspondence with Unicode code points. Through technical details and examples, it clarifies how UTF-8 overcomes the 256-character limit to enable efficient encoding of global characters.
Fundamental Principles of UTF-8 Encoding
UTF-8 is a variable-length character encoding scheme that uses 1 to 4 bytes to represent characters from the Unicode character set. This design allows it to efficiently encode a wide range of characters, from ASCII to complex symbols. Unlike fixed-length encodings, UTF-8 dynamically selects the number of bytes based on the Unicode code point value, enabling support for a vast character repertoire while maintaining backward compatibility.
Byte Allocation and Character Coverage
The byte allocation in UTF-8 follows specific rules: the first 128 characters (corresponding to ASCII) require only 1 byte, ensuring full compatibility with existing ASCII systems. The next 1,920 characters use 2-byte encoding, covering extended Latin alphabets, Greek, Cyrillic, Hebrew, Arabic scripts, and combining diacritical marks. Three-byte encoding is used for the remaining characters in the Basic Multilingual Plane (BMP), including most Chinese, Japanese, and Korean (CJK) characters. Four-byte encoding applies to characters in other Unicode planes, such as less common CJK characters, historic scripts, mathematical symbols, and emojis.
Technical Details of Encoding Structure
UTF-8 multi-byte sequences employ specific bit patterns to indicate byte length and valid data bits. In single-byte characters, the highest bit is 0, with the remaining 7 bits storing the ASCII value. For multi-byte characters, the number of consecutive high bits in the first byte indicates the total byte count, followed by a 0, and the remaining bits combine with the 6 valid bits of subsequent bytes to form the Unicode code point. For example, a 4-byte sequence starts with 11110xxx, where xxx represents 3 bits, plus 6 bits from each of the next three bytes, totaling 21 bits. This allows representation of up to 2^21 (approximately 2 million) code points, far exceeding the current Unicode limit of about 1.1 million code points.
Relationship Between UTF-8 and Unicode
UTF-8 is one of the encoding implementations of the Unicode character set, mapping abstract code points to concrete byte sequences. Unicode itself organizes characters into 17 planes, each containing 2^16 code points, totaling 1,114,112 code points. UTF-8 has a higher design limit, theoretically capable of encoding 2^31 code points, but in practice, it is constrained by Unicode specifications to encode only valid code points (e.g., not exceeding 0x10FFFF). This separation ensures that UTF-8 remains flexible and efficient for storage and transmission while aligning with Unicode standards.
Practical Applications and Advantages
The variable-length encoding of UTF-8 makes it widely used in internet and software systems. It saves space (common characters use fewer bytes), avoids byte-order issues (no byte-order mark), and supports incremental decoding. Developers should note the need to escape special characters in code like print("<T>") to prevent HTML parsing errors. Similarly, when discussing the <br> tag, it should be treated as a text object to ensure content integrity.