Understanding Unicode and UTF-8 Encoding
Unicode is a universal character encoding standard that assigns a unique code point to every character in every language, enabling programs to handle international text.
Beyond English Text
A program that handles only English text can work within the ASCII character set, which covers letters, numbers, and basic punctuation. Real applications often encounter accented letters, Chinese characters, Arabic script, emoji, and other characters that ASCII cannot represent. Supporting this wider range requires a way to identify characters consistently and a way to store or transmit them as bytes.
Unicode answers the question, “Which character is this?” UTF-8 answers the question, “Which bytes represent that character for storage or transmission?”
Character Identity and Byte Representation
Unicode is a universal character encoding standard that assigns a unique numerical identifier, called a code point, to every character in every language. It is the abstract character set: it describes the characters a program needs to work with and gives each one an identity. Unicode can represent over a million characters, rather than being limited to the much smaller range covered by ASCII.
UTF-8 is a variable-width encoding scheme. It converts Unicode code points into bytes so text can be stored or transmitted. The number of bytes depends on the character: ASCII characters use 1 byte, European characters use 2 bytes, and Asian characters use 3 to 4 bytes. Unicode and UTF-8 therefore describe different layers of the same text-handling process.
Tracing a Character into UTF-8
Representing an Accented Character
Trace what happens when the character é must be stored or transmitted using UTF-8.
Identify the character: The text contains the character é, which is a European character outside the basic ASCII character set.
Use its Unicode identity: Unicode assigns the character a unique code point. At this stage, the program is working with the identity of the character rather than a storage byte sequence.
Apply UTF-8: UTF-8 converts the Unicode code point into bytes. According to the variable-width scheme, a European character uses 2 bytes.
Store or transmit: Those bytes can be used for storage or transmission. When the text is needed as characters again, the bytes must be interpreted using the appropriate text encoding.
The character remains one character at the Unicode level but is represented by 2 UTF-8 bytes for storage or transmission.
Variable-Width Storage
UTF-8 is called variable-width because characters do not all occupy the same number of bytes. Basic ASCII characters use 1 byte. European characters use 2 bytes. Asian characters use 3 to 4 bytes. This allows common ASCII text to use a compact representation while still supporting characters from many languages.
Unicode Strings in Python
In the Python model described here, the u prefix creates a Unicode string. For example, u"hello world" tells Python to use the Unicode type instead of the regular string type. A regular string without the prefix has type str, while a string with the prefix has type unicode.
str
unicodeThe u prefix identifies the text as Unicode in the Python model covered by this material. UTF-8 is still the representation to apply when that text crosses a file or network boundary as bytes.
Crossing File and Network Boundaries
Unicode strings are used internally in Python, while UTF-8 encoding is used when reading from or writing to files and networks. The practical boundary is important: inside the program, work with text and its Unicode characters; when data must be stored or transmitted, represent that text as UTF-8 bytes. When data comes back from storage or communication, it must be interpreted as text again.
Apply UTF-8 when your program writes text to a file, reads text from a file, sends text across a network, or receives text from a network. Pay particular attention when the data may contain languages or symbols outside ASCII.
Mistakes Beginners Make
Treating Unicode and UTF-8 as interchangeable terms.
Unicode is the abstract character set with code points, while UTF-8 is the encoding scheme that converts those code points into bytes.
Fix:
Use Unicode to discuss character identity and UTF-8 to discuss the byte representation used for storage or transmission.Assuming ASCII can represent all text.
ASCII covers letters, numbers, and basic punctuation, but it cannot represent the full range of characters used across the world's languages.
Fix:
Use Unicode text when the program must handle international characters.Assuming every character uses one UTF-8 byte.
UTF-8 is variable-width: ASCII characters use 1 byte, European characters use 2 bytes, and Asian characters use 3 to 4 bytes.
Fix:
Remember that the byte count depends on the characters in the text.Using a Unicode string internally but ignoring encoding at an external boundary.
Files and networks use stored or transmitted bytes, so text must be encoded for those boundaries.
Fix:
Use UTF-8 encoding when writing, reading, sending, or receiving text through files and networks.
Apply the Boundary Test
For each situation, decide whether you are dealing mainly with Unicode character identity, UTF-8 byte representation, or both: a Python variable containing international text; writing that variable to a file; receiving text from a network; and deciding how many bytes a European character uses.
Hints
- Ask whether the situation is about a character inside the program or bytes crossing a storage or communication boundary.
- Remember that UTF-8 is variable-width.
Checking the Four Situations
Classify the four situations using the Unicode and UTF-8 distinction.
Python variable containing international text: This is primarily Unicode text inside the program.
Writing the variable to a file: This crosses a storage boundary, so UTF-8 is needed to represent the text as bytes.
Receiving text from a network: The program receives transmitted bytes and must interpret them as text.
Counting bytes for a European character: This concerns UTF-8 representation, where a European character uses 2 bytes.
Unicode describes the text the program works with; UTF-8 describes the bytes used when that text is stored, transmitted, or interpreted across a boundary.
Key Takeaways
- Unicode is a universal character set that assigns a unique code point to each character.
- UTF-8 is a variable-width encoding scheme that converts Unicode code points into bytes.
- In the Python model covered here, the u prefix creates a Unicode string.
- ASCII characters use 1 UTF-8 byte, European characters use 2 bytes, and Asian characters use 3 to 4 bytes.
- Use UTF-8 when text crosses file or network boundaries.
Key Takeaways
- Unicode identifies characters across languages through code points.
- UTF-8 converts those Unicode characters into variable-length byte sequences.
- The u prefix creates a Unicode string in the Python model presented here.
- UTF-8 becomes important when text is read from or written to files and when it is sent across networks.
- Do not confuse the character-level representation inside a program with the byte-level representation used outside it.