Network Communication and APIs
Unicode is a universal character encoding standard that assigns a unique code point to every character in every language, enabling programs to handle international text.
Why English-Only Text Breaks Down
A program that handles only English text can work within ASCII, a limited character set covering letters, numbers, and basic punctuation. Real applications also encounter accented letters, Chinese characters, Arabic script, emoji, and text from many other languages. To handle that text correctly, a program needs Unicode and a reliable way to turn Unicode text into bytes for storage or transmission.
Unicode identifies characters. UTF-8 represents those characters as bytes when the text must be stored or transmitted.
Following a Character from Meaning to Bytes
Inside a Python program, Unicode text is handled as characters represented by Unicode code points. A code point is an abstract numerical identifier assigned to a character. That abstract identifier is not yet a sequence of bytes that a file or network connection can store or transmit. UTF-8 performs the conversion from code points to bytes. The bytes can then cross a storage or communication boundary.
Unicode and UTF-8 Compared
Unicode is the abstract character set and standard. It assigns a unique code point to every character in every language and can represent over a million characters. UTF-8 is a variable-width encoding scheme that converts those code points into bytes. The two ideas work together, but they answer different questions: Unicode asks which character is meant, while UTF-8 asks how that character is represented as bytes.
| Question | Unicode | UTF-8 |
|---|---|---|
| What does it define? | Characters and their code points | A byte representation for code points |
| What is its role? | Abstract character set and standard | Concrete encoding method |
| Where is it used? | Inside a program to represent and manipulate text | At file, network, and other storage or transmission boundaries |
Unicode and UTF-8 describe different stages of handling text.
Code Points and Variable-Width Bytes
Unicode code points are commonly written with a U+ prefix followed by hexadecimal digits. The source examples include A at U+0041, é at U+00E9, and 😀 at U+1F600. UTF-8 uses a variable number of bytes: ASCII characters use one byte, European characters use two bytes, and Asian characters use three or four bytes. This lets common English characters remain compact while still supporting international text.
Tracing the Character é
Explain the stages involved when a program handles the character é and later writes it to a file.
Identify the character: The program is working with the character é, not merely with an arbitrary byte.
Use its Unicode identity: Unicode assigns é the code point U+00E9. This code point identifies the character independently of the file or network representation.
Encode at the boundary: When the text is written to a file, UTF-8 converts the Unicode code point into bytes. The source describes European characters such as é as using two bytes in UTF-8.
Store the bytes: The resulting UTF-8 bytes are the concrete data written to storage. The same principle applies when text is sent over the internet.
Unicode supplies the character identity; UTF-8 supplies the bytes needed by the file or communication boundary.
Unicode Strings in Python
In the source material, the u prefix creates a Unicode string. It tells Python to use the Unicode type rather than the regular string type. A Unicode string is handled internally using Unicode code points, allowing the program to represent and manipulate characters from many languages.
What do you think happens?
A program creates u"hello world". Has the program necessarily produced UTF-8 bytes for a file or network connection?
Reveal answer
Answer: No, the u prefix creates Unicode text; UTF-8 is needed when the text crosses a storage or transmission boundary
The source distinguishes Unicode strings used internally in Python from UTF-8 encoding used when reading from or writing to files and networks.
Encoding at Program Boundaries
A useful boundary rule is to keep text as Unicode while the program is working with it, then use UTF-8 when the text leaves the program for a file or network. When text comes back from a file or network, it must be converted back into Unicode so the program can work with the characters. The source describes this conversion as encoding when text leaves the program and decoding when text is read back into it.
When reading or writing files containing non-English text, specify encoding='utf-8' with the open function. This tells Python which conversion to use so Unicode strings are converted to UTF-8 bytes when written and converted back to Unicode when read. The exact syntax can depend on the Python version and file operation, so consult the current Python documentation for the operation you are using.
Mistakes with Text Representations
Treating Unicode and UTF-8 as interchangeable names
Unicode is the abstract character set and assigns code points. UTF-8 is the encoding method that turns those code points into bytes.
Fix:
Describe the text internally as Unicode and identify UTF-8 as the representation used at storage and communication boundaries.Assuming the u prefix creates UTF-8 bytes
The u prefix creates a Unicode string for internal handling. It does not by itself describe the byte representation needed outside the program.
Fix:
Apply UTF-8 encoding when the text must be written to a file or sent over a network.Ignoring encoding for international files
Python needs an encoding choice to convert between Unicode strings and file bytes correctly.
Fix:
Specify encoding='utf-8' for the relevant file operation and check the current documentation for the exact syntax.Expecting every character to use one byte in UTF-8
UTF-8 is variable-width. ASCII characters use one byte, European characters use two bytes, and Asian characters use three or four bytes.
Fix:
Remember that UTF-8 uses different numbers of bytes depending on the character.
Boundary Decisions
For each situation, decide whether the program should primarily work with Unicode text or with UTF-8 bytes: manipulating a message inside Python, writing that message to a file, reading the file back, and sending the message over the internet. Explain what conversion occurs at each boundary.
Hints
- Unicode is used internally for representing and manipulating text.
- UTF-8 converts Unicode code points into bytes for storage and transmission.
- Reading reverses the boundary conversion by turning bytes back into Unicode text.
Choosing the Representation
A program receives international text, changes part of the message, and saves the result.
Receive: The incoming file or network data is represented as bytes, so it must be decoded into Unicode text for the program to handle the characters.
Manipulate: Keep the message as Unicode while changing or examining its characters inside the program.
Save: Encode the Unicode text as UTF-8 when writing it to the file or sending it onward.
Use Unicode for internal text processing and UTF-8 at the input and output boundaries.
Key Takeaways
- Unicode is a universal character set and standard that assigns a code point to each character.
- UTF-8 is a variable-width encoding method that converts Unicode code points into bytes.
- The u prefix creates a Unicode string in the Python model described by the source material.
- Keep text as Unicode while processing it inside the program.
- Use UTF-8 when reading from or writing to files and when sending text across networks.
Key Takeaways
- Unicode identifies characters with code points; UTF-8 encodes those code points as bytes.
- The Python u prefix creates a Unicode string for internal text handling.
- UTF-8 is necessary when text crosses a file or network boundary.
- UTF-8 is variable-width: ASCII characters use one byte, European characters use two bytes, and Asian characters use three or four bytes.
- Specify UTF-8 for file operations involving international text and apply the same boundary principle to network communication.