Reading Files with UTF-8 Encoding

From Text to Stored Bytes

A Python string can contain characters such as é, €, 中, and こんにちは. A file system stores file data as bytes, so Python must translate the Unicode characters in the string into bytes before storing them. This translation is called encoding. UTF-8 performs that translation automatically when you open the file with encoding='utf-8'.

The UTF-8 Conversion Step

When f.write() is called on a file opened with encoding='utf-8', Python converts each Unicode character into one or more bytes according to the UTF-8 encoding scheme. Characters do not all require the same number of bytes. For example, the Euro sign € has Unicode code point U+20AC and is represented in UTF-8 by the three bytes 0xE2, 0x82, and 0xAC. Those bytes are stored sequentially in the file.

Opening a Text File

import io with io.open("messages.txt", mode="wt", encoding="utf-8") as f: f.write("Français\n") f.write("こんにちは\n") f.write("€")

The file contains:
Français
こんにちは
€

Encoding Parameter Control

The encoding='utf-8' argument tells io.open how Unicode text should be translated before it is written. Without an explicitly selected encoding, the behavior may not be consistent across systems. Specifying UTF-8 makes the intended conversion clear and helps preserve non-ASCII characters so that other programs can read them correctly when they understand UTF-8.

Unicode String Literals

In Python 3, strings are Unicode by default. A u prefix, as in u"Français", is optional but can clarify that the value is intended as Unicode text. The prefix was more significant in Python 2, where regular strings were byte strings by default. In Python 3, these two forms express the same Unicode-string intent.

Safe File Cleanup

A file should be closed after writing. You can close it explicitly, but a with statement is the safer and cleaner approach. The context manager automatically closes the file when the block ends, including when an error occurs during writing.

When Encodings Disagree

Encoding problems become easier to investigate when you trace the complete path: Unicode text in the Python string, the encoding selected when the file is written, the bytes stored in the file, and the encoding understood by the program or editor that reads it. If the intended UTF-8 encoding is not used consistently, non-ASCII characters may not appear correctly in the output.

• Opening a text file without explicitly specifying UTF-8

  The intended text encoding is not stated explicitly, so behavior may not be consistent across systems.

  Fix: Use io.open with mode='wt' and encoding='utf-8'.

• Manually converting every character to bytes before writing

  A text file opened with encoding='utf-8' performs the Unicode-to-byte conversion automatically.

  Fix: Write Unicode strings directly to the UTF-8 text file.

• Forgetting to close the file

  File cleanup is not completed reliably.

  Fix: Use a with statement so the file closes automatically, even if an error occurs during writing.

• Assuming the u prefix is required in Python 3

  Python 3 strings are Unicode by default.

  Fix: Use the u prefix only when it improves clarity or compatibility.

Practice the Trace

Key Takeaways

1. Use io.open with mode='wt' and encoding='utf-8' to write Unicode text to a file.
2. UTF-8 converts each Unicode character into one or more bytes, which are stored sequentially.
3. The encoding parameter selects the text-to-byte conversion used during f.write().
4. Python 3 strings are Unicode by default, so the u prefix is optional.
5. Use a with statement to close the file automatically and consistently.