Working with String Types in Python

From Stored Bytes to Python Text

A string that looks like one character to you may be represented by several bytes on disk. Correct file handling connects those two representations: encoded bytes in storage and abstract Unicode characters in Python memory. The central question is not only what text your program wants to use, but also how that text is encoded while it travels into and out of a file.

Bytes are concrete and depend on an encoding. For example, the character é is represented by the byte sequence C3 A9 in UTF-8, while another encoding can use a different sequence. The Unicode character itself is abstract: é is U+00E9 regardless of its storage representation. When io.open is given encoding="utf-8", it decodes UTF-8 bytes into Unicode characters while reading and encodes Unicode characters into UTF-8 bytes while writing.

Declaring and Opening Unicode Text

When a Python source file contains Unicode characters in a string literal, place the encoding declaration at the top of the file. The comment # encoding=utf-8 tells the interpreter that the source file itself uses UTF-8, so it can parse the characters correctly. The u prefix on a string literal, such as u"...", makes the intention to use a Unicode string explicit.

# encoding=utf-8
import io

with io.open("notes.txt", "rt", encoding="utf-8") as f:
    text = f.read()

message = u"Imagine non-English language here"

The source encoding declaration and the file encoding parameter solve different problems. The declaration tells Python how to read characters written directly in the program's source file. The encoding parameter tells io.open how to translate the file's bytes when the program reads or writes external text. Both matter when a program contains Unicode literals and works with UTF-8 files.

Choosing a File Mode

The open() function takes a filename and a mode. The mode determines whether the program reads, writes, or appends, and it affects what happens to existing data. Opening a file returns a file object, which provides methods such as read(), readline(), and write().

Reading Lines and Finding EOF

A file object keeps track of where the next read will begin. readline() returns one complete line, including that line's newline character. When there is no line left, readline() returns an empty string. That empty string is the end-of-file signal.

f = open("log.txt", "r") while True: line = f.readline() if len(line) == 0: break print(line, end='') f.close()

Suppose the first readline() reads line 1. The pointer then stands at the beginning of line 2, so the next call reads line 2. This continues until the pointer reaches the end. The next call returns an empty string, len(line) becomes zero, and the loop stops. The pointer's movement explains why repeated readline() calls do not repeatedly return the first line.

Writing, Saving, and Closing

Writing follows the same three-stage pattern as reading: open the file, perform the operation, and close the file. write() accepts a string and places it in the file. Unlike print(), write() does not add a newline automatically, so include one in the string when a line break is needed.

# encoding=utf-8
import io

f = io.open("greeting.txt", "wt", encoding="utf-8")
f.write(u"Bonjour, café\n")
f.close()

In this sequence, the Unicode string is passed to io.open, which uses UTF-8 to encode the characters into bytes for the file. The close() call finalizes the changes and releases the file. Skipping close() can mean that data is not saved or that the file remains unavailable to other processes.

import io

with io.open("greeting.txt", "rt", encoding="utf-8") as f:
    text = f.read()

Mistakes That Corrupt Text Handling

• Leaving out the source encoding declaration when the source contains Unicode literals.

  The interpreter may fail to parse the source or misinterpret its characters.

  Fix: Place # encoding=utf-8 at the top of the source file.

• Reading or writing UTF-8 text without specifying the encoding.

  Python is not explicitly told how to translate the file's bytes and the program's Unicode characters.

  Fix: Use io.open with encoding="utf-8" for the Unicode file operation.

• Assuming write() adds a newline.

  write() places the supplied string in the file but does not automatically add a newline.

  Fix: Include the newline character in the string when a line break is required.

• Printing a line returned by readline() with the default print ending.

  The file's newline and print()'s newline create double spacing.

  Fix: Use print(line, end='') when displaying lines returned by readline().

• Treating an empty line as the same as end of file.

  The source's EOF test is based on readline() returning an empty string, checked with len(line) == 0.

  Fix: Use the empty-string result from readline() as the EOF signal.

• Opening an existing file in write mode when its content should be preserved.

  Write mode erases existing file content.

  Fix: Use append mode when existing content must be preserved and new data added.

• Forgetting to close a file.

  Changes may not be saved and the file may remain unavailable to other processes.

  Fix: Call close() or use a with statement.

Practice the Full Pipeline

The Reliable File-Handling Pattern

1. Use # encoding=utf-8 when the source file contains Unicode literals, and use u"" to make Unicode string intent explicit.
2. Use io.open with encoding="utf-8" so Python decodes file bytes into Unicode characters when reading and encodes characters into bytes when writing.
3. Choose r to read, w to write and erase existing content, or a to append while preserving existing content.
4. readline() advances the file pointer one line at a time and returns an empty string at end of file.
5. Close files explicitly with close() or use a with statement so changes are finalized and the file is released.