Working with External Data Formats

Why English Text Is Not Enough

A program that handles only English text can work within ASCII, a character standard covering letters, numbers, and basic punctuation. However, ASCII cannot represent many characters used around the world, including accented French letters, Chinese characters, Arabic script, and emoji. To process international text correctly, a program needs Unicode.

From Character to Code Point

Unicode is a universal character encoding standard that assigns a unique code point to every character in every language. A code point is a numerical identifier for a character. Code points are commonly written in hexadecimal notation with a U+ prefix. For example, the letter A is U+0041, the letter é is U+00E9, and the emoji 😀 is U+1F600.

Writing Unicode Strings in Python

In Python, the source material creates a Unicode string with the u prefix, as in u"hello world". The prefix tells Python to use the Unicode type instead of the regular string type. A Unicode string is the form your program uses internally when it needs to represent and manipulate international text.

Encoding at the I/O Boundary

Unicode code points are abstract numbers. A file or internet connection needs concrete bytes, so a Unicode string must be encoded when it leaves the program. UTF-8 is a variable-width encoding scheme that converts Unicode code points into bytes. ASCII characters use 1 byte, European characters use 2 bytes, and Asian characters use 3 or 4 bytes.

The reverse operation happens when external data enters the program. Bytes read from a file or received through a network connection must be decoded back into Unicode text so the program can display, compare, or manipulate the characters. This is why UTF-8 matters at both sides of an I/O boundary: encoding prepares text to leave the program, and decoding restores text after it arrives.

Choosing Encode or Decode

Use UTF-8 encoding when text is written to a file, sent over a network, or otherwise leaves the program. Use decoding when bytes from a file or network connection must become usable Unicode text inside the program. For file operations involving non-English text, the source recommends specifying encoding='utf-8' with the open function so Python can convert correctly while reading or writing.

Mistakes with Text Boundaries

• Treating Unicode and UTF-8 as interchangeable names.

  Unicode is the abstract character set and standard, while UTF-8 is a concrete method for encoding Unicode code points as bytes.

  Fix: Use Unicode to describe the characters and code points your program handles; use UTF-8 to describe the encoding used at an external boundary.

• Assuming ASCII can represent every language.

  ASCII covers English letters, numbers, and basic punctuation but cannot represent many international characters.

  Fix: Use Unicode for text that may contain characters beyond ASCII.

• Forgetting the conversion when text crosses a boundary.

  External storage and transmission require bytes, so Unicode text must be encoded before leaving the program.

  Fix: Encode to UTF-8 for output and decode incoming bytes back into Unicode text.

• Relying on an unspecified file encoding for international text.

  The program may not perform the intended conversion between Unicode text and bytes.

  Fix: Specify encoding='utf-8' for the relevant file operation, following the current Python documentation.

Practice the Direction

Key Takeaways

1. Unicode assigns a unique code point to characters from languages around the world. The u prefix is used in the source material to create a Unicode string in Python. UTF-8 is a variable-width encoding that converts Unicode code points into bytes. Use Unicode while handling text inside the program, then encode to UTF-8 when writing or sending data and decode when reading or receiving it. For file operations involving international text, explicitly specify UTF-8 according to the current Python documentation.