Reading Files and Iterating Over Lines

Finding Useful Lines

A file often contains more information than a program needs. An email log, for example, may contain thousands of lines, but a program might need only the day of the week from lines that begin with From. Parsing is the process of finding those interesting lines and extracting specific pieces of information from them.

Parsing follows a filter-then-parse pattern. First, check whether a line has the structure you expect. Then, split that line into fields and select the field you need. This order matters because not every line in a file has the same structure.

Filtering with startswith

The startswith() method checks whether a line begins with a particular string. For the email log pattern, checking line.startswith('From ') selects lines that begin with From followed by a space. The space is significant because it helps distinguish the expected line structure from a line that merely contains the word From somewhere else.

for line in file:
    if line.startswith('From '):
        print(line)

The loop reads one line at a time. For each line, the conditional check has two possible results. If the line starts with From followed by a space, the indented code runs. If it does not, the line is skipped and the loop moves to the next line.

Turning Lines into Fields

After a line passes the filter, split() breaks the line into a list of words or fields. Each resulting list element has a position called an index. Indexing the list lets you select one exact field instead of using the entire line.

line = 'From sender@example.com Tue Jan 14:20 2025' words = line.split() day = words[2] print(day)

Combining the Two Steps

The complete pattern reads each line, tests its beginning, splits only matching lines, and extracts the desired field. In this example, the desired field is the day of the week at index 2.

file = open('mail_log.txt')

for line in file:
    if line.startswith('From '):
        words = line.split()
        print(words[2])

Tue
Fri
Mon

Mistakes to Avoid

• Splitting every line before checking its structure

  Files can contain headers, comments, blank lines, or other formats that do not contain the expected fields.

  Fix: Use startswith() first, then split only the lines that match.

• Filtering for From without the trailing space

  This can match lines that begin with the word From but do not have the expected From line structure.

  Fix: Use the expected prefix From followed by a space when that is the structure being parsed.

• Using the wrong list index

  In the described From structure, words[1] is the sender and words[2] is the day.

  Fix: Write down the fields and count their positions starting at index 0.

• Ignoring trailing whitespace

  Trailing whitespace can affect how a line is handled during parsing.

  Fix: Strip trailing whitespace when appropriate before applying the rest of the parsing logic.

Practice the Pattern

Key Takeaways

1. Reading a file in a loop lets the program apply the same checks to one line at a time.
2. Parsing means finding interesting lines and extracting specific information from them.
3. Use startswith() to filter lines before attempting to parse them.
4. Use split() to turn a matching line into a list of fields.
5. Use list indexing to select the required field, counting positions from index 0.