Loading and Filtering Data With pandas

A spreadsheet holds information in rows and columns, and you can sort and filter it by hand. pandas is a Python library for data wrangling and analysis, and it brings a comparable table into code. The central structure in pandas is the DataFrame. In plain terms, a DataFrame is a table, similar to an Excel spreadsheet, and pandas provides methods for modifying and operating on that table.

This article follows one small working example. You will build a table from a dictionary, trace how that table is organized, see why its columns can hold different kinds of data, and then filter it with a condition. Along the way you will follow the steps of a query so that you can explain its output, and you will learn where an unexpected result is most likely to come from.

Each key in the dictionary is a column name, and each key points to a list of values, one value per person or record. The last line passes that dictionary to pd.DataFrame, which builds the table. The example below uses a small generated table with the same shape as the source's four-person example: three columns named Name, Location, and Age, and four rows.

Trace the construction step by step. The dictionary keys became the three column names. The first item in each list lands in the first row, the second item in the second row, and so on. Row labels 0 through 3 were not written in the dictionary at all; they are assigned to the rows and appear at the left edge of the table. Because each list is read position by position, the values in a row line up across columns, so every row describes one person.

A DataFrame has columns, and each column has a name. Each column holds a sequence of values, and the rows line up across the columns. The row labels on the left identify each row. A row is therefore one person's record, spread across all the columns. When you read the table, you read it in two directions at once: down a column to see every value of one attribute, and across a row to see every attribute of one record.

The source contrasts pandas with NumPy. NumPy requires that all entries in an array be of the same type. pandas relaxes that rule by allowing each column to have a separate type. A column can hold integers, dates, floating-point numbers, or strings. In the table built above, the Age column holds whole numbers, while Name and Location hold text. This is the feature that distinguishes a DataFrame from a grid where every entry must match.

Use the picture of a spreadsheet as a starting point. In a spreadsheet, each column can contain numbers or text, and the program keeps track of that. A NumPy array is closer to a grid where every cell must match the others, while a DataFrame keeps each column as its own typed sequence inside one labeled table. The consequence matters later: when a column's type is different from what you expect, a condition on that column will compare values of that type.

pandas makes it possible to query a table with conditions. Here is a query on the same DataFrame. It asks for every row whose Age is greater than 30. Before you read the result, make a prediction about which rows the query will return.

The query reads in two stages. First, the expression is checked against every row's Age, producing a yes or no answer per row. Second, the rows that answered yes are kept in the result. Because the query is a filter, its output is a subset of the original table, and the source shows the filtered table keeping the original row labels. That detail is the one to watch when you trace results.

When a result differs from what you expected, the source recommends a fixed order of checks: look at the table first, then the condition, then the output. The goal is to find the stage where the divergence begins, rather than changing the code at random. The mistakes below are the most common places where that divergence starts.