Boolean Indexing and Filtering in pandas
A pandas DataFrame is created with pd.DataFrame().
Why Filter a Table at All
Most real datasets are tables with many rows, and a common task is to pull out only the rows that meet a condition: the people older than a certain age, the orders above a certain value, the records from one city. This article works through a short, complete pandas program that does exactly that. You will build a small table of people and then extract only the rows that satisfy a condition.
The goal is not to memorize a list of functions. The goal is to follow what the code does step by step, so that you can tell where a result matches your expectation and where it does not. As you trace each stage, pay attention to the moment a value you expected to see is missing or different. That is the place to look when a program misbehaves.
Building a Table from a Dictionary
A pandas DataFrame is created with pd.DataFrame(). The source passes it a Python dictionary. Each key in the dictionary becomes a column name, and the list stored under that key supplies the values for that column, from top to bottom. Because each position across the lists forms one row, every list in the dictionary must be the same length.
Read the picture as a column-first structure. Each key is one column, and the list under it is that column's contents. Position 0 across all three lists belongs to the same row, so the first row describes the first person across every column. The table is built from the dictionary's columns, not from its keys read as rows.
Name Age City
0 Asha 25 Pune
1 Ben 34 Leeds
2 Carla 41 Austin
3 Dev 29 Kochi- The dictionary keys Name, Age and City become the column names.
- Each list supplies one column, so position 0 across the lists forms row 0.
- The last line displays the
DataFrame; in a notebook this shows it as a table.
The source's program also imports the display function from IPython.display, which it describes as allowing pretty printing of DataFrames in the Jupyter notebook. The table itself does not depend on that import; pd.DataFrame(data) is what builds it.
Predicting the Rows Before Any Filter
Before the next piece of code runs, look at the table you just built and decide what a filter should do. The next example asks for the rows where the Age column is greater than 30. Make your prediction first, then compare it with the answer.
What do you think happens?
Given the Age column (25, 34, 41, 29), which rows survive the filter Age greater than 30?
Reveal answer
Ben and Carla
Ben is 34 and Carla is 41, both greater than 30. Asha is 25 and Dev is 29, both not greater than 30, so they are dropped.
How the True and False Mask Decides Each Row
Conditional selection uses boolean indexing. The expression data_pandas.Age > 30 is evaluated for every row and produces a series of True or False values, one per row. That series is the mask. Each position in the mask corresponds to one row of the table, so the mask's first value decides the fate of the first row, the second value decides the second row, and so on.
In the diagram, the rows marked True are the ones pandas keeps. Rows 1 and 2 hold ages above 30, so they are kept; rows 0 and 3 are dropped. The mask is not a filtered table. It is only a list of True and False decisions, aligned to the row positions of the original table.
From Condition to Filtered Table
When the mask is placed inside square brackets after the DataFrame, pandas keeps only the rows where the value is True. The source's comment states the intent: select all rows that have an age column greater than 30. The code expresses that intent, so when you check a filter, compare the code with the comment and then compare the displayed rows with your prediction.
Name Age City
1 Ben 34 Leeds
2 Carla 41 Austindata_pandas.Age > 30 builds the True and False mask, one value per row.- The brackets keep only the rows where the mask is True.
- The result is stored in a new variable;
data_pandasitself is unchanged.
data_pandas
# 4 rows: Asha 25, Ben 34, Carla 41, Dev 29All four rows are present in the original table.
data_pandas[data_pandas.Age > 30]
# 2 rows: Ben 34, Carla 41Only the rows with Age above 30 remain in the result.
Worked Example: Filtering by City
Select the rows where City is Kochi
Using the same data_pandas table, return only the rows where the City column equals Kochi.
- 1Read the column
The City column holds Pune, Leeds, Austin, Kochi in rows 0 through 3.
- 2Evaluate the condition
The expression
data_pandas.City == "Kochi" gives False, False, False, True, one value per row. - 3Apply the mask
Placing that mask inside brackets keeps only the True row, which is row 3.
- 4Check the prediction
Before running the code, a learner should predict that only Dev, the row at index 3, survives.
The filtered table contains one row: index 3, Dev, age 29, city Kochi.
Keeping the Original Table Unchanged
Mistakes Beginners Make with Filters
Most errors in this kind of code come from a small set of habits. Each entry below names the mistake, shows what it looks like, and gives the fix.
Common mistake Expecting the original table to shrink after a filter
What it looks likedata_pandas[data_pandas.Age > 30] then printing data_pandas shows all four rows.
Why it failsThe selection produces a new table; the original is not changed unless the result is assigned back.
Do this insteadStore the result in a new variable, such as adults_over_30, or assign it back on purpose.
Common mistake Predicting the rows without reading the mask
What it looks likeGuessing that Asha is kept because she is an adult, when the condition is Age greater than 30.
Why it failsThe filter checks the condition exactly as written, not the meaning you attach to it.
Do this insteadWrite out the True or False value for each row first, then keep only the True rows.
Common mistake Comparing against the wrong value or the wrong column
What it looks likeWriting data_pandas.Age > 30 when you meant data_pandas.Name > 30.
Why it failsThe mask is built from whichever column the condition names, so the selection follows that column.
Do this insteadCompare the code with the comment stating the intent, then check the displayed rows against your prediction.
Common mistake Forgetting the square brackets, so the condition is never applied to the rows
Predict Then Check
Using the data_pandas table with rows Asha 25 Pune, Ben 34 Leeds, Carla 41 Austin, Dev 29 Kochi, write down which rows you expect from data_pandas[data_pandas.Age < 30]. Then write the expression for the rows where City is Pune. Run both and compare each result with your prediction. If a row is missing or unexpected, find the True and False value that row received in the mask.
Hints
- Write out the True or False value for each of the four rows before you run anything.
- Asha is 25 and Dev is 29, so both are below 30.
- Check the mask position: row 0 is Asha, row 3 is Dev.
Key Takeaways
pd.DataFrame(data)builds a table from a dictionary: each key becomes a column and each list supplies that column's values row by row.- A condition such as
data_pandas.Age > 30 yields one True or False per row, forming a mask aligned to the table's rows. - Placing the mask inside square brackets keeps only the True rows, and the result is a new table rather than a change to the original.
- Predicting the filtered rows before running the code and comparing them with the output is the fastest way to find where a result diverges from expectation.