Using value_counts for Data Inspection
A category with a very small count can be a typo or a stray variant rather than a real group. A small count is a prompt to look more closely, not proof of an error.
Why a Column Can Look Fine
Once a dataset has been loaded, a useful first step is to ask whether a categorical column really holds meaningful categories. Scrolling through a few rows is not enough. A column can look perfectly reasonable in the first few lines and still hide problems that only appear when you count its values.
This matters most when the data was typed in by people, such as users filling out a form on a website. Nobody forced those entries into a fixed list, so two people who mean the same thing may type it in two different ways. The check you will learn here is short: run value_counts, read what it reports, and decide whether the column is ready to be used. The question it answers is simple. Which distinct values are present, and how many rows hold each one?
Counting the Values in One Column
Pandas stores each column of a DataFrame as a Series. The value_counts method, called on a Series, reports each unique value together with its count. You can follow the program line by line and pay close attention to the exact spelling of each entry. The example below uses a small, invented gender column so you can see the whole result.
Male 3
Female 2
male 1
Name: count, dtype: int64pd.Seriesbuilds a single column of values from a Python list.value_counts()returns one line per distinct value, with the number of rows that hold exactly that value.- Male and male appear on separate lines, because the comparison is made on the exact text.
Reading the value_counts result for the gender column
The column holds six entries. Use value_counts to find out how many categories appear and how the rows are spread across them.
- 1Count the lines in the output
There are three lines: Male, Female, and male. That means the column contains three distinct values, even though a person would say there are only two genders.
- 2Read the count on each line
Male holds 3 rows, Female holds 2 rows, and male holds 1 row. The counts add up to 6, which matches the six entries in the column.
- 3Look for a small count that may be a variant
The line with a count of 1 is the suspicious one. Its only difference from the Male line is capitalization, so it is most likely a stray variant of the same category rather than a real third group.
The column has a capitalization problem. Male and male must be treated as one category before the column can be used as a clean set of categories.
Reading Each Line of the Output
Each line of value_counts output is a pair. On the left is a distinct value, and on the right is the number of rows carrying that exact value. Two things can be read from the output at a glance. The first is how many categories exist, which is the number of lines. The second is how the rows are spread across those categories.
A category with a very small count can be a typo or a stray variant rather than a real group. A small count is a prompt to look more closely, not proof of an error.
When One Category Splits in Two
The counts never merge entries for you. In the example above, Male and male are two separate entries, even though a person reading them would take them to mean the same thing. Seeing both lines in the output is the signal that the column has a spelling or capitalization problem that may need preprocessing.
Human-entered data frequently contains variants that mean the same thing. The source gives an example in which some people specified gender as male and some as man, and an analyst may want to represent both inputs with the same category. Differences in spelling and capitalization are the main reason a column can contain more distinct values than its meaning requires.
What do you think happens?
If a column contains the values "NY", "ny", and "New York " (with a trailing space), how many lines will value_counts report, and what will each count be?
Reveal answer
Three lines, each with a count of 1
value_counts compares the exact text. NY, ny, and New York with a trailing space are three different strings, so each becomes its own line, and each holds one row.
Deciding Whether a Column Is Ready
One-hot encoding is a method of representing categorical data, and it is a suitable choice when a column has a fixed set of categories. The output of value_counts is the evidence for that decision. If the listing shows exactly the categories you expect and nothing else, the column is in a good format for one-hot encoding.
The source's example is a gender column that contains exactly two values, Male and Female. Its listing shows only those two categories, so the source concludes that the column is already in a good format for one-hot encoding. Compare that with the earlier listing, which showed three lines for the same meaning. The difference is the entire reason for the check.
Apply the same value_counts check to every categorical column, not only the one you looked at first. A single column is only a starting point, and the habit of checking all of them is what catches problems the first column did not reveal.
Mistakes Beginners Make with value_counts
Common mistake Assuming the column is clean because the first rows look correct
What it looks likeScrolling through a few rows shows Male and Female, but a later row contains male.
Why it failsProblems in human-entered data often appear only in some rows, so scrolling cannot reveal them.
Do this insteadRun value_counts on the whole column and read every line of the output.
Common mistake Expecting value_counts to merge Male and male automatically
What it looks likeThe output lists Male with 3 and male with 1.
Why it failsThe counts follow the exact text, so differences in capitalization produce separate entries.
Do this insteadTreat the separate lines as a signal to preprocess the column so that variants map to one category.
Common mistake Ignoring a small count because it looks harmless
What it looks likeA line with a count of 1 sits beside lines with counts of 3 and 2.
Why it failsA category with a very small count can be a typo or a stray variant rather than a real group.
Common mistake Encoding a column right after counting it, without deciding what the categories mean
Why it failsOne-hot encoding suits a column with a fixed set of categories. A column with extra variants does not yet have that set.
Do this insteadOnly move to one-hot encoding once the listing shows exactly the expected categories and nothing else.
Common mistake Forgetting that a trailing space makes the text different from the same word without it
Practice Checking a Column
Suppose a city column produces this value_counts output: Boston with 4 rows, boston with 2 rows, and Chicago with 3 rows. Is the column ready for one-hot encoding? Explain which lines you would inspect and what you would do before deciding.
Hints
- Count the lines first. How many distinct values appear, and do they match the set of cities you expect?
- Ask whether Boston and boston differ only in capitalization.
- Remember that the check is only complete after you have looked at every categorical column, not just this one.
Key Takeaways
value_countson a pandas Series lists each unique value together with the number of rows that hold exactly that value.- Differences in spelling and capitalization produce separate entries, so a column can have more distinct values than its meaning requires.
- A column holding exactly the expected fixed categories, such as Male and Female, is in a good format for one-hot encoding.
- Run the same check on every categorical column, and clean variants only after confirming they represent the same category.