Preprocessing Text Data for Analysis
Turning free text into categories is a preprocessing task.
A Survey With a Text Box
Imagine you are building a survey and want to know each respondent's favorite color. You could show a list of options and let people pick one. That approach keeps the answers tidy: every response is exactly one of the values you offered. Now change one detail. Instead of a menu, you give people an empty text box. Suddenly the answers are no longer guaranteed to match anything you planned for, and the data becomes much harder to analyze.
This article follows one idea from start to finish: how a messy typed answer becomes a clean category that a program can count and compare. You will trace the decisions a small function makes on real-looking responses, then look at a related problem, structured strings such as dates, where the text carries a pattern but its meaning depends on where it came from.
Fixed Lists Versus Free Strings
A free string is text that a person typed without being constrained to a list. Some free strings can still be semantically mapped to categories, meaning that a person can read each one and decide which category it belongs to.
The source describes a color-naming survey whose typed answers included plain names such as black and blue, misspellings, alternate spellings such as gray and grey, and more specific names such as midnight blue. Some answers were far stranger. The answer velociraptor cloaka is very hard to connect to any color automatically, or at all. A fixed-list question never produces answers like these, because every response has to be one of the options offered.
| Property | Fixed list | Free text |
|---|---|---|
| Distinct values | Exactly the options you offered | Potentially unlimited spellings and phrasings |
| Counting responses | Direct, because each value is known in advance | Requires cleaning or mapping before counting is meaningful |
| Typos and variants | Cannot occur | Common: misspellings, alternate spellings, extra words |
| Unexpected answers | Cannot occur | Expected in any real dataset |
| Cost to produce clean data | Low, because the data is clean from the start | High, often manual and time-consuming |
How the two kinds of answers differ for analysis
The contrast is about how many distinct values exist. A fixed list gives you a short, known set, so counting responses is straightforward. A text box gives you an open-ended set, so the same intended answer can appear under many different strings, and each one has to be interpreted before it can be counted with the others.
Mapping Strings onto Categories
Turning free text into categories is a preprocessing task. The raw column of typed responses goes in, and a clean categorical column comes out. The practical advice is to encode this kind of data as a categorical variable. You choose the categories in one of two ways. One option is to look at the most common entries and build categories around them. The other is to define categories that capture responses in a way that fits your application, even if no respondent used those exact words.
A reasonable result for the color survey might include categories for standard colors, a category called multicolored for answers describing more than one color, such as a striped green and red, and an other category for anything that cannot be assigned to the rest. Notice that two different strings can land in the same category, and that a strange response with no reasonable match still gets a home in other.
Read the diagram as a lookup table. Each typed response is a key, and the category assigned to it is the value. The category assignments shown are one reasonable choice for illustration. A different application might decide that midnight blue deserves its own category, which is a judgment about the purpose of the analysis rather than a fact about colors.
Tracing a Category Function
The following small program shows the mapping idea in code. It uses a dictionary for known standard colors, cleans each response by removing surrounding spaces and lowercasing it, and falls back to the multicolored and other categories when no standard match exists. Read it once before you trace it.
- The dictionary maps known spellings to standard category names, so grey and gray both become gray.
strip()removes surrounding spaces andlower()makes the comparison case-insensitive.- The multicolored check is deliberately simple and is a teaching device, not a complete survey solution.
- The final return means no input can cause a failure: anything unmatched becomes other.
The order of the checks matters. The dictionary lookup comes first, so a response that is an exact standard color returns immediately. The multicolored check comes second, and it runs only when the dictionary lookup failed. Once a branch returns, nothing after it runs, so the first matching rule wins even if a later rule would also match.
What do you think happens?
Before reading on: what does categorize_color('midnight blue') return?
Reveal answer
other
The cleaned text is 'midnight blue'. It is not a key in standard_colors, and it contains no ' and ', so the function skips both checks and reaches the final return, which gives 'other'. The function does not return the raw text.
Stepping Through Two Responses
Tracing grey and velociraptor cloaka
Trace categorize_color on the response ' Grey ' and on the response 'velociraptor cloaka'. Track the state after each step.
- 1Clean the first response
strip()removes the surrounding spaces andlower()turns 'Grey' into 'grey'. The cleaned value is now 'grey'. - 2Dictionary check
'grey' is a key in
standard_colors, so the if branch runs and returns the mapped value 'gray'. Nothing else runs. - 3Clean the second response
The text is already lowercase with no outer spaces, so the cleaned value is 'velociraptor cloaka'.
- 4Dictionary check fails
The string is not a key in
standard_colors, so the first if is skipped. - 5Multicolored check fails
The string contains no ' and ', so the second if is skipped.
- 6Final return
Control reaches return 'other'. The function never raises an error on a strange answer.
' Grey ' produces gray, and 'velociraptor cloaka' produces other.
gray
multicolored
other- The leading and trailing spaces in the first call are removed before the lookup.
- The second call matches the multicolored rule because the text contains ' and '.
- The third call falls through every rule and lands in other.
Structured Strings Like Dates and Addresses
Not every manually entered value fits a category. Some values have no fixed list of answers, yet still follow an underlying structure. Addresses, names of places or people, dates, telephone numbers, and other identifiers all belong to this group. The source calls such data structured string data, because the text carries a pattern even though the set of possible values is effectively unlimited. Structured data in this sense is data that has a predefined format or organization.
Structure does not make the data easy. Structured string data is often very hard to parse, and the right treatment depends heavily on context and domain. Consider the string 03/04/2024 as a generated illustration. The pattern looks like two numbers and a four-digit year. The first two numbers could be read as a month and a day in one convention, or as a day and a month in another. The text alone does not settle the question.
| Source convention | Reading of 03/04/2024 | Date meaning |
|---|---|---|
| Month first | month 03, day 04 | March 4, 2024 |
| Day first | day 03, month 04 | 3 April 2024 |
The same string under two conventions. Only the source system can say which one is correct.
Mistakes When Mapping Free Text
Common mistake Skipping the cleaning step before matching
What it looks like' Grey ' is looked up without strip() and lower(), so it misses the 'grey' key and falls through to other.
Why it failsSurrounding spaces and capital letters are not part of the category, so the lookup compares the wrong strings.
Do this insteadClean every response with strip() and lower() before any dictionary or keyword check.
Common mistake Treating other as a failure
What it looks likeA row that maps to other is dropped from the analysis.
Why it failsThe other category exists so that every raw response has a valid home, including unexpected ones.
Do this insteadKeep the other rows, and review them periodically to decide whether a new category is warranted.
Common mistake Assuming the rule order does not matter
What it looks likeA rule for multicolored is placed before the dictionary lookup, so an exact standard color is never reached.
Why it failsThe first branch that returns wins, so reordering rules changes which category a response gets.
Do this insteadWrite rules from most specific to most general, and check the order when you add a new rule.
Common mistake Expecting a keyword rule to handle every case, when judgment calls remain for awkward responses
Practice Mapping Your Own Responses
Extend categorize_color so that the string 'midnight blue' maps to a category called blue. Predict the output of categorize_color('Midnight Blue') for your extended version before you run it, then explain which check handles it.
Hints
- Consider adding a keyword check for the word blue, placed after the dictionary lookup.
- Remember that the response is cleaned to lowercase before any check runs.
- A check that looks for the word blue would also catch 'blue sky', which may or may not be what you want.
A respondent types 'Grey and blue'. Predict which category the current function returns, and explain why the first matching rule wins even though a second rule also matches.
Hints
- Ask whether 'grey and blue' is a key in
standard_colors. - Then ask which of the remaining checks runs first.
Key Takeaways
- Free strings are typed answers with no fixed list of options, so they are harder to count and compare than answers chosen from a menu.
- Free strings can be semantically mapped to a limited set of categories, and a catch-all category such as other gives every raw response a valid home.
- Preprocessing usually follows a fixed order: gather raw text, choose categories, assign each response, and place leftovers in other, with much of the work requiring manual judgment.
- A function that cleans text and checks rules in order returns on the first match, so rule order determines the category assigned.
- Structured strings such as dates and addresses have a visible pattern, but their meaning depends on context and source conventions, so the pattern alone is not enough.