Categorical Variables in Machine Learning

Imagine you are building a survey and want to know each respondent's favorite color. One design offers a list of options and lets people pick one. That approach keeps the answers tidy, because every response is exactly one of the values you offered. Now change one detail. Instead of a menu, you give people an empty text box. The answers are no longer guaranteed to match anything you planned for, and the data becomes much harder to analyze.

This is the situation this article addresses. A free string is text that a person typed without being constrained to a list. Some free strings can still be mapped onto categories, meaning a person can read each response and decide which category it belongs to. Other free strings have an underlying structure, such as an address or a date, but no fixed list of possible answers. Both situations appear constantly in real datasets, so recognizing them is a practical skill.

Turning free text into categories is a preprocessing task. The raw column of typed responses goes in, and a clean categorical column comes out. The work usually happens in a fixed order. First you gather the raw text and look at which entries appear most often. Then you decide on a set of categories, either by promoting the common entries or by designing categories that serve your purpose. Each response is assigned to a category, and anything that does not fit is placed in a catch-all category.

Trace a single raw answer through this diagram. The response 'velociraptor cloaka' enters as raw text. It does not match any of the categories you defined, so it skips the assignment to a standard color and lands in other. Every response leaves the pipeline with some category. That guarantee is what makes the finished column usable, even though the raw input was unpredictable.

A free string is hard to analyze because the set of possible values is effectively open. A fixed-list answer has a known, small set of values, so counting how many respondents chose each option is direct. A free-text answer can contain misspellings, alternate spellings, extra words, and phrasings nobody anticipated. The same underlying answer can therefore show up under many different strings, and a model or a simple count will treat each string as a separate value unless someone intervenes.

The source's color survey shows the range. Typed answers included plain color names such as black and blue, misspellings, alternate spellings such as gray and grey, and more specific names such as midnight blue. Some answers were far stranger, such as velociraptor cloaka, which is very hard to connect to any color automatically, or at all.

You choose categories in one of two ways. One option is to look at the most common entries and build categories around them. The other is to define categories that capture responses in a way that fits your application, even if no respondent used those exact words. A reasonable result for the color survey might include categories for standard colors, a category called multicolored for answers describing more than one color, such as a striped green and red, and an other category for anything that cannot be assigned to the rest.

The catch-all category is what keeps the scheme complete. Without it, every unusual answer would force you to either invent a new category on the spot or drop the row. With it, the mapping is total: each response, however strange, gets a home. The cost is that other can become a large bucket, so checking how many responses land there is a useful habit when you review a mapping.

Read a mapping as a lookup table. Each typed response is a key, and the category assigned to it is the value. Two different strings can land in the same category. A strange response with no reasonable match still gets a home in other. The assignments below are one reasonable choice for illustration; a different application might decide that midnight blue deserves its own category.

Notice that 'grey' and 'GRAY' are different raw strings, yet both map to the single label gray. This many-to-one property is the whole point of the exercise. It turns a column with dozens of spellings into a column with a handful of values that a model or a count can treat consistently.

The following small program shows the mapping idea in code. It uses a dictionary for known standard colors, cleans each response by removing surrounding spaces and lowercasing it, and falls back to the multicolored and other categories when no standard match exists. Its rule for multicolored is deliberately simple. It is a teaching device, not a complete solution for real survey data.

Trace the function on one response at a time. The text 'grey' is cleaned to 'grey', which is found in the dictionary, so the function returns 'gray' and nothing else runs. The text 'velociraptor cloaka' is cleaned to itself. It is not in the dictionary and contains no ' and ', so the function falls through to the final return and produces 'other'. The function never fails on a strange answer, which is exactly the property you want when input is unpredictable.

Not every manually entered value fits a category. Some values have no fixed list of answers yet still follow an underlying structure. Addresses, names of places or people, dates, telephone numbers, and other identifiers belong to this group. The source calls this structured string data because the text carries a pattern, even though the set of possible values is effectively unlimited. Structured data in this sense is data with a predefined format or organization.

A structured string is a different problem from a categorical one. You cannot list every address, so you cannot map addresses onto categories the way you mapped colors. Instead, the pattern is what you care about. An address is typically a sequence of parts such as a street, a city, a region, and a postal code. The order and separators of those parts are conventions that vary by place, which is one reason the parsing is hard.

Structure does not make the data easy. Structured string data is often very hard to parse, and the right treatment depends heavily on context and domain. Take a date written as 03/04/2024. The pattern looks like two numbers and a four-digit year, but the first two numbers could be read as a month and a day in one convention, or as a day and a month in another. Under the first reading the date is 4 March 2024. Under the second it is 3 April 2024. The text alone does not settle the question. You need to know where the data came from.

The pattern the eye sees is three groups separated by slashes. The meaning of the first two groups is not fixed by that pattern. This is why the source treats these strings as hard to handle, and why it does not attempt a full method for them. Its scope stops short of a systematic treatment of cases like this one.