Continuous Features

Imagine a program that looks at a record about one working adult and predicts whether that person earns more than 50,000 dollars a year or not. Each record describes one person through several pieces of information: how old they are, how many hours they work each week, what kind of employer they have, what their education level is, and what their occupation is. Before any learning algorithm can use these pieces of information, we need to understand what kind of information each one is. That distinction is the subject of this article.

The example dataset comes from adult income records in the United States, derived from a 1994 census database. The prediction task is a classification task: each worker belongs to one of two classes, income at most 50K or income above 50K. Predicting the exact income amount would instead be a regression task, which the source notes would be much harder. Keep this two-class setting in mind, because it is the context in which the categorical question becomes important.

Read each column of the dataset as a question you could ask about every worker. Some columns answer a question with a number that can fall anywhere along a scale. Other columns answer a question by choosing one item from a short menu of named options. Each column in a dataset is called a feature, so the task is to decide which features behave like a scale and which behave like a menu.

The two kinds of feature are separated by the meaning of their values. Two definitions capture the difference. The first is the continuous feature, and the second is the categorical variable. Each definition turns on a different question, and the two questions are not interchangeable.

The diagram below groups the features of the adult dataset into the two kinds. Notice that the grouping depends only on what each column means, not on whether its entries happen to be written with digits or with words. An age written as digits is still continuous, and a workclass written as words is still categorical.

The source uses logistic regression as the starting point for the learning task. Logistic regression makes its prediction from a weighted sum of the input features. Each feature is multiplied by its own coefficient learned from the training set, the results are added together, and a constant term $b$ is included. Written with coefficients $w_1, w_2, \ldots, w_n$ for features $x_1, x_2, \ldots, x_n$, the sum has the form $w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b$.

This arithmetic only makes sense when each input is a number. For the age column, the term $w_1 x_1$ multiplies a coefficient by a real quantity, which is meaningful. For the education column, the input is the word Masters or Bachelors. A categorical value such as that cannot be multiplied by a coefficient in any meaningful way. The source makes exactly this point: the formula works when the inputs are numbers, but not when an input is a word. The data therefore has to be represented in some different way before logistic regression can be applied. The course explains that representation in its next step.