Learning Problems and Hypothesis Classes
Linear regression is used to model relationships between explanatory variables and a real-valued outcome.
From Information to an Estimate
Suppose you want to estimate a baby's weight using information such as her age and weight at birth. The age and birth weight are the explanatory variables: they provide the information used to make the estimate. The baby's current weight is the outcome: it is the real-valued quantity being estimated. Linear regression is a statistical tool for modeling this kind of relationship.
The direction of the problem matters: explanatory variables are used as inputs, while the real-valued outcome is the quantity the predictor aims to estimate.
The Learning-Problem View
Linear regression can be viewed as a learning problem. Its input domain, written as X, is a subset of the real-coordinate space R to the power d for some d. Each input is therefore a feature vector with d components, or a member of the allowed collection of inputs. Its label set, written as Y, is the set of real numbers. A label is the real-valued outcome associated with an input.
Classifying the Parts of an Example
Consider the baby-weight estimation task described in the source.
Identify the inputs: Age and weight at birth are information used to make the estimate, so they belong with the explanatory variables.
Identify the outcome: The baby's weight being estimated is the outcome. It is a real-valued quantity.
Place them in the learning formulation: The input belongs to the input domain X, while the estimated outcome belongs to the real-valued label set Y.
The task pairs an input made from explanatory variables with a real-valued outcome.
Choosing the Hypothesis Class
A hypothesis class is the collection of candidate functions that a learning problem is allowed to consider. In linear regression, the hypothesis class is restricted to linear functions. The learning task is to select or learn a suitable member of that class so it can approximate the relationship between the explanatory variables and the outcome.
Generated example: imagine that one candidate linear function predicts an outcome using a feature vector with one set of weights and an intercept, while another candidate uses different weights and an intercept. These are different hypotheses, but both remain inside the linear-function hypothesis class. Learning consists of choosing a suitable candidate for the relationship represented by the data.
Following the Learned Predictor
The goal is not merely to name or describe the variables. A learned function, written as h, maps an input from R to the power d to a real number. Its purpose is to best approximate the relationship between the explanatory variables and the outcome. In this sense, linear regression provides a way to learn a predictor from the relationship being modeled.
Tracing a Prediction Task
Separate the roles of the variables in a task that estimates a baby's weight from age and weight at birth.
Start with the information: Age and weight at birth are selected as explanatory variables because they provide the input information.
Specify the target: The baby's weight is the real-valued outcome that the task aims to estimate.
Restrict the candidates: The learner considers linear functions as candidate predictors because linear regression uses a linear-function hypothesis class.
Learn a predictor: The learned function h maps the input to a real number intended to approximate the relationship between the explanatory variables and the baby's weight.
The complete learning problem moves from explanatory-variable inputs, through a learned linear predictor, to an approximated real-valued outcome.
Mistakes in Problem Formulation
Treating the outcome as an explanatory variable.
The explanatory variables provide information for the estimate, while the outcome is the real-valued quantity the predictor aims to approximate.
Fix:
Ask which quantities are supplied to make the estimate and which quantity is the target.Describing linear regression only as a way to summarize data.
The learning formulation emphasizes learning a predictor that maps an input to a real number.
Fix:
Include the learned function and its role in approximating the relationship.Confusing the hypothesis class with one learned hypothesis.
A hypothesis class is the collection of candidate functions, while the learned predictor is a suitable member selected from that collection.
Fix:
Use hypothesis class for the full collection and learned predictor for the selected function.Forgetting the restriction to linear functions.
Linear regression restricts its candidate functions to linear functions.
Fix:
State that the hypothesis class consists of linear functions.
| Role | Meaning in the learning problem | Baby-weight example |
|---|---|---|
| Explanatory variables | Inputs used to make an estimate | Age and weight at birth |
| Outcome | Real-valued quantity being estimated | Baby's weight |
| Hypothesis class | Collection of candidate functions | Linear functions |
| Learned predictor | Selected function that approximates the relationship | Function h mapping inputs to a real number |
Check Your Formulation
A learning task uses two measurements as input and aims to estimate one real-valued quantity. Explain which measurements belong to the explanatory variables, what belongs to the label set, what the input domain represents, and why the candidate predictors belong to a hypothesis class.
Hints
- Start by identifying what information is available before the estimate is made.
- The target is the real-valued quantity being estimated.
- The input domain contains allowed inputs, and the label set contains real-valued labels.
- For linear regression, the hypothesis class is restricted to linear functions.
What do you think happens?
A task estimates a baby's weight from age and weight at birth. Which item is the outcome?
Reveal answer
Answer: The baby's weight being estimated
Age and weight at birth are explanatory variables used as input information. The baby's weight is the real-valued outcome that the learned predictor aims to approximate.
Key Takeaways
- Linear regression models relationships between explanatory variables and a real-valued outcome.
- The explanatory variables form the input, while the outcome is the quantity being estimated.
- The input domain X is a subset of R to the power d, and the label set Y is the set of real numbers.
- A hypothesis class is a collection of candidate functions; linear regression uses a class of linear functions.
- The learned predictor is a linear function that maps an input to a real number and aims to approximate the modeled relationship.
Key Takeaways
- Linear regression is a statistical tool for modeling a relationship between explanatory variables and a real-valued outcome.
- In the learning formulation, inputs come from a domain X that is a subset of R to the power d, and labels come from the real numbers.
- The hypothesis class is the collection of candidate functions permitted by the learning problem.
- For linear regression, the hypothesis class consists of linear functions.
- A learned predictor maps an input to a real number and approximates the relationship between the inputs and outcome.