Optimization and Parameter Learning
The hypothesis space is the set of possible transformations or solutions available to a learning algorithm.
The Searchable World of a Model
When a learning algorithm fails to produce a desired solution, there are two different possibilities. The solution may not be available within the model's hypothesis space, or the solution may be available but the parameter-learning process may fail to find it. Keeping these possibilities separate is the central idea of optimization and parameter learning.
The hypothesis space is the set of possible transformations or solutions available to a learning algorithm.
A model can search only within the transformations allowed by its topology. Learning does not redesign that topology; it searches for a useful set of values for the weight tensors involved in the available operations.
From Topology to Tensor Operations
A network topology is more than a drawing of which components are connected. It specifies a sequence of tensor operations that can carry inputs toward outputs. That sequence determines the kinds of mappings the network can represent. The topology therefore constrains the hypothesis space before any parameter values are learned.
Separating the Operation Path from the Parameter Choice
Consider a network whose topology specifies two tensor operations in sequence. Explain what is fixed by the topology and what learning must determine.
Identify the available path: The topology specifies that the input passes through the first operation and then the second operation before producing an output.
Identify the hypothesis space: The sequence of operations defines the family of input-to-output transformations that this network can make available.
Identify the learning task: Parameter learning searches for values for the weight tensors involved in those operations.
Separate the two roles: The topology determines what kinds of mappings are available, while the learned weights select a particular mapping from that available family.
Topology determines the available family of transformations; parameter learning selects one transformation from that family.
Why Depth May Not Add Expressive Power
It is tempting to assume that adding layers always creates a larger hypothesis space. The more precise prediction is different: the effect of depth depends on the kind of operation performed by the layers. If added layers repeat the same restricted kind of operation, the resulting space can remain restricted. In particular, repeated linear layers remain limited to linear operations.
Predicting the Effect of Repeated Linear Layers
A model has one linear layer. A second layer is added, but it is also linear. Does the added depth automatically guarantee a larger hypothesis space?
Inspect the original operation: The original model performs a linear operation from input to output.
Inspect the added operation: The new layer repeats the same restricted kind of operation: a linear operation.
Consider the whole sequence: The sequence remains limited to linear operations rather than introducing a different kind of operation.
Make the prediction: The added layer does not automatically make the hypothesis space larger in the relevant sense. Repeated linear layers remain limited to linear operations.
Depth alone does not determine the full range of functions; the operation type used by the layers matters.
When evaluating whether extra layers help, ask what operations the new layers add. Do not use layer count by itself as a prediction of hypothesis-space size.
Available Is Not Yet Learned
Suppose a desired transformation belongs to a network's hypothesis space. That means the network has the structural capacity to represent it. It does not mean that parameter learning will successfully select the required weights. Learning searches through the available family, and that search can fail even when the desired solution is included.
Diagnosing a Failed Learning Attempt
A desired transformation is not obtained. Use the two-question diagnosis to separate representational failure from search failure.
Question one: is it available?: Ask whether the desired transformation belongs to the hypothesis space defined by the network topology.
If it is excluded: The topology does not make that transformation available, so parameter learning cannot obtain it by searching within that network.
If it is included: The topology allows the desired transformation, so the failure cannot be explained by representational exclusion alone.
Question two: was it found?: If the transformation is included but was not obtained, the parameter-learning process failed to find that included solution.
Failure can come from the hypothesis space being too restricted or from parameter learning failing to find a solution that the space already contains.
How Parameter Search Progresses
The topology first defines the available sequence of tensor operations. Parameter learning then searches for a good set of values for the weight tensors in that sequence. Each candidate set of values corresponds to a particular transformation available within the topology. The search may move toward a useful solution, or it may fail to find one even when the desired solution belongs to the space.
This process gives a useful vocabulary for interpreting outcomes. If the topology excludes the target transformation, no choice of values within that topology can make the target available. If the topology includes the target but learning does not reach it, the limitation lies in finding the included solution rather than in the definition of the hypothesis space.
Common Reasoning Errors
Assuming that more layers always create a larger hypothesis space.
Depth by itself does not determine the full range of functions. Repeated linear layers remain limited to linear operations.
Fix:
Inspect the kind of operation added by the new layer and ask whether it changes the available family of transformations.Treating an included solution as a successfully learned solution.
Membership in the hypothesis space only establishes that the solution is available in principle.
Fix:
Separate the capacity question from the parameter-learning question.Blaming every failure on an inadequate hypothesis space.
Parameter learning can fail to find a solution that is already included in the hypothesis space.
Fix:
First ask whether the solution is representationally available, then ask whether learning found it.Treating topology as only a visual connection diagram.
Topology determines the sequence of tensor operations, not merely the number of layers shown.
Fix:
Trace the operations from input to output and use that sequence to reason about the hypothesis space.
Practice the Two-Part Diagnosis
A network fails to produce a desired transformation. Explain the two questions you should ask before deciding why it failed. Then consider a second network with additional layers that repeat the same restricted operation. What should you inspect before predicting that its hypothesis space is larger?
Hints
- Start by separating availability from discovery.
- For the second network, focus on the kind of operation performed by the added layers rather than layer count alone.
What do you think happens?
A target transformation is not learned. Which diagnosis must come first?
Reveal answer
Answer: Either diagnosis is possible, so first check whether the target belongs to the hypothesis space.
Failure can result from representational exclusion or from parameter learning failing to find an included solution. These are different questions and must be separated.
Key Takeaways
- The hypothesis space is the set of possible transformations or solutions available to a learning algorithm.
- Network topology determines the sequence of tensor operations and therefore constrains the available hypothesis space.
- Adding layers does not automatically expand that space; repeated restricted operations can remain restricted, including repeated linear operations.
- A target can belong to the hypothesis space without being found by parameter learning.
- When learning fails, distinguish representational exclusion from failure to discover an included solution.
Key Takeaways
- Hypothesis space describes what transformations a model can make available.
- Topology defines the tensor-operation sequence; learned weights select a transformation within that family.
- More layers help only when they change the kinds of operations available, not merely when they repeat a restricted operation.
- A solution can be inside the hypothesis space while still being missed by parameter learning.
- Diagnose failure by checking availability first and successful discovery second.