Optimization and Model Training

Many introductions to machine learning show you what a method does and how it behaves, and they leave the underlying mathematics for later. That is a sensible order. It lets a newcomer build intuition about common algorithms before facing every equation. This article picks up where those introductions tend to stop and asks a simple question: if the algorithms can be understood intuitively, what mathematics sits underneath them?

The short answer is that many of these models rely on principles from three areas: probability theory, linear algebra, and optimization. You do not need to master every implementation detail of these algorithms to use them, but knowing some of the theory behind them is valuable. Before reading on, make a guess about the first of these areas.

A useful way to picture a machine learning model is as a machine built from several parts. Some parts describe how uncertain or noisy the data is. Some parts organize the data into a structured form so that it can be manipulated. And some parts decide how the model should change to become better at its task. Each of the three labels names a body of mathematics that supplies tools for one of these roles.

The diagram below shows the relationship. Each of the three foundations supplies principles to the same family of models, so a single model can draw on more than one of them at once. The diagram is a simplification. It does not claim that every model uses every foundation equally, and it does not claim these three areas are the only mathematics involved. It only reflects the fact that many models use principles from all three.

Probability theory is the first foundation. Many machine learning models use its principles. Probability gives a language for situations where the outcome is not fully known in advance. Data collected from the world almost always contains variation, and a model that makes predictions from such data has to deal with that variation somehow. Probability theory is one of the main mathematical tools for reasoning about that uncertainty.

Linear algebra is the second foundation. It is concerned with collections of numbers arranged in ordered structures, and with the operations that can be performed on those structures. In machine learning, a dataset is often arranged in exactly this kind of ordered form: each example is one row of numbers, and each measured feature is one column. Linear algebra supplies the rules for working with such arrangements efficiently.

The value of this view is that a whole table of numbers can be handled as one object. Instead of processing each number separately, the rules of linear algebra let you describe and manipulate the entire arrangement at once.

Optimization is the third foundation. In broad terms, it is about finding the best option from a set of possibilities according to some measure of quality. You start with several candidates, you score each one with the quality measure, and you keep the one with the best score. The measure decides what counts as best, so choosing it is part of the problem.

Learning in a machine learning model can be seen as a search of exactly this kind. The model's settings are adjusted so that its behavior on the training data becomes better by the chosen measure. The candidates in the diagram are simply different settings, and the quality measure is the yardstick used to compare them.

Training is not a single calculation. It is a loop. Begin with the model's current settings, measure how it behaves on the training data, compare that behavior to the chosen quality measure, and adjust the settings. Then measure again. The diagram traces one pass through this loop and the return to the start.

The following trace is an invented teaching example, not a description of any particular algorithm. It has one setting, called w, and a quality score measured on the training data, where higher is better. Each round proposes a small nudge upward and keeps the nudge only if the score improves.

Understanding the theory behind an algorithm helps a practitioner reason about why it behaves as it does, even when the implementation is handled by existing tools. Knowing that probability describes uncertainty, that linear algebra organizes data as ordered numbers, and that optimization searches for better settings gives you three questions to ask about any model you meet.

The source recommends books for readers who want to go further than an introductory treatment. It suggests reading at least one of them and digging deeper. The recommended books are not named in the material available for this article, so look for the books on probability, linear algebra, and optimization that suit your level.