Linear Algebra for Data Representation

Many introductions to machine learning start by showing what a method does and how it behaves, and they leave the underlying mathematics for later. That order is useful. It lets a newcomer build intuition about the most common algorithms before worrying about every equation. This article picks up where such an introduction usually stops and asks a simple question: if the algorithms can be understood intuitively, what mathematics sits underneath them?

The source gives a direct answer to the question. Many of the models discussed in an introductory treatment rely on principles from three areas: probability theory, linear algebra, and optimization. The source also states that a practitioner does not need to master every implementation detail of these algorithms, but that knowing some of the theory behind them is valuable. The rest of this article explains each area in turn and shows how they fit together.

Think of a machine learning model as a machine built from several parts. Some parts describe how uncertain or noisy the data is. Some parts organize the data into a structured form so that it can be manipulated. Some parts decide how the model should change to become better at its task. The source places these three roles under the labels probability theory, linear algebra, and optimization. Each label names a body of mathematics that supplies tools for one of these roles.

The diagram is a simplification. Each foundation supplies principles to the same family of models, so one model can draw on more than one of them at once. The diagram does not claim that every model uses every foundation equally, and it does not claim that these three areas are the only mathematics involved. It only reflects the statement that many models use principles from all three.

The first foundation is probability theory, and the source states that many machine learning models use its principles. Probability gives a language for situations where the outcome is not fully known in advance. Data collected from the world almost always contains variation, and a model that makes predictions from such data has to deal with that variation somehow. Probability theory is one of the main mathematical tools for reasoning about that uncertainty.

The second foundation is linear algebra. It is concerned with collections of numbers arranged in ordered structures, and with the operations that can be performed on those structures. In machine learning, a dataset is often arranged in exactly this kind of ordered form: each example is one row of numbers, and each measured feature is one column. Linear algebra supplies the rules for working with such arrangements efficiently.

The third foundation is optimization. In broad terms, optimization is about finding the best option from a set of possibilities according to some measure of quality. Learning in a machine learning model can be seen as a search of this kind. The model has settings, and those settings are adjusted so that its behavior on the training data becomes better by the chosen measure.

The source states that a practitioner does not need to master every implementation detail of these algorithms. It also says that knowing some of the theory behind them is valuable. Theory gives a practitioner a way to reason about why a model behaves as it does: what its data is assumed to look like, how its settings are being searched, and what kind of uncertainty its predictions carry.

For deeper study, the source closes by recommending several books for readers who want to go further than an introductory treatment, and it advises reading at least one of them. The source gives a brief description of each recommendation. Look up those recommendations in the source itself to choose a starting point.