Machine Learning Algorithms

Machine learning is a field of study that trains algorithms to make predictions or decisions based on data.

Programming

Learning as Adjusting an Algorithm

Machine learning is a field of study. Its central activity is training algorithms so that they can make predictions or decisions based on data. Before looking at any code, hold that definition in mind, because nearly every idea in this course is a refinement of it.

Two words in that definition do most of the work. Training is the process of adjusting an algorithm using data, and its result is a trained model that holds what was learned. Prediction is the act of applying that trained model to a new case whose answer is not yet known. Keep these two moments separate in your mind; the rest of this article depends on the difference.

The introduction this article draws from is organized around two goals. The first is to understand the fundamental concepts and applications of machine learning. The second is to learn the advantages and shortcomings of widely used machine learning algorithms. This article focuses on the first goal and touches the second only as a preview, since the advantages and shortcomings are not spelled out at this stage.

Training Versus Prediction

Training and prediction are two different moments in the life of a machine learning system. Training uses data that already has answers to shape the algorithm. Prediction uses that shaped algorithm on a case whose answer is unknown. Mixing these two moments up is one of the most common early confusions.

What data goes into training, what comes out of it, and how does a new input flow through the learned result to produce a prediction?

Read the diagram from left to right. Example data goes into training, training produces a trained model, and the model is then applied to a new case to produce a prediction or decision. The new case matters most for the purpose of the system, because it is data the algorithm was not trained on. Training happens once from the examples; prediction can be repeated for any new input.

Tracing a Learned Threshold in Code

The following example is an illustration written for this article, not material taken from the source. It uses a deliberately simple algorithm that learns a single number, a threshold, from labeled examples. A label of 1 means high and a label of 0 means low. Study the code first, then follow how the training step changes the state of the program.

python

# Labeled training data: (value, label) pairs
training_examples = [(1, 0), (2, 0), (3, 0), (6, 1), (8, 1), (9, 1)]

# Before training, the program holds no learned value
threshold = None

def train(examples):
    # Learn the smallest value that was labeled 1
    positives = [value for value, label in examples if label == 1]
    return min(positives)

# Training step: the learned number is stored in threshold
threshold = train(training_examples)
print(threshold)

def predict(value):
    # Prediction step: uses the stored threshold, not the examples
    return 1 if value >= threshold else 0

print(predict(5))
print(predict(7))

# Labeled training data: (value, label) pairs
training_examples = [(1, 0), (2, 0), (3, 0), (6, 1), (8, 1), (9, 1)]

# Before training, the program holds no learned value
threshold = None

def train(examples):
    # Learn the smallest value that was labeled 1
    positives = [value for value, label in examples if label == 1]
    return min(positives)

# Training step: the learned number is stored in threshold
threshold = train(training_examples)
print(threshold)

def predict(value):
    # Prediction step: uses the stored threshold, not the examples
    return 1 if value >= threshold else 0

print(predict(5))
print(predict(7))

Output (expected)

6
0
1

Notes on the code

training_examples holds the labeled data and never changes
threshold starts as None, meaning no learned value exists yet
train() reads the examples and returns the learned number 6
predict() reads threshold; it does not look at training_examples
predict(5) returns 0 because 5 is below 6; predict(7) returns 1

What Changes in Program State

The before and after states show what training actually does in this example. The examples themselves never change. What changes is the threshold variable, which goes from having no value to holding a learned number of 6. Once that number exists, the prediction step can use it on any new value, such as 5 or 7, without looking at the original examples again.

How does the stored value change from before training to after training, and how does that changed value then affect the program's next prediction?

Notice the asymmetry. Training reads the examples and writes one number into the program. Prediction reads only that number. If you delete the training examples after training finishes, predict() still works, because the learned value is what the model holds. This is exactly what the source means when it says the trained model holds what was learned.

Mistakes About Learning From Data

Common mistake
Believing the algorithm learns while it predicts
What it looks like
Assuming that every call to predict(7) in the threshold example updates the threshold.
Why it fails
Training and prediction are different moments. Prediction applies the trained model to a new case; it does not reshape the algorithm.
Do this instead
Treat train() as the step that changes state and predict() as the step that only reads it.
Common mistake
Thinking the training examples are stored inside the model
What it looks like
Assuming predict(5) needs training_examples to decide the answer.
Why it fails
In this example, predict() uses only the learned threshold. The trained model holds what was learned, not the raw data.
Do this instead
Ask what value the prediction step actually reads, and trace it back to the training step that wrote it.
Common mistake
Assuming a prediction on a training example proves the model will handle new cases

Three Mathematical Foundations

Many introductions begin by showing what a machine learning method does and how it behaves, leaving the underlying mathematics for later. That approach lets a newcomer build intuition first. The source then asks what mathematics sits underneath the algorithms, and answers that many of the models discussed rely on principles from three areas: probability theory, linear algebra, and optimization.

The source also states that a practitioner does not need to master every implementation detail of these algorithms, but that knowing some of the theory behind them is valuable. Understanding the theory is valuable, though it is not required before using the algorithms intuitively.

How do the three mathematical areas relate to one another and to the parts of a machine learning model they support?

The diagram reflects the relationship the source describes. Each foundation supplies principles to the same family of models, so one model can draw on more than one at once. It is a simplification: it does not claim that every model uses every foundation equally, or that these three areas are the only mathematics involved.

Why the Theory Helps and Where to Go Next

Understanding the theory behind an algorithm helps a practitioner reason about why a model behaves as it does, which is the kind of understanding the three foundations are meant to support. The source is clear that this theory is valuable but not a prerequisite for using the algorithms intuitively, so it is reasonable to begin with intuition and deepen the theory over time.

The source closes by recommending books for readers who want to go further than an introductory treatment, and it recommends reading at least one of them. The book descriptions are part of the source material; consult the source's own reading recommendations for the titles and what each one covers.

Predicting a Threshold Before Revealing It

What do you think happens?

Using the same train() and predict() functions with threshold = 6, what does predict(4) return?

0
1
None
6

Reveal answer

Answer:

4 is less than 6, so the condition value >= threshold is False and predict returns 0. The examples are not consulted; only the stored threshold is used.

EASY

Suppose the training examples were changed so that the smallest value labeled 1 is 4 instead of 6. Without rerunning the code, predict what train() would store in threshold, then predict the output of predict(5). Explain which step, training or prediction, determined each answer.

Hints

Look at the line in train() that returns min(positives).
Trace the value of threshold before you call predict().

Your attempt (not saved)

Key Takeaways

Machine learning is a field of study that trains algorithms to make predictions or decisions based on data.
Training adjusts an algorithm using data and produces a trained model; prediction applies that model to a new case whose answer is unknown.
In the threshold example, training changes the threshold variable from None to a learned value, and prediction reads that value without revisiting the examples.
The source names probability theory, linear algebra, and optimization as foundations that many machine learning models use.
Probability theory handles uncertainty, linear algebra organizes data as rows and columns, and optimization searches for better model settings; the theory is valuable though not required before using the algorithms intuitively.

Related concepts

Training and Prediction in Machine Learning

More concepts starting with M