Model Evaluation and Tuning

Parameter tuning adjusts the settings that shape a model, and evaluation is how you compare the tuned versions.

Programming

Building Fast Without Reading Everything

Imagine you have a deadline and need a working machine learning system within a short time. Opening a long technical text on the first page and reading straight through may feel like the responsible choice, but the course says it is not the most efficient route for a beginner. This section explains how the course suggests you move through its material when speed matters, and why that route still leaves you with the knowledge you need.

The course divides its material by purpose. Chapters 2 and 3 supply the actual algorithms, meaning the detailed procedures behind each supervised model. Those chapters are thorough, but a beginner may not need to master every one of those algorithms before building something useful. The course treats the algorithms as reference material to be used when a specific model is needed, not as a prerequisite for starting.

The Starting Material

Chapter 1 and the opening sections of Chapter 2 introduce all of the core concepts. If you read those parts, you acquire the vocabulary and mental framework that the rest of the course builds on, without committing to the full depth of every algorithm. Think of these opening parts as the set of ideas you need in order to understand any model you later choose to study more closely.

Key point

The opening material is the foundation for every later choice. It gives you the terms and the overall workflow, so the detailed algorithm chapters become readable reference material rather than a required first read.

Tracing the Recommended Route

The recommended path moves in a fixed order, and each stage hands you what the next stage needs. The route is best followed as a sequence rather than as a set of independent reading tasks. Start with Chapter 1 and the opening sections of Chapter 2. From there, the course recommends jumping ahead to the Summary and Outlook section on page 127 of Chapter 2 rather than continuing in order. Once you have chosen a model, you return to that model's dedicated section for details. Only after your model is understood do you turn to the evaluation techniques in Chapter 5 to judge and tune it.

Which stages come first, which comes next, and where a beginner can skip ahead instead of reading in order

Read the flowchart as a fixed sequence with one decision point. The decision point comes right after the opening material. At that moment you can jump to the summary instead of continuing through the algorithm chapters. Notice that the route is described as a first pass. Once your system runs, the detailed sections for your chosen model are where you deepen your understanding.

Using the Summary as a Menu

The Summary and Outlook section, found on page 127 in Chapter 2, lists every supervised model covered in the course. This list works as a menu. You scan it, compare the options, and select the model whose strengths match the problem in front of you. The summary answers the question of what is available.

Once you have chosen a model, the navigation reverses direction. You flip back to the section devoted to that model and read it for the details. The summary tells you what is available, and the dedicated section tells you how the chosen model works. Keeping those two roles separate is what makes the route efficient. Trying to get the mechanics from the summary, or trying to compare every model through the detailed sections, mixes up the two jobs.

Navigation Mistakes That Cost Time

Common mistake
Reading Chapters 2 and 3 straight through before building anything
Why it fails
The course notes that a beginner may not need to master every algorithm before building something, and reading in order is described as less efficient when speed matters.
Do this instead
Read Chapter 1 and the opening sections of Chapter 2 first, then use the summary to choose a model.
Common mistake
Jumping straight into a model's detailed section without consulting the summary
Why it fails
Without the summary you have no overview of the supervised models covered, so you cannot compare options that fit your problem.
Do this instead
Scan the Summary and Outlook list first, pick the model that fits, and only then flip back to its dedicated section.
Common mistake
Treating the summary as enough to understand how a model works
Why it fails
The summary lists what is available; the dedicated section explains how the chosen model works.
Do this instead
After choosing, read the dedicated section for the details.
Common mistake
Skipping the evaluation techniques in Chapter 5
Why it fails
A model that exists still needs a way to judge how well it works, and tuning needs a way to compare its versions. Without evaluation you cannot tell whether a tuned model is better.
Do this instead
Use the Chapter 5 techniques to evaluate and tune your chosen model, and revisit them whenever you change or retune it.

What Evaluation and Tuning Are For

Building a model is only part of the work. Once a model exists, you need a way to judge how well it works, and you may need a way to adjust the settings that shaped it. Model evaluation judges how well a model performs, and this section introduces advanced methods for doing so. Parameter tuning adjusts the settings that shape a model. Evaluation is how you compare the tuned versions.

The two topics are closely linked. Tuning produces different versions of a model, and evaluation is how you compare those versions. Read together, they form a loop of adjusting, judging, and adjusting again. The diagram below shows the steps of that loop in order.

In what order do the adjust, train, evaluate, and compare steps repeat, and where does the loop stop?

The loop is read from top to bottom and then back to the start. Each pass produces a tuned version, judges it, and compares it with the others before you adjust again. The source for this section does not give a specific stopping rule, so treat the stop point as a judgment you make from the comparisons rather than a rule stated in the course.

Pipelines as Ordered Chains

The section adds a third idea that ties the workflow together: pipelines. A pipeline chains steps so that each step receives the output of the previous one, and it encapsulates the workflow as one unit. The pipeline does not change what each step does. It arranges the steps and runs them in sequence.

Key point

The order of steps in a chain affects the final result, so order must be chosen deliberately. Each step works on whatever the previous step produced.

The next diagram shows a value moving through two different chains. Both chains use the same two steps, but in opposite orders, so you can see where the values diverge.

When a value enters a chain of ordered steps, what does it look like after each step, and how does swapping two steps change the final output?

Both chains start with the same value and use the same two steps, yet they end at different results. The only difference is the order. Each step receives whatever the previous step produced, so the intermediate value changes the outcome of the next step.

Predicting a Chain Before Running It

What do you think happens?

Suppose a pipeline first adds one to a value and then doubles it. If you start with 3, what does the pipeline return? Work it out on paper before reading the answer.

Reveal answer

Answer:

Adding one to 3 gives 4, and doubling 4 gives 8.

Worked example

Tracing the same two steps in the opposite order

Run the steps add one and double on the starting value 3, first in the order add one then double, and then in the order double then add one. Compare the two final results.

1
Order A, step 1
Add one to 3, giving 4.
2
Order A, step 2
Double 4, giving 8. This is the final result of order A.
3
Order B, step 1
Double 3, giving 6.
4
Order B, step 2
Add one to 6, giving 7. This is the final result of order B.
5
Compare
The same two steps on the same starting value give 8 in one order and 7 in the other. Order changes the result.

Result

Order A returns 8 and order B returns 7.

The same idea can be written as a small program. The code below defines the two steps and a function that runs any list of steps in sequence, passing each output into the next step.

Code walkthrough

def add_one(x):
    return x + 1

def double(x):
    return x * 2

def run_pipeline(steps, value):
    for step in steps:
        value = step(value)
    return value

print(run_pipeline([add_one, double], 3))
print(run_pipeline([double, add_one], 3))

1
The first call runs add_one then double on 3, so the value goes 3, 4, 8.
2
The second call runs double then add_one on 3, so the value goes 3, 6, 7.
3
The function run_pipeline is the encapsulated workflow: the list of steps is one unit that can be run on any starting value.

add_one and double are the individual steps
run_pipeline holds the ordered list of steps and runs them one after another
value is reassigned each time, so each step receives the previous step's output

Output

8
7

Mistakes When Reading Pipelines

Common mistake
Assuming the order of steps does not matter because each step is correct on its own
What it looks like
Add one then double gives 8 from 3, but double then add one gives 7 from 3.
Why it fails
Each step works on whatever the previous step produced, so the intermediate value depends on the order.
Do this instead
Trace the value through each step in the order the chain actually runs, and choose the order deliberately.
Common mistake
Thinking a pipeline changes what each step does
Why it fails
The pipeline arranges the steps and runs them in sequence; it does not alter the steps themselves.
Do this instead
Check each step on its own first, then consider how the chain connects them.
Common mistake
Treating evaluation as a one-time check instead of a comparison across tuned versions

Practice Tracing a Different Chain

EASY

A pipeline first subtracts 2 from a value and then triples the result. Start with 5. What does the pipeline return? Then predict what happens if the same two steps run in the opposite order, tripling first and then subtracting 2. Does the result change?

Hints

Trace the value after each step, the same way the worked example traced 3.
Subtracting 2 from 5 gives 3 before tripling, so the first ordering is (5 minus 2) times 3.
Tripling 5 first gives 15, and then subtracting 2 is the second ordering.

Your attempt (not saved)

At a glance

A beginner can start with Chapter 1 and the opening sections of Chapter 2, which introduce the core concepts, and does not need to master every algorithm before building a first system.
The Summary and Outlook section on page 127 of Chapter 2 is a menu of supervised models: scan it to pick a model, then flip back to that model's dedicated section for the details.
Model evaluation judges how well a model performs, and parameter tuning adjusts the settings that shape it; tuning produces versions that evaluation compares, forming a loop of adjusting, judging, and adjusting again.
A pipeline chains steps so each one receives the previous output and the whole workflow is encapsulated as one unit, and because order changes the result, the order must be chosen deliberately.

Related concepts

More concepts starting with M