Choosing the Right Machine Learning Model

When you start a machine learning project, you often face a choice: dozens of model types could plausibly solve the problem, and each one makes different assumptions about the data. Choosing the right machine learning model means deciding which approach fits the problem you have. One of the models you can choose from is the Nearest Neighbors model. It is a useful first candidate because it is simple to describe, simple to build, and gives you a reference point that more complicated models must beat.

This article follows one small lookup from start to finish. You will see what the model does with a new value, trace the steps by hand and in Python, and then check your understanding with a practice tracing task.

The name describes the core idea. To make a decision about a new data point, the model looks at the existing data points that are nearest to it. The word neighbors refers to stored examples that sit close to the new point. The word nearest refers to the ones that are closest by the measure you are using. The new point is the query, the thing you want a decision about. The stored examples are the data the model already has.

Think of the stored examples as points scattered across a space, where each point represents one example. The query is placed into the same space. Closeness is the measure that tells you which stored examples count as neighbors. In the simplest case, with one number per example, closeness is just how far apart two numbers are on a line.

Before reading any code, try the lookup yourself. Suppose you have five stored examples. Each one is a single number paired with a label: 1.0 is labeled low, 2.5 is labeled low, 4.0 is labeled mid, 7.5 is labeled high, and 9.0 is labeled high. A new query value arrives: 6.2. The model must decide which stored example is closest to 6.2.

For each stored value, compute the distance to the query. With single numbers, the distance is the absolute difference, written $|6.2 - x_i|$ where $x_i$ is a stored value. The distances are: 1.0 gives 5.2, 2.5 gives 3.7, 4.0 gives 2.2, 7.5 gives 1.3, and 9.0 gives 2.8. The smallest distance is 1.3, which belongs to the stored value 7.5. So the nearest stored example is 7.5, and its label is high.

The same logic can be written as a short Python program. The code keeps track of the index of the best stored example seen so far. It starts with the first example, then checks each remaining example. Whenever it finds a stored value with a smaller distance to the query, it updates the best index. At the end, the best index points to the nearest stored example.

Notice what the program does not do. It never fits a formula to the five stored values and never produces a learned rule. Each prediction is computed by going back to the stored examples and comparing the query against them. This is why the stored examples are central to the model: they are what the model consults every time it makes a decision.

Choosing a model means comparing how candidates make predictions and what data they need. Nearest Neighbors works directly from its stored examples. When a new point arrives, it measures closeness to those examples and uses the nearest ones. Other candidate models instead learn parameters from the data during a training step, and then use those learned parameters, not the original examples, to make predictions.

The source identifies small datasets as the setting where Nearest Neighbors is a good fit. When a dataset has relatively few examples, a model that works directly from those examples is practical to try, and it gives you a clear starting point before you commit to more complicated approaches. The source does not give a size threshold, so you must judge what counts as small for your own problem.

The model is also useful as a baseline. A baseline is a simple reference model. Its job is not necessarily to be your final choice, but to give you a point of comparison. If a more complex model cannot clearly beat a simple baseline, that is useful information about the problem and about the extra complexity you were considering. Building and measuring a Nearest Neighbors model first gives you that reference number before you move on to other candidates.

The reasoning behind Nearest Neighbors can be stated in one sentence: a new point is compared with the stored examples, and the ones closest to it are the ones that matter. Because the idea is this direct, the model is easy to explain to colleagues, stakeholders, or anyone who needs to understand why a prediction was made. The source recommends it as an early model partly for this reason.

In the lookup above, you can show exactly why the answer was high. The query 6.2 was closest to the stored example 7.5, at distance 1.3, and that example carries the label high. Anyone reviewing the prediction can see the stored example that influenced it and the distance that made it the nearest. That traceability is what makes the explanation concrete.