Limitations of Machine Learning Models

Imagine you work at a travel agency and have been asked to predict what a flight will cost. You already have a useful archive. Every past flight has a price, a date, an airline, a departure city, and a destination. A learning algorithm can take that table and find real relationships in it. This article asks what you should do when the table is not enough, because some of the forces that move prices never appear in the table at all.

Many machine learning projects are motivated by a wish to stop writing rules by hand. Instead of a person deciding in advance that a certain airline is expensive or that a certain route is cheap, the model is expected to discover such relationships from data. Given that goal, it is tempting to think that prior knowledge about the application should be thrown away. The source pushes back on that temptation. Discarding prior knowledge is not required. The answer is not to abandon what you know about travel. Expert knowledge can be turned into new columns, called features, that give the model information it could not otherwise reach.

Start with one row of the archive. This raw record is the representation the model receives by default. Each column becomes an input, and the price is the target the model must predict. Follow what the model can see as it moves from the raw record toward a prediction.

The raw record carries the date, but it does not say anything about holidays. The expert rule supplies that missing relation. Once the relation is stored as a column, the model receives the holiday knowledge as an ordinary input, in the same way it receives the airline or the route.

The flight price example makes the limitation concrete. Flights are usually more expensive during peak vacation months and around holidays. A model that only sees a date can pick up on seasonal patterns that show up in the data. But holidays do not all behave the same way with respect to the calendar, and this is where the raw date starts to fail.

Three kinds of holiday are worth separating. Some holidays fall on a fixed date, such as Christmas, so their effect on prices can be learned directly from the date. Other holidays depend on the phases of the moon, such as Hanukkah and Easter, so the same calendar date does not always mark the same event. A third group is set by authorities, such as school holidays, whose timing is a policy decision rather than an astronomical or calendar rule.

The consequence is direct. A model that stores only Gregorian dates does not contain the information behind moon-dependent or authority-set holidays. The effect of those holidays cannot be read off a date alone, because the date does not determine the event.

The fix the source describes is simple to state. Add a feature that records how a flight relates to a public or school holiday. The new feature takes one of three values: the flight falls on a holiday, the flight comes before one, or the flight comes after one. The model and the target price stay exactly the same. Only the input representation changes.

The left column shows what a model sees when only the raw columns are used. The right column keeps those same columns and adds one expert-derived feature. Nothing about the algorithm or the target has changed. The only difference is that the right-hand input carries knowledge about the holiday structure of the task, which the raw date alone could not express.

Adding a candidate feature gives the algorithm an option, not an obligation. An added feature does not force the algorithm to use it, and an uninformative feature does not hurt. This asymmetry is why domain-driven features are worth trying even when you are unsure whether they will help.

The cost of a weak guess is usually small, while the benefit of a good one can be large, because it may expose a pattern the raw record could never reveal. The strategy is therefore to propose candidate features from what experts know, add them, and let the model decide whether they carry signal.

Consider a simple competition in which entries are ranked by accuracy, the fraction of predictions that match the true labels. The code below is a small, illustrative version of that scoring rule. It is written for this article and is not taken from the source.

The output 0.8 means four of the five predictions match. That single number is what the leaderboard ranks. Notice what the code does not contain: it does not say whether accuracy is the right thing to measure for the real situation, or whether the five labels represent the flights a travel agency actually cares about.

Machine learning competitions are an exciting way to test your skills. You download a dataset, train a model, submit predictions, and watch your name climb a leaderboard. Two decisions are made for you before you start. The first is the target: the competition supplies a particular metric to optimize, so your job becomes improving a number that someone else selected. The second is the data: you usually receive a fixed dataset that has already been preprocessed. Its contents, its format, and the way it was prepared are all settled before you open the files.

These two givens make competitions clean and comparable. Everyone is measured on the same number using the same data, which is exactly what a fair contest needs. The same givens also mean that the skills a competition rewards are narrower than the full set of skills a real project demands.

A strong leaderboard result measures performance on the competition's chosen metric and data, which may differ from the real-world goal. The misconception to watch for is this: if a model ranks first in a competition, it is the best possible answer to the underlying real-world question. That belief quietly assumes that someone has already done the hardest parts of the job. A leaderboard only compares entries against one scoring rule and one prepared dataset. Whether that rule and dataset are the right ones for the actual situation is a separate question, and the leaderboard does not answer it.

In real work, defining the problem and collecting the data are important aspects of the task. Before a model can be trained, someone must decide what the model is supposed to predict, for whom, and what outcome would count as success. Someone must also gather the information that will feed the model. Neither step is handled by a competition, so a real project starts with questions a competition never asks.

Competitions are still valuable, and the same platforms that host them are rich sources of material for practice. The Kaggle forums are a good place to learn about the latest tools and techniques in machine learning, and a wide range of datasets is available on the site. The OpenML platform offers even more: it hosts over 20,000 datasets with over 50,000 associated machine learning tasks.