Rule-Based Systems Versus Learned Models

Imagine you are responsible for an email program. You want incoming messages that look like junk to land in a spam folder instead of the inbox. The broad question behind this task is how to get a program to make decisions like this on its own, and how the approach taken by machine learning differs from other approaches.

Keep this spam filter in mind as you read. It is a concrete problem that makes the contrast between two ways of building intelligent behavior easy to see. Each message that arrives is an input, and the filter's job is to produce an output: spam folder or inbox. The interesting question is where the logic that connects input to output comes from.

In the early days of intelligent applications, many systems relied on handcoded rules. These are explicit if and else decisions that process data or adjust to user input. A person with expertise in the problem writes each rule down, and the program simply applies them in order. Nothing about the program is learned from data; the rules are the program's entire decision logic.

A spam filter built this way might use a blacklist of words. If an incoming message contains a word on the list, the message is marked as spam and moved to the spam folder. The designer decides which words belong on the list, and that decision is the entire intelligence of the system. This is an example of using an expert-designed rule system to build an intelligent application.

Read the diagram as a trace. The message enters at the top, and the only check is whether any blacklisted word appears. The answer to that single yes or no question determines the output. If the check is answered yes, the decision comes out as spam folder. If it is answered no, the decision comes out as inbox. There is no step where the program weighs evidence or learns anything new; the outcome is fixed by the rule the designer wrote.

Machine learning takes a different route. Rather than having a person spell out the decision rules, it is a method of creating intelligent applications by presenting a program with a large collection of data. The program is not handed the rule; it works out what matters from the examples it sees.

The source describes this as a method that allows algorithms to determine the characteristics needed to identify a specific entity, such as a face, from a large collection of data. The key shift is in where the knowledge comes from. In a handcoded system, the knowledge lives in the rules a person wrote. In a machine learning system, the knowledge is extracted from the data.

Read the two columns side by side. Both aim to produce an intelligent application, and in both cases an input goes in and a decision comes out. They disagree about the middle of the process. In the left column a person writes the criteria, and the program applies them. In the right column the criteria are not written in advance; they are determined by the algorithm from the data it is given.

The source uses the example of identifying a face within images. A person could try to write explicit rules describing a face, but the source frames the machine learning approach as one where the algorithm itself determines the characteristics needed to identify the entity from a large collection of data. The program's knowledge is therefore a product of what it was shown.

Notice what the diagram does not contain: a list of rules written by a person. The characteristics sit between the data and the prediction, and they were produced by the algorithm. The same pattern applies to the spam filter. Instead of asking a designer which words belong on a blacklist, a learning approach would derive what distinguishes spam from non-spam from a large collection of messages.

Hand-written rules are not a bad idea by default. The source notes that crafting decision rules manually is feasible for some applications, particularly those where humans understand the process being modeled well. When the rules can be stated clearly by someone who knows the domain, a rule system can work well.

The source states that using handcoded rules to make decisions has two major disadvantages. The core issue is that a rule system depends entirely on what the designer thought to write down. When the problem is too intricate for a person to specify in full, the rules become the bottleneck, because the program can only decide what its rules already cover.

The examples the source gives, a spam filter and identifying a face in images, both involve deciding something about data that a person might find hard to capture in explicit rules. This is the pattern that makes learned models attractive. When a designer cannot state the criteria completely, the approach that determines characteristics from examples has a route that a rule list does not.

Learners often make predictable errors when they first contrast the two approaches. The list below names the most common ones and corrects each.