Introduction to Topic Modeling

Earlier in this course you worked with CountVectorizer and TfidfVectorizer. These classes turn a collection of documents into numerical representations that a program can work with. They are useful starting points, but they implement relatively simple text-processing methods. When a task needs more linguistic analysis or a different kind of modeling, the course points you toward other Python packages.

This article introduces the three packages the course recommends for more advanced text processing, explains what each one is good for, and shows how to tell them apart. It also introduces the idea of structure in a prediction problem, which becomes important once simple text methods are no longer enough.

To see what a vectorizer actually produces, picture a tiny collection of two documents. Each document becomes one row, and each word in the shared vocabulary becomes one column. Every cell holds a number for that document and that word. CountVectorizer fills the cells with counts of how often each word appears, while TfidfVectorizer fills them with weights. The grid below uses made-up counts for two short documents to show the shape of the result.

The course names three packages for more advanced text processing. Each one is recommended for a somewhat different reason, so it helps to hold on to a single defining characteristic for each. spacy is relatively new and is described as very efficient and well designed. nltk is well established and complete, but somewhat dated. gensim is a natural language processing package with an emphasis on topic modeling.

Notice that each package is described by a different kind of property. One is described by its efficiency, one by its completeness and age, and one by its focus area. These are not the same measurement, so the three packages cannot be ranked against each other on a single scale. The useful question is not which package is best in general, but which property matches the goal you have.

Here is how the matching works in practice. Suppose your goal is to group a large set of articles by the subjects they discuss. Because gensim is the package whose emphasis is topic modeling, it is the first candidate to examine for that goal. If instead your goal is to process text with a tool that is described as very efficient and well designed, spacy is the first candidate to evaluate. nltk is the package to weigh when a complete, long-established toolkit matters more than modern design, with the caveat that the course describes it as somewhat dated.

Choosing a package is easy to get wrong when a learner reads one short description and stops there. The mistakes below come directly from the way the course describes the three packages.

A very common instinct when starting a machine learning project is to collect every available measurement, pass them all into a general-purpose learner, and trust the algorithm to work out what matters. Many software packages make this easy, since they ship predefined models that each apply one particular algorithm. The approach feels efficient, but it rests on an assumption worth examining: that the learner will discover everything useful about the problem by itself, using nothing but the raw numbers.

The useful observation is that structure can often be written down in the language of probability theory. Once it is written down, it becomes something the model can use directly, rather than something a person has to hope the data will reveal on its own.

A problem is structured when there is a mathematical model of the situation being predicted, and probability theory is a natural language for expressing that structure. So the test is practical: before you rely on a generic learner, ask whether you can describe the situation you are predicting with a model. If you can, the model's regularities are something you can encode rather than something you must leave for the learner to find.

The concept map below shows the chain of ideas. A real-world problem with regularities leads to a mathematical model of the situation, which can be expressed with probability. Incorporating that expressed structure into the model is what can yield better-performing predictions.

Return to the navigation app. Each sensor gives evidence about where the user is, but none is perfect. A probabilistic model can state how the readings relate to the true position and to each other. The goal of the model is not only to produce a single number. It is meant to reason about how much each measurement should be trusted, and to combine the measurements into a best guess for the user's location.

The same reasoning covers a second kind of problem. Suppose a position estimate would place the user inside a wall. Given the map, that estimate is implausible, so the model can treat it with skepticism. In both cases the model is not simply accepting whatever the raw numbers say. It uses what is known about the situation to weigh the evidence.

Once you have expressed the situation and your model of how its factors work together, you still need a way to compute predictions from that custom model. Probabilistic programming languages address this need. They are described as the most general methods for computing predictions from custom probabilistic models, and they offer a very elegant and compact way to express a learning problem. Examples named in the course are PyMC and Stan.

The course does not describe the internal mechanics of how these languages perform inference, so this article does not go into that detail. For a learner, the key point is the division of labor: you define the structure of the problem, and the language provides a general way to compute predictions from that definition.

The course points to two books for deeper study. One is an overview of natural language processing that also introduces nltk. The other is a more conceptual reference covering fundamental algorithms in information retrieval, natural language processing, and machine learning. The course notes that both books have online versions that can be accessed free of charge.