Document Clustering

Imagine a newsroom archive holding tens of thousands of articles that nobody has ever tagged. A reader who wants every article about the economy, or every article about a local election, has no easy way to find them. The archive is just a large pile of raw sentences with no category attached to any of them.

This situation is common. Customer reviews, support tickets, research abstracts, and social media posts all arrive as raw text with no category attached. Reading every item and labeling it by hand is slow, and for large collections it is impractical.

Instead of asking a person to read everything and label it, topic modeling tries to discover what the documents are about. The technique is often applied to text data for exactly this kind of situation.

A familiar example comes from news. Articles might be grouped into topics such as politics, sports, and finance. An editor would recognize these as the sections of a newspaper. Topic modeling aims to produce this kind of grouping from the text of the articles, rather than from a list of section names that someone wrote down beforehand.

It helps to follow the state of the data from start to finish. At the beginning, the collection is only raw text, and every document is unlabeled. No topic names exist yet. The method must produce the topics and the assignments together, starting from the documents alone.

Topic modeling is a broad task, and it has two cases. In the first case, each document is assigned a single topic. In the second case, each document can have more than one topic. The first case is document clustering, which is a special case of topic modeling. The diagram below shows how the two cases sit under the same umbrella.

Because document clustering is a special case, everything the umbrella task does also applies to it, but the single-topic restriction is added. Keeping that restriction in mind is what lets you tell the two cases apart.

The clearest way to separate the two cases is to ask how many topics each document is assigned. In document clustering, the answer is exactly one. In decomposition-based topic modeling, the answer can be several, and each topic comes with its own degree of relatedness.

When a document is allowed several topics, it helps to picture the document as a bundle of topics rather than a single label. A stronger coefficient means a stronger connection between that document and that topic. The diagram below uses one hypothetical news article to show the idea. The three coefficient values are invented for teaching purposes and are not output from any real run.

Notice what this reading does not claim. It does not say the article belongs to exactly one topic. Each topic receives its own degree of relatedness, and that is the practical meaning of a document having multiple topics.

The multi-topic case is linked to decomposition methods. In a decomposition method, the data is expressed in terms of a set of components. In topic modeling, each component the method learns corresponds to one topic. The representation of a single document then records how much of each component it contains, and those amounts are its coefficients.

Latent Dirichlet Allocation, usually abbreviated LDA, is the decomposition method people most often mean when they talk about topic modeling. In these terms, LDA is one specific way of learning the components, and each component is read as a topic. The shape of the result is the important takeaway: topics as components, and documents as coefficients over those components.