Word Embedding Techniques

A computer cannot read a word the way a person does. To use words in a calculation, we first have to turn each word into a numerical form. A common form is a vector: a list of numbers that stands for the word. Such vectors are called word embeddings. Several algorithms and frameworks have been built to produce them, and Word2Vec is one of the earlier and most widely known.

This article follows one question through: if a method only reads text, how can it end up placing related words near one another? Keep that question in mind, because the answer is built from a simple observation about where words appear, and that observation explains nearly every later choice in Word2Vec.

The central idea behind Word2Vec is simple to state: words that appear in similar contexts tend to have similar meanings. The context of a word is the set of words around it. If two different words keep showing up next to the same neighbours, the method treats that shared neighbourhood as evidence that the two words are related in meaning.

The method never needs a dictionary. It only needs running text and the patterns of which words travel together. A learner who sees a word in isolation has very little to go on, but a learner who sees the words that regularly appear beside it can start to infer what kind of word it is.

Word2Vec does not study a whole document at once. It moves a window across the text. At each position, one word is the target and the words around it form its context. Each position therefore yields one training example. Follow the sentence 'the cat sat on the mat' with a window that reaches one word to each side of the target. The diagram below shows three of the steps.

Notice what changes at each step. The target moves forward by one word, and the context shifts with it. The word 'sat' is a neighbour of 'cat' in one step and the target in the next. Across the whole sentence, every word gets a turn as the target, and every word gets a turn as a member of someone else's context. That repeated pattern of pairings is the raw material the network learns from.

Word2Vec is a trained neural network architecture. Its job is to produce word embeddings by learning to predict context. Training can run in either of two directions. In the first, the network is given a target word and is trained to predict the words around it. In the second, the network is given the surrounding words and is trained to predict the target word. The phrase 'or vice versa' in the definition captures exactly this reversal.

Take the target 'sat' from the sentence above. In the target-to-context direction, 'sat' goes in and the network must predict its neighbours, 'cat' and 'on'. In the context-to-target direction, 'cat' and 'on' go in and the network must predict 'sat'. The window of text is identical in both cases. Only the arrow of the task is reversed, and both directions rest on the same bet about context.

Because the training signal comes from predictions, the numbers that end up in each embedding are shaped by how well the network predicts. Words that keep sharing neighbours receive similar pressure during training, which is how the shared-neighbour idea gets written into the numbers.

An embedding is usually a list of many numbers, and a list of many numbers cannot be drawn directly. To look at the structure, the embeddings are projected into two dimensions for visualization. When this is done, terms with similar meanings tend to land close together on the plot, forming groups. Reading the plot is a way to check whether the learned representations place related words in related regions.

In the illustration, the two animal words sit together and the two vehicle words sit together. The grouping is the lesson. Closeness on the page is a rough guide to similarity. It is evidence of relatedness to be read carefully, not a statement of exact synonymy, and it is not a faithful map of the full space, because a flattened view inevitably loses some of the structure of the original.

The same logic applies to the rain and snow example. The reasoning that puts them near each other used only the surrounding words. No definition of rain or snow was consulted, and none was needed.

The plot is persuasive, which makes it easy to over-read. The table below lists common assumptions and the more careful reading that the source supports.

Each embedding is a list of values, and the number of values is called its dimensionality. The source describes embeddings whose dimensionality can range from one to thousands. The number is a design decision, and it carries a trade-off. Higher dimensionality may capture more nuanced relationships between words, but this comes at a cost in computational efficiency. Lower dimensionality is cheaper to work with, though it has less room to express subtle differences.

The projection to two dimensions is a separate matter. Whatever dimensionality the embeddings were trained with, a two-dimensional plot is only a view that makes groups easier to see. Choosing a dimensionality for training and choosing a two-dimensional view for inspection answer different questions.