SwiGLU Activation Function

An activation function takes a number and produces a new number, and a deep learning model applies one after many of its internal computations. The source material for this article centers on GELU, a smooth activation function. The module-level topic here is SwiGLU, but the source pack describes SwiGLU only as the other notable alternative to ReLU that large language models use, so the detailed mechanics taught below are GELU's. Where a claim about SwiGLU would need details the source does not supply, this article says so rather than guessing.

The exact GELU activation is defined as x times $\Phi(x)$, where $\Phi$ is the cumulative distribution function of the standard Gaussian distribution. The Gaussian is the familiar bell-shaped distribution. Its cumulative distribution function gives, for any input, the probability that a standard Gaussian random variable falls at or below that input.

Read the definition as a rule with two parts. The input x is kept as the value being transformed, and $\Phi(x)$ is a number that always lies between 0 and 1. The output is therefore the input scaled down by a factor between 0 and 1. Large positive inputs have $\Phi(x)$ close to 1 and pass through almost unchanged, while large negative inputs have $\Phi(x)$ close to 0 and are pushed toward zero. Because the scaling depends smoothly on the input, GELU does not switch abruptly at a single cutoff point.

The exact definition needs a Gaussian cumulative distribution function, which is not a simple operation to compute directly. The source explains that practitioners commonly implement a computationally cheaper approximation instead. The approximation keeps the same character as the exact function: it is smooth, it passes positive inputs mostly through, and it suppresses negative inputs toward zero. What changes is how the scaling factor is computed. Instead of a Gaussian CDF, it uses a tanh expression built from simple powers, multiplications, and a square root.

Read the expression from left to right. The leading 0.5 times x is the input term, playing the role that x plays in the exact definition. The term $1+\tanh(\cdots)$ takes the place of $2\,\Phi(x)$, since $\tanh$ ranges from negative one to one, so $1+\tanh$ ranges from zero to two and half of it lands between zero and one, matching the range of $\Phi(x)$. The constants $\sqrt{2/\pi}$ and $0.044715$ are the values the source says were found via curve fitting, chosen so the approximation tracks the exact function's shape while using only cheap operations.

The approximation is used because it is computationally cheaper than the exact form, and it is the version the source says was used in training the original GPT-2 model. Its constants were found by curve fitting, so the approximation was designed to closely follow the exact function's shape rather than to be a crude substitute. Engineers usually implement the approximation, while the exact definition remains the reference that explains what the approximation is trying to match.

ReLU has long been the default activation in deep learning because of its simplicity and effectiveness across many architectures. GELU is the more complex of the two. It is smooth, where the scaling depends continuously on the input, and the source reports that this offers improved performance for deep learning models. The key visual difference lies in the behavior near zero and in the negative region: GELU eases into its output instead of switching at a sharp boundary, and it lets negative inputs approach zero gradually rather than cutting them off at one point.