GELU Activation Function

Two activation functions can look similar at a glance and still behave differently in ways that matter during training. ReLU and GELU are a good pair to study for this reason. Both rise for positive inputs and look broadly alike there, so the interesting part is what happens on the left side of the axis, where the inputs are negative.

Rather than starting with formulas, this article starts with a short script that evaluates both functions on the same evenly spaced inputs and draws their outputs side by side. Watching what each function does across a range of values, especially the negative ones, gives you a concrete picture to reason from before any vocabulary is introduced. Read the script once for its structure, then focus on what the two panels are meant to reveal.

The script assumes that a GELU class and the PyTorch neural network module are already available, as they are set up earlier in the same chapter. It creates one GELU object and one ReLU object, generates 100 input values from -3 to 3, passes those inputs through both functions, and then plots each set of outputs in its own panel. Because both panels use the same input range, any difference you see between them is a difference in how each function transforms its inputs.

When you run the script, look at the two panels together. The inputs on the horizontal axis are identical in both. The vertical axis shows the outputs. The only thing that changes between the panels is the function applied to the inputs, so the shape difference you see is entirely caused by that function.

ReLU outputs its input for positive values and zero for any negative input. It is a piecewise linear function, meaning it is built from straight-line pieces joined together. For positive inputs it passes the value through unchanged, and for negative inputs it returns zero. Those two rules meet at zero and form a sharp corner in the curve, which is the point where the function changes from one linear piece to the other.

Trace a few negative inputs through ReLU. For $x=-3$, the output is 0. For $x=-2$, the output is 0. For $x=-0.5$, the output is 0. No matter how negative the input is, the output stays fixed at zero, which is why the ReLU panel shows a flat line sitting on the baseline across the entire left half of the axis.

Both curves rise for positive inputs and look broadly alike there. The separation happens on the negative side. Where the ReLU panel shows a flat line on zero, the GELU panel shows a curve that dips slightly below the baseline near the negative side before returning toward zero. The diagram below compares the two outputs at one negative input.

GELU is a smooth, nonlinear function that approximates ReLU in its overall shape. Instead of flattening to zero for negative inputs, it keeps a small, nonzero output, so the curve bends gradually rather than cutting sharply at the origin. A negative input to a GELU unit still produces a signal, even though that signal is much weaker than a positive input of similar size.

One way to describe GELU is as the input multiplied by a gate. The gate is the Gaussian cumulative distribution function $\Phi(x)$, which takes values between 0 and 1. Positive inputs tend to receive a gate value close to 1, so they pass through almost unchanged. Strongly negative inputs receive a gate value close to 0, so they are suppressed heavily but not cut off by a hard threshold the way ReLU cuts them at zero.

The gradient is the slope of the function at a given input. It determines how strongly a change in the input moves the output, and during training it is what carries the learning signal backward through a network. For ReLU, the slope on the negative side is zero everywhere, because the output is constant there. A neuron with a negative input under ReLU therefore passes no gradient backward at that step.

GELU behaves differently. Across almost all of its negative values it has a nonzero gradient, so a neuron with a negative input still takes part in learning. The source notes one specific exception: near $x \approx -0.75$, the gradient is zero. Outside that small neighborhood, the slope is nonzero across the negative range.

The source ties the shape of each curve to how easily a network can be optimized. A smooth curve lets parameter adjustments respond in fine increments, which the source describes as more nuanced changes to the model's parameters. ReLU's sharp corner at zero can make optimization harder, and the source says this can be especially true in very deep networks or networks with complex architectures.

The corner matters because the slope changes abruptly there. Just to the left of zero, ReLU's slope is 0. Just to the right, it is 1. The derivative jumps at the point where the two linear pieces meet. GELU's curve passes through zero with no such jump, so its slope changes continuously, which is the property the source connects to easier optimization.

The second consequence concerns neurons that receive negative input. Under ReLU, those neurons output zero, so they contribute nothing to the learning signal at that step. Under GELU, a neuron with negative input still produces a small output and still takes part in learning, though to a lesser degree than a neuron with positive input. This is the reason the source gives for GELU keeping such neurons involved in training.