Revisiting Neural Network Parameters

A neural network is defined by its parameters, the learned numbers that the network adjusts during training. Before you can reason about training cost, memory, or model size, you need to know how many of these parameters a network has. This article works through one concrete network and counts its parameters two ways: once by hand, and once with code. The goal is not just to get the right answer but to build a habit of checking one method against the other.

The network in this example has two inputs and two outputs. It also has two hidden layers, with 30 nodes in the first hidden layer and 20 nodes in the second. Every count in this article refers to this same architecture, so you can follow the numbers from one section to the next without changing the network.

Read the architecture as a chain of connections. The 2 input values feed the first hidden layer of 30 units. That layer feeds the second hidden layer of 20 units, which feeds the 2 output units. Each arrow in this chain is a layer whose parameters must be counted, and the network's total count is built from those layers.

Start with the programmatic route. The code below builds the network, asks it for its parameters, keeps only the parameters that are trainable, and adds up how many numbers they contain. Read it once before the explanation that follows, and notice what the code counts: it counts individual numbers (elements), not parameter tensors.

The program reports 752. Keep that number in mind, because the next step asks you to reach the same total without running any code.

The total number of parameters in a neural network is found by summing the parameters in each layer. So the central task is to determine the parameter count for one layer and then repeat that for every layer. The source describes the rule for a single layer: the number of parameters is the product of the number of inputs and the number of units, plus the number of bias units.

In this formula, $n_{\text{in}}$ is the number of incoming values to the layer, $n_{\text{units}}$ is the number of units in the layer, and $n_{\text{bias}}$ is the number of bias units, which is equal to the number of units, so $n_{\text{bias}} = n_{\text{units}}$. Each unit receives every incoming value through its own weight, which is why the product appears, and each unit also carries one bias, which is why the bias term is added.

The same procedure applies to the remaining layers. The second hidden layer receives 30 values from the first hidden layer and has 20 units, so its weights number 30 x 20 = 600 and its bias units number 20, giving 620. The output layer receives 20 values and has 2 units, so its weights number 20 x 2 = 40 and its bias units number 2, giving 42.

Once each layer has its own count, the network total is the running sum from the first layer to the last. The layer counts 90, 620, and 42 add to 752, which matches the programmatic result. The diagram below shows how the total accumulates as each layer's count is added in sequence.

The first-layer breakdown is worth studying in detail. Its 90 parameters are not just 30 weights per unit. They come from 2 weights per unit across 30 units, which is 60, plus one bias for each of the 30 units. The diagram below breaks that single layer into its parts.

The manual method and the programmatic method count the same thing: every trainable parameter in the network. The programmatic version counts individual numbers through the model's own parameter list, and the manual version reconstructs the same count from the architecture. If the architecture you described by hand is the architecture the code built, the two totals must match. When they do not match, the mismatch itself is useful information, because it tells you that one of the two descriptions of the network is wrong.

When a manual total and a programmatic total disagree, avoid guessing at a fix. Instead, compare them layer by layer. Compute the manual count for each layer and print the element count for each parameter from the model. The first layer where the two sides disagree is the place to look. This approach turns a single wrong total into a specific location, which is far easier to fix.