Hand-Calculating Layer Connections
The source's network has two inputs, two outputs, and two hidden layers of 30 and 20 nodes, and its code reports 752 trainable parameters.
Running the Snippet Before the Theory
This article starts with a number. A worked exercise on a small neural network creates a model, counts its trainable parameters, and prints the total. When the snippet runs as written, it reports 752. Keep that number in mind, because the rest of the article explains where it comes from.
The line that does the counting is the sum expression. Read it once from top to bottom, then notice that it produces a single number rather than a list of parameters.
model.parameters()supplies each parameter object in the modelp.numel()reports how many values that parameter object holds- if p.requires_grad keeps only the trainable objects
sum(...)adds the surviving counts into one total
What do you think happens?
Before reading the theory, what do you think the snippet's printed total depends on: the code's exact syntax, or the sizes of the layers in the network?
Reveal answer
The sizes of the layers in the network
The source says the number of inputs, outputs, and hidden layers together fix the architecture, and the architecture fixes how many trainable parameters the model holds. The code only counts what the architecture already contains.
Counting Connections Layer by Layer
The source's network has two inputs, two outputs, and two hidden layers, with 30 nodes in the first hidden layer and 20 nodes in the second. The layers are connected in order. The inputs feed the first hidden layer, the first hidden layer feeds the second, and the second feeds the outputs. Each arrow in that chain carries learned values, so each connection between two layers contributes to the parameter total.
The source does not state the per-layer labels for these connections. They come from one common assumption: each layer is fully connected to the next, meaning every node in one layer links to every node in the following layer, and each node in the receiving layer carries one bias value. Under that assumption, a layer with $n_{in}$ inputs and $n_{out}$ outputs has $n_{in} \times n_{out}$ weights and $n_{out}$ biases.
The diagram shows one connection set per arrow. Each arrow is a complete layer-to-layer link, not a single line between two nodes. The first arrow carries the 2 input nodes into the 30 hidden nodes, which is 2 times 30 weights plus 30 biases.
Totaling the Source Network
Count the trainable parameters for a network with 2 inputs, hidden layers of 30 and 20 nodes, and 2 outputs, assuming full connections and one bias per receiving node.
- 1Layer 1: inputs to hidden layer 1
With $n_{in} = 2$ and $n_{out} = 30$, the weights are $2 \times 30 = 60$ and the biases are 30. The layer total is $60 + 30 = 90$.
- 2Layer 2: hidden layer 1 to hidden layer 2
With $n_{in} = 30$ and $n_{out} = 20$, the weights are $30 \times 20 = 600$ and the biases are 20. The layer total is $600 + 20 = 620$.
- 3Layer 3: hidden layer 2 to outputs
With $n_{in} = 20$ and $n_{out} = 2$, the weights are $20 \times 2 = 40$ and the biases are 2. The layer total is $40 + 2 = 42$.
- 4Add the three layer totals
$90 + 620 + 42 = 752$. This matches the total the snippet reports.
752 trainable parameters, which matches the code output.
Inside the Sum Expression
The sum expression works like a loop. It visits each parameter object in the model, asks how many values that object holds, and adds those counts together. The condition at the end filters the loop so that only trainable objects are counted. Each piece of the expression has a separate job, and the final number depends on all of them.
Predicting a Total for a New Architecture
Now apply the same method to a different network. This one has 3 inputs, a single hidden layer of 4 nodes, and 1 output. Work through the layers before you look at the answer.
Totaling a Three-Layer Network
Count the trainable parameters for a network with 3 inputs, one hidden layer of 4 nodes, and 1 output, assuming full connections and one bias per receiving node.
- 1Layer 1: inputs to the hidden layer
With $n_{in} = 3$ and $n_{out} = 4$, the weights are $3 \times 4 = 12$ and the biases are 4. The layer total is $12 + 4 = 16$.
- 2Layer 2: hidden layer to the output
With $n_{in} = 4$ and $n_{out} = 1$, the weights are $4 \times 1 = 4$ and the biases are 1. The layer total is $4 + 1 = 5$.
- 3Add the layer totals
$16 + 5 = 21$.
21 trainable parameters under the fully connected, one-bias-per-node assumption.
What do you think happens?
Using the method above, what total would you predict for a network with 3 inputs, one hidden layer of 4 nodes, and 1 output?
Reveal answer
21
The first layer gives 12 weights and 4 biases, which is 16. The second layer gives 4 weights and 1 bias, which is 5. Together they make 21, under the fully connected assumption the source uses.
Replace the architecture in the snippet with your own network's sizes, run it, and compare the printed number with your hand calculation. Write down the layer totals you expect before you run the code, so that you can see exactly where your count and the code's count split.
Hints
- Compute
n_intimesn_outfor each pair of adjacent layers before adding any biases. - Add exactly one bias for each node in the receiving layer, not the sending layer.
- Total the layers in order so you can compare them one at a time with the code.
Finding Where a Hand Count Disagrees
When your hand total and the code output differ, do not recount everything at once. The source's advice is to compare layer by layer, because that is the quickest way to find the layer where the two counts diverge. The flowchart below sets out the order to check the likely causes.
Common mistake Counting only the weights and forgetting the biases
What it looks likeFor the layer from 2 inputs to 30 hidden nodes, writing 60 instead of 90.
Why it failsThe method under discussion gives each receiving node one bias, so every layer adds $n_{out}$ to its weight count.
Do this insteadCompute $n_{in} \times n_{out} + n_{out}$ for every layer.
Common mistake Using the sending layer's size for the bias count
What it looks likeFor the layer from 30 hidden nodes to 20 hidden nodes, adding 30 biases instead of 20.
Why it failsThe bias count follows the number of output nodes, which are the nodes in the receiving layer.
Do this insteadAdd biases equal to the size of the layer the connection feeds into.
Common mistake Forgetting that requires_grad changes the total
What it looks likeExpecting the same total after marking a layer as not trainable.
Why it failsThe filter skips any parameter object whose
requires_gradis False, so its count is removed from the sum.Do this insteadCheck the requires_grad state of each parameter before comparing your total with the code.
Common mistake Adding the layer totals before checking each one, which hides the layer where the mismatch begins
Trainable and Frozen Parameters Side by Side
The filter gives the total its meaning. A parameter object with requires_grad set to True is trainable and is counted. A parameter object with requires_grad set to False is skipped. The architecture stays the same in both cases, so the difference between the two totals is exactly the count of the skipped objects.
| Setting | Included in the sum? | Effect on the total |
|---|---|---|
| requires_grad is True | Yes | Its value count is added |
| requires_grad is False | No | Its value count is skipped |
How the requires_grad filter changes which parameter objects are counted
Key Takeaways
- The snippet's total of 752 comes from the source network's architecture: 2 inputs, hidden layers of 30 and 20 nodes, and 2 outputs.
- Under a fully connected, one-bias-per-node assumption, each layer contributes $n_{in} \times n_{out} + n_{out}$, and the layers 90, 620, and 42 add to 752.
- The sum expression visits each parameter object, counts its values with
p.numel(), and keeps only those withrequires_gradset to True. - When a hand total differs from the code, compare layer by layer, starting with the layer sizes, then the weights, the biases, and the filter.