Backpropagation and Weight Updates

When you first build a neural network, it is tempting to give every weight the same starting value. Zero, or any single constant, looks tidy and predictable. If every weight matches, there is nothing to puzzle over when you print a weight matrix, and the setup feels like the safe choice. This article addresses that misconception directly: identical starting weights are a trap, not a convenience. The rest of the article explains why training needs each node to begin in a slightly different place, how small random starting values provide that difference, and how to make those random values repeatable when you need to compare or debug runs.

Before reading the explanation, try a prediction. Suppose two hidden nodes in the same layer start with exactly the same weights and receive the same input. What do you expect to happen to those two nodes after training updates the weights?

Identical starting weights make nodes in a layer perform the same operations and receive the same updates, so they never learn different features. Training therefore needs this symmetry broken. Each node must begin in a slightly different place so that gradient updates push different nodes toward different useful patterns. Only then can a network build the varied internal representations that let it map inputs to outputs in complicated ways.

The diagram shows the identical case in the top row and the case with differing starting values in the bottom row. The training signal is the same in both cases. The difference lies entirely in the starting point, and that small difference at initialization is what allows the layer to develop distinct features. The numbers 0.5 in the diagram are only placeholders for a shared starting value; the point is that the two nodes match.

Training repeats a cycle. The network produces an output from its current weights, the output is compared with what was wanted, and backpropagation determines how each weight should change to reduce the mismatch. The weight update step then applies those changes. The flow below shows that cycle, with the error signal carried backward from the output to the weights that produced it.

The source describes the consequence of this cycle for symmetric nodes. If two nodes share weights and receive the same input, the gradients computed for their weights are the same, so the update step moves them identically. Backpropagation cannot separate nodes that the starting values have made indistinguishable. This is why the starting values, not the update rule, must supply the initial difference.

The source specifies the remedy for symmetry: initialize weights with small random numbers. Each node then receives a different starting point, so the nodes can diverge during learning instead of acting as copies of one another. The source describes random initialization as the desired approach for this reason. Two properties of the starting values matter, and each does a different job.

Another property is easy to observe. Each time you create a network, its random values differ. Two separately instantiated networks do not start from the same weights, so the weight matrix you print will usually look different from one run to the next. That behavior is expected, and it is exactly the variation that breaks symmetry.

Run this code and the two printed matrices will usually differ, and the final comparison will usually print False. The exact numbers depend on the random stream, so they are not something to memorize. The lesson is that independent instantiations start from different values, which is the expected behavior.

Randomness has a practical cost. If every run produces different weights, you cannot easily compare results, debug a problem, or confirm that a change you made actually changed the outcome. The source resolves this by keeping the random values but making them repeatable. You seed PyTorch's random number generator with torch.manual_seed. Setting a seed before creating the network means the same sequence of random numbers is produced each time, so the initial weights come out the same across runs.

With both seed calls in place, the final comparison should print True. Reproducibility comes from resetting the generator before each draw, not from anything about the network itself. If you forget the second seed call, the two networks will differ, and that mismatch is a fact about the random number stream rather than a bug in the architecture.

The position of the seed call in the code matters. It must come before the network is created, because the weights are drawn at the moment of instantiation. A seed placed after the model is built has no effect on the weights that model already received. The worked example below traces this with a seed placed too late.