Weights and Biases in Neural Networks
A feedforward neural network is a directed acyclic graph whose nodes are neurons and whose directed edges are weighted connections.
Try it: Neural Network Forward Pass
How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.
How it works
- Each hidden neuron computes z = w1·x1 + w2·x2 + b.
- It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
- The output neuron does the same with the hidden activations as its inputs.
- Changing any input, weight or bias changes every value downstream of it.
Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.
Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).
Loading the simulation…
A One-Way Computational Route
A feedforward neural network is best understood as a one-way route for information. Values enter through the input layer, travel along directed weighted connections, pass through one or more hidden layers, and eventually reach the output layer. The network is not a loose collection of independent neurons: its neurons are nodes in a directed acyclic graph, and its connections determine how one stage supplies information to the next.
Layer Order and Information Flow
The full neuron set is divided into layer sets V0 through VT. V0 is the input layer. The hidden layers are the layers between the input and output layers, represented as V1 through VT-1. VT is the output layer. The index records processing order: information starts in V0, moves through the hidden layers in order, and finishes in VT.
| Layer | Role in the flow |
|---|---|
| Input layer V0 | Exposes the starting feature values and the constant value |
| Hidden layers V1 through VT-1 | Perform intermediate transformations |
| Output layer VT | Contains the network's final output |
The layer names describe positions and roles in the computation.
Features and the Constant Neuron
For an n-dimensional input space, the described input layer contains n feature neurons plus one additional constant neuron. Each feature neuron represents one input feature. The constant neuron always outputs 1. Because it supplies a fixed value through a weighted connection, it gives the network a constant contribution to a neuron's combined input; this is the role associated with the bias contribution in this representation.
Counting the Input Neurons
Consider an input space with n features. What belongs in the input layer?
Represent the features: Place one feature neuron in the input layer for each of the n input features.
Add the constant neuron: Add one more neuron whose output is always 1.
Interpret the extra connection: The constant neuron's weighted connection supplies a fixed contribution to downstream neurons.
The described input layer contains n feature neurons plus one constant neuron.
From Weighted Inputs to Neuron Output
A neuron receives outputs from earlier neurons through weighted connections. It combines those weighted incoming outputs and then processes the combined result with an activation function. The activation function therefore sits between the neuron's incoming information and the output that it passes to the next layer. The constant neuron's weighted contribution is included in the same combination as the other incoming contributions.
Tracing a Layered Network
Use a small network with an input layer, one hidden layer, and an output layer. The input layer exposes its feature values and the constant value. Directed weighted connections carry those outputs to hidden neurons. Each hidden neuron combines its incoming weighted outputs, applies its activation function, and sends its result forward. The output neuron then receives the hidden outputs, combines them through its own weighted connections, applies its activation function, and produces the final network output.
Following One Value Forward
Trace the route of an input feature through a network with one hidden layer.
Start at V0: The feature value is represented by a neuron in the input layer, alongside the constant neuron that outputs 1.
Reach the hidden layer: Directed weighted connections carry the feature output and the constant contribution to hidden neurons.
Process at a hidden neuron: Each hidden neuron combines its weighted incoming outputs and processes the combination with an activation function.
Continue to VT: The hidden outputs travel through directed weighted connections to the output neuron.
Produce the final output: The output neuron performs the same general pattern: combine weighted incoming outputs, apply an activation function, and expose the result in the output layer.
The value moves forward layer by layer; it does not travel backward or enter a cycle.
What do you think happens?
If a value begins in the input layer, can it skip the ordered feedforward route and return to an earlier layer before reaching the output layer?
Reveal answer
Answer: No, the network is organized as a directed acyclic graph with forward layer order.
The input layer starts the route, hidden layers lie between input and output, and the output layer finishes it. The directed acyclic structure prevents the information route from forming a cycle.
Mistakes in Reading the Graph
Treating the network as a collection of independent neurons
The network's computation depends on the graph formed by neurons and weighted communication links.
Fix:
Trace both the neurons and the directed connections between layers.Confusing the input layer with the first hidden layer
The input layer V0 has the special role of representing starting features and the constant value.
Fix:
Identify V0 first, then identify the layers between V0 and VT as hidden layers.Forgetting the constant neuron
The described input layer contains n feature neurons plus one constant neuron that always outputs 1.
Fix:
Include the constant neuron when describing the input layer.Stopping before the activation function
A neuron processes the combined incoming result with an activation function before sending its output onward.
Fix:
Separate the combination step from the activation step.Assuming layer names are merely labels
The layer index records the processing order of the feedforward route.
Fix:
Read the layer index as a guide to where information is in the computation.
Practice the Forward Route
Describe the route through a network with input layer V0, hidden layers V1 and V2, and output layer V3. Include the feature neurons, the constant neuron, the weighted connections, and the activation step at a neuron in V1 and at the output layer.
Hints
- Begin with the role of V0 and state what the constant neuron outputs.
- Name the order in which V1, V2, and V3 receive information.
- At each receiving neuron, mention weighted incoming outputs followed by the activation function.
Key Takeaways
- A feedforward neural network is a directed acyclic graph whose nodes are neurons and whose directed edges are weighted connections.
- The input layer is V0, hidden layers lie between the input and output layers, and the output layer is VT.
- For an n-dimensional input space, the described input layer contains n feature neurons and one constant neuron that always outputs 1.
- A neuron combines weighted incoming outputs and processes the combination with an activation function.
- Information moves forward from input to hidden layers to output; the layer structure records the order of that computation.
Key Takeaways
- A feedforward neural network is a directed acyclic graph of neurons connected by directed weighted edges.
- Input, hidden, and output layers describe the positions and roles of neurons in the forward flow.
- The input layer represents n features with n feature neurons and includes a constant neuron that outputs 1.
- Each neuron combines weighted incoming outputs and then applies an activation function.
- Tracing the layer order makes it possible to follow information from the input layer to the final output.