Forward Propagation in Neural Networks
A feedforward neural network is a directed acyclic graph whose nodes are neurons and whose directed edges are weighted connections.
Try it: Neural Network Forward Pass
How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.
How it works
- Each hidden neuron computes z = w1·x1 + w2·x2 + b.
- It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
- The output neuron does the same with the hidden activations as its inputs.
- Changing any input, weight or bias changes every value downstream of it.
Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.
Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).
Loading the simulation…
A One-Way Route for Information
A feedforward neural network provides a one-way route for information. Values enter through the input layer, move through weighted connections, are processed by neurons, and eventually reach the output layer. The network is not a loose collection of independent neurons: its neurons and communication links form a directed graph, and the links determine how one stage supplies information to the next.
A Small Forward Trace
Consider a small network with two feature neurons in its input layer, two neurons in one hidden layer, and one neuron in its output layer. Information begins at the two features. Each hidden neuron receives outputs through directed weighted connections, combines those incoming values, and applies its activation function. The resulting hidden-layer outputs then travel through weighted connections to the output neuron, which performs the same kind of processing for the final stage.
Following one input through the stages
Trace the order of processing in a network with an input layer, one hidden layer, and an output layer.
Start at the input layer: The feature neurons expose the starting values. The input layer is the first layer in the forward route.
Pass values to the hidden layer: Directed weighted connections carry the input-layer outputs to hidden neurons.
Process at hidden neurons: Each hidden neuron combines its weighted incoming outputs and processes the result with an activation function.
Pass transformed values onward: The hidden neurons' outputs travel through directed weighted connections to the output layer.
Reach the output layer: The output layer contains the network's final output.
Forward propagation proceeds from input to hidden processing and then to output processing, without reversing direction.
The Layered Graph Structure
A feedforward neural network is a directed acyclic graph. Its nodes are neurons, and its directed edges are weighted connections. Directed means that each connection has a forward direction. Acyclic means that the network's structure does not form a loop through which information would return to an earlier stage. The layer index records the processing order: the input layer is V0, hidden layers are V1 through VT-1, and the output layer is VT.
| Layer | Notation | Role in the route |
|---|---|---|
| Input | V0 | Exposes the starting feature values |
| Hidden | V1 through VT-1 | Performs intermediate transformations |
| Output | VT | Contains the network's final output |
The layer names describe positions and roles in forward information flow.
Features and the Constant Neuron
For an n-dimensional input space, the described input layer contains n feature neurons plus one constant neuron. Each feature neuron represents one input feature. The additional constant neuron always outputs 1, so it is part of the input layer's set of available outputs as information begins its forward route.
What Each Neuron Does
A neuron receives outputs from incoming connections. It combines those incoming outputs according to the connections' weights and then processes the result with an activation function. The activation function therefore marks a processing step inside the neuron: the weighted incoming information is transformed before the neuron's output is passed to the next stage.
When tracing a network, name both parts of a neuron's work: first identify the weighted incoming outputs it combines, then identify the activation-function step that produces the output sent onward.
Mistakes in Forward Tracing
Treating the network as a collection of independent neurons
The directed edges determine how one stage supplies information to the next.
Fix:
Trace the route from the input layer through weighted connections to hidden and output neurons.Calling every layer a hidden layer
The input, hidden, and output labels identify different positions and roles in the processing order.
Fix:
Identify V0 as the input layer, V1 through VT-1 as hidden layers, and VT as the output layer.Leaving out the constant neuron
The described input layer contains n feature neurons plus one constant neuron.
Fix:
Include the constant neuron, whose output is always 1.Stopping a neuron's work after combining incoming values
A neuron combines incoming outputs and processes the result with an activation function.
Fix:
Record the activation function as part of each neuron's processing before following its output onward.Tracing information backward during forward propagation
Feedforward information moves through directed connections from the input side toward the output side.
Fix:
Begin at V0 and follow the direction of the connections toward VT.
Practice the Route
A network has an input layer, two hidden layers, and an output layer. Describe the order in which information moves through the network. In your description, include the role of the feature neurons, the constant neuron, the weighted connections, and the activation function inside each neuron.
Hints
- Start with V0 and identify what its neurons provide.
- Move through the hidden layers in their layer order.
- For each neuron, mention weighted incoming outputs followed by activation-function processing.
- Finish at VT, the output layer.
What do you think happens?
Before checking the answer, predict whether a forward trace should move from an output neuron back to a hidden neuron.
Reveal answer
Answer: No, because the route follows directed connections toward the output layer.
A feedforward network is described as a directed acyclic graph. Forward propagation starts at the input layer, passes through hidden layers, and reaches the output layer without looping back to an earlier stage.
The Forward-Propagation Checklist
- Locate the input layer V0 and identify its feature neurons and constant neuron.
- Follow the directed weighted connections from the input layer toward later layers.
- At each neuron, account for the combination of weighted incoming outputs.
- Include the activation-function step before following the neuron's output onward.
- Continue through hidden layers until the output layer VT contains the final output.
Key Takeaways
- A feedforward neural network is a directed acyclic graph of neurons connected by directed weighted edges.
- The input layer is V0, hidden layers lie between the input and output layers, and the output layer is VT.
- For an n-dimensional input space, the input layer contains n feature neurons plus a constant neuron that always outputs 1.
- Each neuron combines weighted incoming outputs and processes the result with an activation function.
- Forward propagation follows the network's one-way route from input values through hidden transformations to the final output.