Feedforward Neural Network Structure
Activation functions determine how a neuron converts its scalar input into an output.
Try it: Neural Network Forward Pass
How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.
How it works
- Each hidden neuron computes z = w1·x1 + w2·x2 + b.
- It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
- The output neuron does the same with the hidden activations as its inputs.
- Changing any input, weight or bias changes every value downstream of it.
Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.
Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).
Loading the simulation…
From Connections to Output
A neuron does not send the combined influence of its inputs directly to its output. Instead, it first receives one scalar input formed from a weighted sum of the outputs of connected predecessor neurons. It then applies an activation function to that single scalar input. The activation function is the rule that converts the neuron's combined input into its output.
Tracing the Weighted Input
The predecessor neurons provide outputs to the next neuron. Those connected outputs do not remain as separate inputs at the activation stage. Their weighted combination becomes one scalar value, often denoted by a. This scalar weighted sum is the input supplied to the activation function.
Following One Scalar Through a Neuron
Suppose the connected predecessor outputs and their weights combine to produce the scalar input a = 4. Describe the next step for the neuron.
Combine: The connected predecessor outputs have already been combined through their weighted sum, producing the single scalar input a = 4.
Apply: The neuron applies its selected activation function to that scalar input. The activation is applied to a, not separately to each predecessor output.
Produce: The result of the activation function becomes the neuron's output.
The neuron processes the sequence weighted predecessor outputs, then scalar input a, then activation function, then neuron output.
Three Activation Behaviors
The activation function determines how the neuron converts its scalar input into an output. The sign function makes a sign-based transformation. The threshold function makes a cutoff-based transformation. The sigmoid function uses a smooth formula that approximates threshold behavior rather than changing in an abrupt way.
| Function | How it uses the scalar input | Behavior described by the source |
|---|---|---|
| Sign | Bases the output on whether the scalar input is negative, zero, or positive | Direct sign-based transformation |
| Threshold | Uses a cutoff to make a threshold-style decision | Cutoff-based transformation |
| Sigmoid | Uses a smooth formula related to threshold behavior | Smooth approximation to threshold behavior |
The key conceptual difference is how each function transforms the one scalar input.
Reading Sign and Threshold Results
Classifying Three Scalar Inputs
A neuron uses either a sign-based activation or a threshold-based activation. Consider scalar inputs below zero, equal to zero, and above zero.
Sign function: For an input below zero, the sign function produces its negative-sign result. For an input equal to zero, it produces its zero-sign result. For an input above zero, it produces its positive-sign result.
Threshold function: A threshold function compares the input with its cutoff. The result is determined by which side of the cutoff the input occupies. If the cutoff or its equality convention is not supplied, the exact numeric output cannot be determined.
Interpretation: Both functions make a sign-based or cutoff-based transformation of the scalar input. Neither receives the predecessor outputs separately at this stage.
The sign result is determined from the input's sign. The threshold result is determined from the stated cutoff and its convention.
What do you think happens?
A neuron has already formed its scalar weighted-sum input. What should happen next: combine the predecessor outputs again, or apply the activation function to the scalar input?
Reveal answer
Answer: Apply the activation function to the scalar input.
The weighted sum has already produced the neuron's single scalar input. The activation function then converts that scalar input into the neuron's output.
Common Reasoning Mistakes
Treating the activation function as if it were applied separately to every predecessor output.
The neuron first receives a weighted sum from connected predecessor outputs and then applies the activation function to that single scalar input.
Fix:
Trace the neuron in order: predecessor outputs, weighted sum, scalar input, activation, neuron output.Assuming that the weighted sum is already the neuron's final output.
The combined influence is not passed directly to the output. The activation function must convert the scalar input first.
Fix:
After identifying the weighted sum, apply the selected activation function.Describing sigmoid as an abrupt threshold.
The source describes sigmoid as a smooth approximation to threshold behavior.
Fix:
Describe threshold behavior as cutoff-based and sigmoid behavior as a smooth change that approximates it.Inventing an output convention that has not been specified.
The source pack states the behaviors but does not specify those numeric conventions.
Fix:
State the sign category or explain which side of the supplied cutoff contains the input.
Practice Trace
A neuron receives outputs from several connected predecessor neurons. These outputs are combined into a scalar weighted sum a. Write the next two stages in the correct order, and explain how the answer would differ if the neuron used a sign function, a threshold function, or a sigmoid function.
Hints
- The first stage after the weighted sum is not another combination of predecessor outputs.
- The activation function receives the single scalar input a.
- Use sign-based, cutoff-based, and smooth-approximation language when comparing the three functions.
When tracing any neuron, write the intermediate scalar input explicitly before deciding the output. This keeps the weighted-sum stage separate from the activation stage and makes it easier to identify whether the neuron uses sign, threshold, or sigmoid behavior.
Key Takeaways
- A neuron receives a scalar weighted sum formed from the outputs of connected predecessor neurons.
- The activation function is applied after the weighted sum, not separately to each predecessor output.
- The sign function makes a direct sign-based transformation, while the threshold function makes a cutoff-based transformation.
- The sigmoid function uses a smooth formula that approximates threshold behavior.
- Exact numeric outputs require the relevant sign or threshold convention when the source does not specify one.