Weighted Sums in Neural Network Neurons
Activation functions determine how a neuron converts its scalar input into an output.
Try it: Neural Network Forward Pass
How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.
How it works
- Each hidden neuron computes z = w1·x1 + w2·x2 + b.
- It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
- The output neuron does the same with the hidden activations as its inputs.
- Changing any input, weight or bias changes every value downstream of it.
Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.
Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).
Loading the simulation…
From Predecessors to Output
A neuron does not send the combined influence of its incoming signals directly to its output. First, the neuron receives a scalar weighted sum from the connected predecessor neurons. It then applies an activation function to that single scalar input. The activation function is the rule that converts the neuron's combined input into its output.
The Neuron's Internal Sequence
- Outputs arrive from neurons connected to the receiving neuron.
- The receiving neuron combines those connected outputs into a weighted sum.
- That weighted sum is treated as one scalar input, represented here by a.
- The selected activation function receives a as its input.
- The activation function produces the neuron's output.
Tracing a Scalar Input
Following a Weighted Sum Through a Sign Function
Suppose connected predecessor neurons provide outputs that, after their weighted contributions are combined, give the receiving neuron the scalar input a. Determine what operation happens next when the chosen activation function is the sign function.
Aggregate: The predecessor outputs and their weights first produce the single scalar weighted sum a.
Apply the function: The sign activation function is applied to that scalar, using the rule σ(a) = sign(a).
Interpret the result: The output is determined by whether a is negative, positive, or at the sign boundary. It is not determined by applying the sign function independently to each predecessor output.
The receiving neuron follows the sequence weighted predecessor outputs, scalar input a, sign transformation, neuron output.
σ(a) = sign(a)Three Activation Behaviors
| Function | Behavior described in the source | What happens around a cutoff or sign boundary |
|---|---|---|
| Sign | Bases the output on the sign of the scalar input | Makes a sign-based transformation rather than a gradual one |
| Threshold | Makes a cutoff-based transformation | Changes behavior at a threshold |
| Sigmoid | Uses a smooth formula that approximates threshold behavior | Changes gradually rather than reproducing an abrupt cutoff exactly |
The sign and threshold functions are both decisive transformations, but they use different ideas. The sign function looks at the sign of its scalar input. The threshold function uses a cutoff. The sigmoid function is different in shape: it uses a smooth formula whose behavior approximates the threshold function rather than making the same abrupt cutoff.
Why Sigmoid Is Smooth
A threshold function is described as cutoff-based: its behavior changes at a threshold. The sigmoid function is described as a smooth approximation because it captures threshold-like behavior without making that change as an abrupt jump. As the scalar input moves through the transition region, the sigmoid changes gradually.
Boundary Cases and Conventions
Applying the activation function to each predecessor output separately
The neuron first receives a weighted sum from its connected predecessor neurons and then applies the activation function to that single scalar input
Fix:
Trace the weighted sum first, then apply the selected activation function to the resulting scalarTreating the weighted sum as the neuron's final output
The activation function still has to convert the scalar weighted sum into the neuron's output
Fix:
Continue from a to σ(a)Calling the sign function gradual
The source describes the sign function as a direct sign-based transformation rather than a gradual one
Fix:
Associate gradual behavior with the sigmoid description, not with the sign functionAssuming that every threshold convention is specified automatically
The source describes threshold behavior as cutoff-based but does not specify the exact output convention at the boundary
Fix:
Use the threshold definition supplied for the problem; if none is supplied, identify the boundary case instead of inventing its exact output
For the sign function, a negative scalar input has a negative sign and a positive scalar input has a positive sign. The source does not specify the convention for an input of exactly zero, so that boundary output should not be guessed. Similarly, a threshold function requires its threshold and boundary convention before an exact numerical output can be determined.
Trace It Yourself
A receiving neuron gets outputs from connected predecessor neurons. Their weighted contributions combine to form a scalar input a. Trace the neuron using the sign function. What should you inspect first, and what transformation is applied after the scalar has been formed?
Hints
- Begin with the combined scalar weighted sum, not with separate activation outputs for each predecessor.
- Use σ(a) = sign(a) after the weighted sum has been formed.
- If the value of a is not given, describe the result in terms of whether a is negative, zero, or positive.
Compare two descriptions: one function makes a cutoff-based transformation, while another uses a smooth formula that approximates that behavior. Identify the threshold and sigmoid descriptions, then explain the difference in the transition around the cutoff.
Hints
- Threshold is associated with cutoff-based behavior.
- Sigmoid is associated with a smooth approximation.
- The key contrast is abrupt change versus gradual change.
Key Takeaways
- A neuron first receives a scalar weighted sum from connected predecessor outputs.
- The activation function is applied to that one scalar weighted sum, not separately to each incoming value.
- The sign function performs a direct sign-based transformation.
- The threshold function performs a cutoff-based transformation.
- The sigmoid function uses a smooth formula that approximates threshold behavior through a gradual transition.
Key Takeaways
- Incoming predecessor outputs and their weights form a single scalar weighted sum.
- The activation function converts that scalar into the neuron's output.
- Sign behavior depends on whether the scalar input is negative, zero, or positive, with the zero convention requiring a definition.
- Threshold behavior is cutoff-based, while sigmoid behavior is a smooth approximation to that cutoff behavior.
- To trace a neuron correctly, aggregate first and activate second.