Layers and Connections in Feedforward Networks
Activation functions determine how a neuron converts its scalar input into an output.
Try it: Neural Network Forward Pass
How a neural network turns inputs into an output: each neuron computes a weighted sum plus bias, applies an activation function, and passes the result on.
How it works
- Each hidden neuron computes z = w1·x1 + w2·x2 + b.
- It applies an activation: ReLU keeps positive values and zeroes negatives; sigmoid squashes z into 0–1.
- The output neuron does the same with the hidden activations as its inputs.
- Changing any input, weight or bias changes every value downstream of it.
Default run (7 steps): Inputs x1 = 1, x2 = 0.5. Activation: ReLU. … Prediction y = ReLU(1.25) = max(0, 1.25) = 1.25.
Simplified: Tiny 2-2-1 network with hand-set weights, not a trained model. No training happens here. Activations offered: ReLU, sigmoid and sign (texts differ on the value of sign at exactly 0; here it is 0).
Loading the simulation…
From Many Signals to One Output
A neuron may receive outputs from several connected predecessor neurons. Those predecessor outputs first contribute to one scalar input through a weighted sum. The neuron does not send that combined influence directly to its output. Instead, it applies an activation function to the single scalar input, and that function determines the neuron's output.
Tracing the Scalar Input
The important boundary inside a neuron is the boundary between combination and activation. Signals from connected predecessor neurons are combined first. Their weights determine how they contribute to the weighted sum. That weighted sum is a single scalar input, which can be called a. Only after that scalar has been formed does the activation function act on it.
Following one neuron’s input
Suppose a receiving neuron has three connected predecessor outputs. Trace the information flow without assigning particular numerical values.
Collect predecessor outputs: The receiving neuron gets outputs from its connected predecessor neurons.
Form the weighted sum: Each predecessor output contributes through its connection weight. Together, these contributions form one scalar input.
Apply the activation function: The activation function receives that one scalar input rather than receiving the predecessor outputs separately.
Produce the neuron output: The activation function converts the scalar input into the neuron's output.
The path is predecessor outputs, weighted sum, activation function, and neuron output.
Three Ways to Transform a Scalar
An activation function is the rule that converts a neuron's scalar input into its output. The sign function bases that output on whether the scalar input is negative, zero, or positive. A threshold function makes a cutoff-based transformation. A sigmoid function uses a smooth formula that approximates threshold behavior.
| Function | What determines the output | Main behavior |
|---|---|---|
| Sign | The sign of the scalar input | Sign-based transformation |
| Threshold | Whether the input is on one side of a cutoff | Cutoff-based transformation |
| Sigmoid | A smooth formula applied to the scalar input | Gradual approximation of threshold behavior |
Reading Connections Across Layers
A feedforward network can be read as a sequence of connected layers: outputs from predecessor neurons provide contributions to neurons receiving them. For each receiving neuron, trace the connections into a weighted sum, then trace that scalar through the neuron's activation function to its output. This local trace can be repeated from one layer to the next.
Output Decisions at the Boundary
Classifying inputs by sign
Determine the sign-based result for a negative scalar input, a zero scalar input, and a positive scalar input.
Negative input: Because the input is negative, its sign-based output is negative.
Zero input: The output depends on the sign convention supplied by the definition. The source pack does not select a convention for zero.
Positive input: Because the input is positive, its sign-based output is positive.
The sign function separates negative and positive inputs by sign; a zero input requires an explicitly stated convention.
Reading a threshold question correctly
A problem asks for the output of a threshold function but gives only a scalar input and no cutoff.
Inspect the definition: A threshold function is cutoff-based, so its rule depends on a cutoff.
Check the supplied information: If the cutoff and boundary rule are missing, the exact output cannot be determined from the stated information.
State the limitation: Report that the threshold definition is incomplete rather than silently choosing a cutoff.
A threshold output is determinate only when the threshold rule, including its cutoff and boundary behavior, is supplied.
What do you think happens?
A neuron has already formed its weighted-sum scalar input. What happens next: does the neuron send that scalar directly to its output, or does it apply an activation function first?
Reveal answer
Answer: It applies an activation function first.
The neuron receives a weighted sum from connected predecessor neurons and then applies an activation function to that single scalar input. The activation function converts the combined input into the neuron's output.
Mistakes in Function Tracing
Treating the activation function as if it receives every predecessor output separately.
The activation function is applied to the one scalar weighted sum formed from the connected predecessor outputs.
Fix:
Trace the predecessor outputs into the weighted sum first, then apply the activation function once to that scalar.Sending the weighted sum directly to the neuron's output.
The weighted sum is the neuron's scalar input, not automatically its output.
Fix:
Continue the trace through the activation function.Confusing sign behavior with threshold behavior.
The sign function bases its output on the sign of its input, while a threshold function makes a cutoff-based transformation.
Fix:
Identify whether the rule asks about sign or comparison with a cutoff.Assuming a zero-input convention that was never stated.
The source pack does not specify the boundary convention for zero.
Fix:
Use the convention supplied by the exercise or state that the boundary case is unspecified.Calling the sigmoid an abrupt threshold.
The sigmoid uses a smooth formula and is described as a smooth approximation to threshold behavior.
Fix:
Describe threshold behavior as cutoff-based and sigmoid behavior as gradual.
Practice Trace
A receiving neuron has outputs arriving from connected predecessor neurons. Explain the processing order in four stages: incoming outputs, weighted sum, activation function, and final neuron output. Then state how the sign function treats a negative scalar input and a positive scalar input. Finally, explain why a sigmoid is called a smooth approximation to threshold behavior.
Hints
- The activation function acts on one scalar, not directly on the separate predecessor outputs.
- For the sign function, inspect whether the scalar input is negative or positive.
- Compare the gradual change of the sigmoid with the cutoff-based change of a threshold function.
- First identify the outputs of the connected predecessor neurons.
- Next identify the weighted sum that combines those contributions.
- Then apply the named activation function to the resulting scalar input.
- For sign, classify the scalar as negative, zero, or positive and check the zero convention.
- For threshold, locate the cutoff and check the rule at the cutoff.
- For sigmoid, emphasize that the change is smooth rather than an abrupt threshold switch.
Key Takeaways
- Connected predecessor outputs form a weighted-sum scalar input for a receiving neuron.
- The activation function is applied after the weighted sum and converts that scalar into the neuron's output.
- The sign function uses the sign of its scalar input, while the threshold function uses a cutoff-based rule.
- The sigmoid uses a smooth formula and approximates threshold behavior through a gradual transition.
- Exact boundary outputs require the relevant convention, cutoff, and threshold rule to be stated.
Key Takeaways
- A neuron first receives a weighted sum from connected predecessor outputs.
- The activation function then converts that one scalar input into the neuron's output.
- Sign and threshold functions make sign-based and cutoff-based transformations respectively.
- The sigmoid is a smooth approximation to threshold behavior because it changes gradually instead of switching abruptly.
- Always check the stated convention for zero, the cutoff for a threshold function, and the rule at the boundary.