Recurrent Artificial Neural Networks
Feedforward describes the absence of loops in an ANN's connections.
The Decisive Structural Question
A neural network may receive information at one side and eventually produce an output, but that description alone does not tell you whether it is feedforward. The decisive question is whether the network's connections contain a loop. A feedforward artificial neural network has no loop in its connections. If even one loop exists, the network is recurrent.
Feedforward describes the connection pattern of the network, not merely the fact that the network has an input and an output.
Layers and Information Flow
A feedforward ANN is organized into an input layer, one or more hidden layers, and an output layer. The input layer receives the input. The output layer produces the output. Any layer that is neither the input layer nor the output layer is a hidden layer. Hidden means that the layer is internal to the network; it does not mean that the layer is disconnected or inactive.
The layer arrangement helps you describe where units are positioned, but the absence of loops supplies the classification. A network can have identifiable input, hidden, and output layers and still be recurrent if its connections include a loop.
Connections and Weights
The units in an ANN are connected by links. Every connection has an associated real-valued weight. In terminology used for biological neural networks, a weight roughly corresponds to the efficacy of a synaptic connection. These weights are part of the network's specification, but they are not what determines whether the network is feedforward. That classification depends on whether the links form a loop.
| Feature | What it tells you |
|---|---|
| Layer organization | The network has input, hidden, and output positions. |
| Connection weight | Each link has an associated real-valued weight. |
| Loop structure | No loop means feedforward; at least one loop means recurrent. |
Separate the network's components from the structural test used for classification.
Classifying a Network Diagram
A loop changes the classification
Consider a network organized with an input layer, a hidden layer, and an output layer. Its connections go from the input layer to the hidden layer and from the hidden layer to the output layer. Then add one connection from a hidden unit back to an earlier unit.
Identify the layers: The entrance is the input layer, the internal layer is the hidden layer, and the exit is the output layer.
Inspect the connections: The forward connections describe movement through the layer arrangement, but the additional connection points back to an earlier unit.
Check for a loop: The backward connection creates a loop through which a unit's output can influence its own input.
Classify the network: The presence of even one loop means the network is recurrent rather than feedforward.
The network is recurrent. Its input, hidden, and output layers can still be identified, but the loop prevents it from being feedforward.
Common Classification Mistakes
Calling every network with an input and an output feedforward.
Having an input and an output is not sufficient. A recurrent network can also have input, hidden, and output layers.
Fix:
Inspect the connection pattern and determine whether it contains a loop.Assuming that a hidden layer is disconnected or inactive.
Hidden means that the layer is internal rather than the entrance or exit of the network.
Fix:
Identify a hidden layer as any layer that is neither the input layer nor the output layer.Using the weights to decide whether an ANN is feedforward.
Weights describe connections, but feedforward classification depends on whether the connections form a loop.
Fix:
Record that each link has a real-valued weight, then perform a separate loop check.Ignoring a single loop because most connections point toward the output.
The presence of even one loop changes the classification to recurrent.
Fix:
Treat any loop as decisive.
Practice the Loop Test
A network has an input layer, two hidden layers, and an output layer. Every connection proceeds through the arrangement toward the output except one connection that returns from the second hidden layer to the first hidden layer. Is the network feedforward or recurrent? Explain which structural feature determines your answer.
Hints
- First identify the input, hidden, and output layers.
- Then ignore the number of layers and inspect whether any connection creates a loop.
- A single loop is sufficient to determine the classification.
The practice network is recurrent because the connection returning to the first hidden layer creates a loop. The number of hidden layers does not determine the classification.
Summary
- A feedforward ANN is defined by the absence of loops in its connections. It is organized with an input layer, one or more hidden layers, and an output layer. The input layer receives input, the output layer produces output, and internal layers are hidden layers. Every connection between units has an associated real-valued weight. If even one connection creates a loop through which a unit's output can influence its own input, the network is recurrent rather than feedforward.
Key Takeaways
- Feedforward describes the absence of loops in an ANN's connections.
- A feedforward ANN has an input layer, one or more hidden layers, and an output layer.
- Each connection between units has an associated real-valued weight.
- The presence of even one loop makes the network recurrent.
- Classify the network by inspecting its connection pattern, not merely by observing that it has inputs and outputs.