Vectorized Implementations

One Rule per Entry

Many tensor operations can be understood by asking whether each output entry depends only on the input entry at the same position. An element-wise operation applies one rule independently to each tensor entry. ReLU follows this pattern: it examines an entry and applies the same ReLU rule to that entry without needing information from neighboring entries.

ReLU in Two Dimensions

A straightforward implementation makes the independence visible with two loops. The outer loop selects a row, and the inner loop selects each column in that row. At a selected position identified by row index i and column index j, the implementation applies the assignment max(x[i, j], 0). This assignment replaces a negative entry with zero and preserves a nonnegative entry.

Tracing a ReLU Pass

The index map emphasizes two facts. First, each input position has a corresponding output position. Second, ReLU changes only the entries whose values are negative. The positions of the entries do not move; the rule changes their values at those positions.

Why Copy Before Updating

A ReLU implementation copies the input before changing entries so that the updates are made to the copy rather than to the input tensor. The copied tensor becomes the output containing the ReLU results, while the original input remains available in its original form. Without this copy, updating entries would overwrite the input tensor.

From Loops to Parallel Work

Nested loops are a straightforward way to organize the work, but they are not the only organization possible. Since each tensor entry can be processed independently, the entries do not need to wait for neighboring entries to provide information. This makes element-wise operations highly amenable to massively parallel implementations. Implementations organized in this way are also called vectorized implementations, a term associated with vector processor supercomputer architecture.

Debugging by Position

When an element-wise result is incorrect, do not treat the entire tensor as one undifferentiated result. Find the entry where the result first differs from what you expected. The loop identifies that location with row index i and column index j. At that position, compare the old value, the ReLU rule, and the new value. This gives debugging a precise location and a small sequence of facts to inspect.

• Treating ReLU as if it used neighboring entries.

  The element-wise ReLU rule examines each selected entry independently.

  Fix: Inspect only the value at the current row and column, then apply max(x[i, j], 0).

• Assuming every entry changes.

  The assignment preserves nonnegative entries and replaces only negative entries with zero.

  Fix: Check the sign of each selected entry before deciding whether its value changes.

• Updating the input when the intended implementation should preserve it.

  The input is overwritten, so the original tensor is no longer preserved.

  Fix: Copy the input first and apply the updates to the copy.

• Looking only at the final mismatch without recording its position.

  The loop processes a specific position at a time, so the row and column identify where to compare the old value, rule, and new value.

  Fix: Record the position as [i,j] and inspect that entry's transformation.

Check Your Trace

Key Takeaways

1. An element-wise operation applies one rule independently to each tensor entry.
2. Nested loops make the row-and-column processing order explicit for a two-dimensional tensor.
3. The ReLU assignment max(x[i, j], 0) replaces negative entries with zero and preserves nonnegative entries.
4. Because entries do not require information from neighboring entries, element-wise work is suitable for massively parallel or vectorized implementations.
5. Copying before modification preserves the input tensor while producing changed ReLU results in a separate output.