Data Representations for Neural Networks

From Raw Data to Learnable Structure

A neural network cannot learn directly from raw sound, images, text, or an unprepared table. Preprocessing changes raw data into numerical forms that the network can accept and learn from. The central representation is the tensor: a multi-dimensional array used to organize neural-network data.

Tensor Ranks and Axes

A tensor may be a scalar with no dimensions, a vector with one dimension, a matrix with two dimensions, or a higher-dimensional tensor with more dimensions. Dimensions and axes organize the values so that neural-network operations can act on them. The representation is not merely a container: its organization determines how values can be combined and transformed.

Operations That Change Tensors

Tensor operations are the transformations that let a neural network manipulate its representations and perform computations. Element-wise operations act on corresponding values. Broadcasting supports computation across dimensions when a value or smaller representation is applied across a larger one. A tensor dot operation combines values across dimensions. Reshaping changes the organization of values without being described as a new set of values. These roles should be kept distinct: some operations primarily change values, some combine values, and reshaping changes arrangement.

Why Learning Needs Gradients

Tensor operations provide the computation performed by a neural network, but computation alone does not explain learning. Learning uses gradient-based optimization to adjust the network's parameters and minimize its loss function. A gradient is described as derivative information from a tensor operation. It indicates how the computation changes in relation to the parameters, giving optimization a direction to use.

Stochastic gradient descent is one method used in this optimization process. Backpropagation chains derivatives so that the effects of tensor operations contribute to parameter updates. The connection is important: tensor operations produce the computation, their derivatives provide information about that computation, and stochastic gradient descent uses that information to move parameters toward a lower-loss region.

Preparing Values Before Training

Vectorization produces tensors from sources such as text, images, and tabular data. This step converts raw inputs and targets into numerical representations that neural-network computations can process. Both inputs and targets must be represented as tensors because the network's computations and learning procedures operate on tensor representations.

Neural-network data should usually contain small, relatively homogeneous values. Normalization keeps values small and helps avoid problems caused by incompatible feature ranges. Two related preparation ideas must be distinguished. Scaling places values into a small range. Standardization changes a feature so that its mean is 0 and its standard deviation is 1. Both address the relationship between feature magnitudes, but standardization specifically targets those two statistical properties.

Common Representation Mistakes

• Treating raw data as ready for a neural network

  The network requires numerical tensor representations.

  Fix: Preprocess the source through vectorization and other appropriate preparation steps.

• Calling every tensor operation a reshape

  Element-wise operations, broadcasting, tensor dot operations, and reshaping have different roles.

  Fix: State whether the operation changes corresponding values, applies across dimensions, combines dimensions, or changes organization.

• Using a meaningful zero as the missing-value marker

  The network cannot distinguish the two meanings from that representation.

  Fix: Use 0 as a missing marker only when 0 has no meaningful interpretation for the feature, and use the same convention consistently.

• Ignoring incompatible feature ranges

  Normalization helps avoid problems caused by incompatible feature ranges.

  Fix: Prepare values so they are small and relatively homogeneous, using scaling or standardization as appropriate.

Check Your Understanding

1. A neural network receives and transforms tensors rather than raw data. Scalars, vectors, matrices, and higher-dimensional tensors organize values by their dimensions. Element-wise operations, broadcasting, tensor dot operations, and reshaping perform different kinds of transformations. Tensor computations produce the derivatives used by backpropagation and stochastic gradient descent to update parameters and minimize loss. Vectorization creates tensor inputs and targets, while normalization helps keep values small and relatively homogeneous. Scaling creates a small range, standardization targets a mean of 0 and a standard deviation of 1, and missing values require a marker that cannot be confused with a meaningful data value.