Deep Learning Theory — Wide vs Deep, UAT • Victor Guerra

Why deep learning took off

Massive labeled datasets — DL needs large, diverse data to avoid overfitting.
GPU acceleration — made massively parallel matmuls practical.
Algorithmic unlocks: ReLU (avoided vanishing gradients), Dropout (overfitting), BatchNorm (stable/faster training), Adam (fast convergence, less tuning), residual connections (train very deep nets), pretraining/transfer learning (fewer labels).

Wide vs deep

Deep nets are more expressive than wide ones for the same parameter budget:

Depth → compositionality — each layer transforms the previous representation, building hierarchical features (low-level → high-level abstractions). Depth exponentially increases representational power: a deep net can represent some functions with far fewer parameters than a shallow one, and reuses intermediate features across layers.
Wide/shallow nets learn features in parallel, not hierarchically. They can approximate any continuous function (UAT below) but may need exponentially many neurons. Fine for simple, low-dimensional tasks.

Linear regions (how networks approximate functions)

A ReLU network is piecewise linear — it partitions input space into linear regions, and with more regions it approximates a continuous function ever more finely. A shallow net with D hidden units creates between 2^{D_in} and 2^D regions (the “joints” in the ReLUs are D hyperplanes whose intersections carve up the space). Depth’s payoff: deep nets produce many more linear regions per parameter than shallow ones — you can think of each layer as folding the input space, so regions get clipped and recombined, giving far more complex functions for a fixed parameter budget. This is why deep beats wide.

Universal Approximation Theorem (UAT)

A feed-forward net with a single hidden layer and enough neurons can approximate any continuous function on a compact domain to arbitrary precision. But the caveats are the point:

You have finite neurons/weights in practice.
Complex functions may need exponentially many neurons in a shallow net vs efficiently in a deep one.
Wide-shallow nets have poorly conditioned optimization landscapes → hard to find the right params.
A 1-layer net may overfit without capturing the true function.
Arbitrarily small error may require huge/tiny weights → gradient explosion/vanishing, instability.

UAT guarantees existence, not learnability or practicality. (Related: activation-functions on why non-linearity is required at all — stacked linear layers collapse to one.)