Scaling Laws & Chinchilla

“Training Compute-Optimal Large Language Models” (Hoffmann et al., DeepMind, 2022) — the Chinchilla paper. One of the most consequential results in LLM training strategy.

The Question

Given a fixed compute budget C, how do you split it between:

• N — model size (parameters)
• D — training data (tokens)

Training compute scales roughly as:

$C \approx 6\,N\,D \quad \text{(FLOPs)}$

With C fixed, N·D is constant → the tradeoff is zero-sum: every FLOP spent on a bigger model is a FLOP not spent on more tokens.

C ≈ 6ND is accounting, NOT an optimality statement

A common conflation worth keeping straight — these are two separate things:

1. The identity (bookkeeping): C ≈ 6ND says what any run costs. It holds for every (N, D) combination — good and terrible ones alike.
2. The optimality result (Chinchilla’s contribution): among all (N, D) pairs costing the same C (a hyperbola: N·D = C/6), which achieves the lowest loss?
   • (huge N, tiny D) → under-trained → bad loss (Gopher’s corner)
   • (tiny N, huge D) → model too small to absorb the data → bad loss
   • in between → minimum loss ← N ∝ C^0.5, D ∝ C^0.5, ~20 tokens/param

At budget C, your loss is the best achievable only if you allocate Chinchilla-optimally. C determines the floor; the (N, D) split determines whether you reach it.

Formally: minimize the fitted L(N, D) subject to 6ND = C → the solution traces the compute-optimal frontier L(C) (lower envelope of what each budget can buy). Gopher and Chinchilla sat at the same C — Gopher above the frontier, Chinchilla on it. Same budget, different split, better model: proof the split (not the budget) was being wasted.

Visualizing the budget curve (the hyperbola)

Only two axes: x = N, y = D. There is no third axis — C/6 is just a constant (the value the product must equal). Fixing C imposes N·D = C/6 = k, same shape as x·y = k → a hyperbola in the N–D plane:

 D (tokens)
 │ ●  (tiny N, huge D — model too small)
 │  \
 │   \
 │    ●  ← Chinchilla-optimal point (D/N ≈ 20)
 │     `--.
 │         `---.
 │              `----●  (huge N, tiny D — Gopher's corner)
 └──────────────────────── N (params)
        every point on the curve: N·D = C/6  (cost exactly C)

• Slide right → bigger model, fewer tokens (the zero-sum trade). Slide left → smaller model, more tokens.
• Bigger budget → a hyperbola further out; budgets form a family of nested hyperbolas.
• Loss is a third quantity — picture L(N, D) as contour lines drawn over this plane. The Chinchilla question: walk along your budget’s hyperbola; where is the loss lowest? That tangency point is (N_opt, D_opt), at D/N ≈ 20.
• Gopher and Chinchilla are on the same hyperbola (same C); Gopher sits far down-right where loss is worse.

Worked example: GPT-3’s budget, Chinchilla-optimal split

GPT-3 sat at (N, D) = (175B, 300B tokens) → ratio ≈ 1.7 tokens/param. Where should it have been on the same hyperbola?

$N \cdot D = 175\text{B} \times 300\text{B} = 5.25\times10^{22} \qquad D = 20N$ $\Rightarrow\; 20N^2 = 5.25\times10^{22} \;\Rightarrow\; N = \sqrt{2.625\times10^{21}} \approx 51\text{B}, \quad D = 20N \approx 1.02\text{T}$

Same compute → a 51B model on ~1T tokens: 3.4× smaller, 3.4× more data, lower loss, and 3.4× cheaper at inference. (Both factors equal because moving from ratio 1.7 to 20 shifts each axis by √(20/1.7) ≈ 3.4.)

Method + sanity checks: substitute the ratio into the budget constraint → quadratic in N → don’t forget the square root. Then verify the candidate (N, D) against BOTH original equations (product = budget, ratio = 20) — catches algebra slips instantly.

Ratio arithmetic (interview check): double N at fixed C → D halves → the tokens-per-parameter ratio D/N drops 4× (numerator halves AND denominator doubles): $\frac{D/2}{2N} = \frac{1}{4}\cdot\frac{D}{N}$ e.g. 20 tokens/param → 5 tokens/param → under-trained model.

Before Chinchilla: Kaplan et al. (2020)

The earlier OpenAI scaling laws said: as compute grows, scale model size faster than data. This produced the era of giant under-trained models:

• GPT-3: 175B params, ~300B tokens → ~1.7 tokens/param
• Gopher: 280B params, ~300B tokens → ~1 token/param

Chinchilla’s Result

From a careful sweep (400+ models, three analysis methods): the loss-optimal split scales N and D equally:

$N_{opt} \propto C^{0.5} \qquad D_{opt} \propto C^{0.5}$

Rule of thumb: D ≈ 20·N — about 20 tokens per parameter (that’s just the ratio D/N ≈ 20; tokens per parameter is D/N).

Validation: Chinchilla (70B params × 1.4T tokens = 20:1 ) trained with the same compute as Gopher (280B × 300B) beat Gopher across the board — and is 4× cheaper at inference.

The fitted loss model

$L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}$

• E — irreducible loss (entropy of language)
• A/N^α — finite model-capacity error (α ≈ 0.34)
• B/D^β — finite data error (β ≈ 0.28)
• α ≈ β → roughly symmetric → balanced scaling is optimal.

Why It Mattered

1. Reset training recipes — post-2022 models train on far more tokens per parameter.
2. Canonical methodology — fit scaling laws on small runs, extrapolate to choose (N, D) for the big run.

Beyond Chinchilla: the Inference-Cost Correction

Chinchilla optimizes training compute only. Every serving request costs ∝ N (forward pass), and a deployed model serves billions of requests — so total cost of ownership favors training a smaller model far longer than the 20:1 optimum (“overtraining”):

• LLaMA-3 8B: ~15T tokens → ~1,875 tokens/param — deliberately ~90× past Chinchilla-optimal.
• You pay extra training FLOPs once to save inference FLOPs forever.

Key nuance: 20:1 is the training-compute-optimal ratio — a fitted constant, not a law of nature. The deployment-optimal ratio is much higher.

D and Training Time