NumPy Basics • Victor Guerra

One-Hot Encoding

Use the identity matrix trick — clean and handles batches naturally:

n_classes = 5
one_hot = np.eye(n_classes)[class_idx]           # single vector
one_hot_batch = np.eye(n_classes)[indices]        # batch, shape (N, n_classes)

For large n_classes, avoid the intermediate matrix:

one_hot = np.zeros(n_classes)
one_hot[class_idx] = 1.0

# batch version
one_hot_batch = np.zeros((len(indices), n_classes))
one_hot_batch[np.arange(len(indices)), indices] = 1.0

argmax

Returns the index of the maximum value:

np.argmax(a)              # global max index
np.argmax(a, axis=1)     # per-row max index (batch of logits)

Complement: np.argmin for minimum index.

Rounding Floats

round(3.14159265, 6)          # Python built-in → 3.141593
f"{3.14159265:.6f}"           # string formatting
np.round(arr, 6)              # numpy array

Converting Array Types

arr.tolist()        # numpy array → Python list of floats (for JSON etc.)
arr.astype(float)   # stays ndarray, ensures float64
arr.astype(int)     # float → int
arr.astype(np.int32)

Array ↔ Python List

Array → list:

arr.tolist()   # ✓ recursive; casts elements to NATIVE Python types (int/float)
list(arr)      # ✗ iterates only the first axis; elements stay numpy scalars (np.float64)

Use .tolist() — handles any dimensionality and produces native types (needed for json.dumps, which chokes on numpy scalars):

np.array([[1, 2], [3, 4]]).tolist()   # [[1, 2], [3, 4]] — nested
list(np.array([1.0, 2.0]))            # [np.float64(1.0), np.float64(2.0)] — not Python floats

List → array:

np.array([1, 2, 3])               # infers dtype (int64)
np.array([[1, 2], [3, 4]])        # nested → 2D
np.array([1, 2, 3], dtype=float)  # force dtype
np.asarray(my_list)               # skips copy if already an ndarray

np.array always copies by default; np.asarray avoids the copy when the input is already an ndarray of the right dtype — prefer it for “array-like” function inputs.
Ragged lists (unequal sublengths) → error in modern numpy (older: object array — avoid):
```
np.array([[1, 2], [3]])   # ERROR
```

Initializing a 2D Matrix with Random Values

Standard normal:

np.random.randn(rows, cols)    # legacy
rng = np.random.default_rng(seed=42)
W = rng.standard_normal((rows, cols))   # modern

Scale for neural network weights:

W = np.random.randn(rows, cols) * 0.01

Uniform distribution:

np.random.rand(rows, cols)              # [0, 1) — legacy
np.random.uniform(0, 1, (rows, cols))   # explicit range — legacy

# Modern approach
rng = np.random.default_rng(seed=42)
rng.uniform(0, 1, (rows, cols))         # [0, 1)
rng.uniform(-1, 1, (rows, cols))        # [-1, 1)
rng.random((rows, cols))                # [0, 1) — shorthand

np.random.default_rng

Modern random number generator (NumPy 1.17+). Preferred over legacy np.random.*.

rng = np.random.default_rng(seed=42)   # reproducible
rng = np.random.default_rng()          # random each run

Why preferred:

Explicit seed on the object — no global state interference
Higher quality algorithm (PCG64 vs Mersenne Twister)

Common methods:

rng.standard_normal((3, 4))
rng.random((3, 4))             # uniform [0, 1)
rng.integers(0, 10, (3, 4))
rng.choice([1, 2, 3], size=5)
rng.shuffle(arr)

Transpose / Permute Dimensions

A.transpose(0, 2, 1, 3)          # pass new order of axes
np.transpose(A, (0, 2, 1, 3))    # same

(0, 2, 1, 3) means: keep axis 0, swap axes 1 and 2, keep axis 3.

PyTorch equivalent:

A.permute(0, 2, 1, 3)   # arbitrary reordering
A.transpose(1, 2)        # swaps only two dims at a time

Applying a Function Element-wise

vf = np.vectorize(my_func)
vf(A)   # applies my_func to each scalar element

np.vectorize is syntactic sugar over a Python loop — not truly fast. Prefer native numpy ops:

A ** 2 + 1          # C-level vectorized — much faster
np.maximum(A, 0)

Use np.vectorize only when the function can’t be expressed with native numpy ops.

If the function needs the row index, build an index matrix or loop over rows:

# index matrix approach
rows, cols = A.shape
i_matrix = np.repeat(np.arange(rows).reshape(-1, 1), cols, axis=1)
vf(A, i_matrix)

# loop approach — cleaner
result = np.stack([my_func(row, i) for i, row in enumerate(A)])

Broadcasting

NumPy broadcasts automatically when dimensions are equal or 1:

row = np.ones((1, d))   # (1, d)
A   = np.ones((m, d))   # (m, d)
A + row                 # (1, d) → repeated m times automatically

Explicit broadcasting:

np.broadcast_to(row, (m, d))   # read-only view

PyTorch:

row + A                # same automatic broadcasting
row.expand(m, d)       # explicit

Vector (i,) → Column Vector (i,1)

v.reshape(-1, 1)        # numpy and pytorch — most portable
v[:, None]              # slicing with None adds a dim
v[:, np.newaxis]        # numpy — same as None, explicit
v.unsqueeze(1)          # pytorch
np.expand_dims(v, 1)    # numpy function form

Reshape

A.reshape(3, 4)      # explicit dims — total elements must stay the same
A.reshape(3, -1)     # -1 infers the missing dim automatically
A.reshape(-1)        # flatten to 1D

A.flatten()          # always returns a copy
A.ravel()            # returns view when possible (faster)

PyTorch equivalent:

A.reshape(3, 4)      # same API
A.view(3, 4)         # faster but requires contiguous memory

Logical Reductions & the “axis-that-disappears” rule

Applying a logical op per column/row and reducing = a boolean mask + any/all along an axis.

np.any(mask, axis=0)     # per COLUMN: True if ANY row is True   → shape (ncols,)
np.all(mask, axis=0)     # per COLUMN: True if ALL rows are True → shape (ncols,)
np.any(mask, axis=1)     # per ROW:    reduce over columns        → shape (nrows,)

Usually the mask comes from a comparison, then you reduce:

(x > 0).any(axis=0)      # per column: does any row exceed 0?
(x == 0).all(axis=0)     # per column: is every row zero? ("dead column" check)

The rule that prevents the off-by-one-axis bug: the axis you pass is the one that gets collapsed.

x shape (R, C):
  axis=0  → reduce rows    → (C,)   "per column"
  axis=1  → reduce columns → (R,)   "per row"

So “per column, reduce over rows” is axis=0 — it eats the row dimension and leaves the columns. Add keepdims=True only when you want the result to stay 2-D as (1, C) so it broadcasts back against the original (R, C) (same reason as the LayerNorm/log-sum-exp shift); by default the reduced axis is dropped, giving (C,).

(x <= 0).all(axis=0)                 # → (C,)      — drop reduced axis (default)
(x <= 0).all(axis=0, keepdims=True)  # → (1, C)    — keep it for broadcasting

Dead-neuron fraction (activations (batch, features); dead = ≤0 for every sample):

dead = (acts <= 0).all(axis=0)       # (features,) bool — reduce over batch (rows)
frac = dead.mean()                   # fraction True in [0, 1]

Mean and Variance

np.mean(x)                            # global mean
np.var(x)                             # variance (1/N by default)
np.std(x)                             # standard deviation

np.mean(x, axis=-1, keepdims=True)    # across features — keepdims for broadcasting
np.var(x, axis=-1, keepdims=True)

PyTorch:

x.mean(dim=-1, keepdim=True)
x.var(dim=-1, keepdim=True, unbiased=False)  # unbiased=False → 1/N (matches LayerNorm)

np.var uses 1/N (population variance) by default. PyTorch torch.var defaults to unbiased=True (1/(N-1)) — set unbiased=False to match numpy/LayerNorm.

Computing mean/variance: stability & passes

Two concerns hide behind “optimal”: numerical stability and number of passes. They can conflict — and the fast shortcut is the unstable one.

Correct default (keepdims=True so x - mu broadcasts — the LayerNorm pattern):

mu  = np.mean(x, axis=-1, keepdims=True)
var = np.var(x,  axis=-1, keepdims=True)   # ✅ np.var is internally two-pass & stable

Efficiency note: np.var computes the mean itself → calling mean + var computes the mean twice. If you need both, reuse mu:

var = np.mean((x - mu)**2, axis=-1, keepdims=True)   # reuses mu; still stable

The trap — the “one-pass” formula (DON’T):

var = np.mean(x**2, axis=-1, keepdims=True) - mu**2   # ⚠️ E[x²] − E[x]²

Algebraically correct but catastrophic cancellation: subtracts two large near-equal numbers. When the mean is large vs the variance (values ~1e6, variance ~1), you lose most significant digits and it can go negative. Centering first (x - mu) squares small numbers → no cancellation. That’s why the two-pass form is preferred and why np.var uses it.

Approach	Passes	Stable?	Verdict
`np.mean` + `np.var`	2 (mean twice)	✅	simplest default
reuse `mu`, `mean((x-mu)²)`	one mean saved	✅	if you need both
`mean(x²) − mu²`	1	❌	avoid
Welford’s algorithm	1, streaming	✅	data too big for memory

For in-memory arrays the two-pass (vectorized C) is stable and fast — premature single-pass optimization is where bugs come from. Welford’s only earns its keep for online/streaming data.

`torch.std` / `torch.var` correction (the N vs N−1 trap)

Same divisor gotcha for std. Two equivalent knobs:

x.std()                  # ⚠️ default: divisor N−1 (Bessel-corrected / unbiased / "sample")
x.std(correction=0)      # divisor N (population)  — newer API
x.std(unbiased=False)    # divisor N (population)  — older API, same thing

Mental model: divisor = N − correction.

correction=0 → ÷N → population std (treat data as the entire population).
correction=1 → ÷(N−1) → sample / unbiased std ← torch.std’s default.

Cross-library: np.std defaults to ÷N (ddof=0), but torch.std defaults to ÷(N−1) — so default-vs-default they disagree. torch.std(x, correction=0) matches np.std(x).

Reduce over all elements → pass no dim (scalar out). Pass dim= for per-axis stds. Both std/var need a float dtype. torch.std_mean(x) / torch.var_mean(x) return both in one pass.

Decoding an instruction like “population std over all elements, not Bessel-corrected”: → torch.std(x, correction=0) — correction=0 (population) + no dim (all elements).

Reduction Along an Axis

np.max(A, axis=0)              # max along rows → shape (cols,)
np.max(A, axis=1)              # max along cols → shape (rows,)
np.max(A)                      # global max → scalar

np.max(A, axis=1, keepdims=True)   # shape (rows,1) — preserves dim for broadcasting

Same API applies to: np.min, np.sum, np.mean, np.argmax, np.argmin.

NumPy vs PyTorch: NumPy uses axis, PyTorch uses dim — same concept, different keyword.

np.max(A, axis=1)     # numpy
torch.max(A, dim=1)   # pytorch

Multiplication Operations

a * b              # element-wise
np.multiply(a, b)  # same

np.dot(a, b)       # 1D vectors: inner product → scalar
a @ b              # dot product or matmul depending on shape (preferred)

np.matmul(A, B)    # matrix multiplication (m,k)@(k,n) → (m,n)
np.outer(a, b)     # outer product (m,)×(n,) → (m,n)

Operation	Inputs	Output
Element-wise	`(n,)` × `(n,)`	`(n,)`
Dot product	`(n,)` × `(n,)`	scalar
Matrix × vector	`(m,n)` × `(n,)`	`(m,)`
Matrix × matrix	`(m,k)` × `(k,n)`	`(m,n)`
Outer product	`(m,)` × `(n,)`	`(m,n)`
Batched matmul	`(b,m,k)` × `(b,k,n)`	`(b,m,n)`

@ is preferred for matrix/vector multiplication — works for 2D and batched cases. Use * only for element-wise.

Computing Norms

NumPy:

np.linalg.norm(v)              # L2 norm (default)
np.linalg.norm(v, ord=1)       # L1 norm
np.linalg.norm(v, ord=np.inf)  # L∞ norm

PyTorch:

torch.linalg.norm(v)           # L2 norm (default)
torch.norm(v)                  # older API, same result
torch.linalg.norm(v, ord=1)

ReLU

No built-in in NumPy — use:

np.maximum(x, 0)     # recommended
x * (x > 0)          # alternative
np.clip(x, 0, None)  # alternative

PyTorch:

torch.relu(x)
torch.nn.functional.relu(x)

Numerically Stable Sigmoid

NumPy has no built-in sigmoid. Options:

from scipy.special import expit   # recommended
expit(x)

# Manual stable implementation:
def sigmoid(x):
    return np.where(x >= 0,
                    1 / (1 + np.exp(-x)),
                    np.exp(x) / (1 + np.exp(x)))