A model is wrong. Which knob should move, and by how much?

Wrongness as a number

Take the simplest possible model: one parameter $w$, one input $x$, prediction $\hat{y} = w · x$. We want $\hat{y} ≈ y$ for the true value $y$. “Closeness” needs a number; the standard choice is squared error $L(w) = (w·x − y)²$ — zero when the prediction is exact, large when it’s far, and (importantly) smooth everywhere so we can take derivatives.

The widget fixes $(x, y) = (2, 6)$, so the loss is $L(w) = (2w − 6)² = 4(w − 3)²$ — a parabola with minimum at $w\_ = 3$. The whole problem reduces to: how do we get from any starting $w₀$ to $3$?

Which direction? — the derivative

The derivatives module gives the answer in one line: $L'(w) = 2x(wx − y)$ — the slope of the loss curve at the current $w$. If L'(w) > 0, $L$ is rising as $w$ grows; we should make $w$ smaller. If L'(w) < 0, the opposite. The descent direction is always $−L'(w)$.

In the widget, the green arrow at the current point shows that direction. As $w$ approaches the minimum, the arrow shrinks; right at $w\_$ it has zero length — there’s no signal left, because the gradient is exactly zero at the minimum.

# Toy: one parameter w, one example (x, y) = (2, 6).
# Loss(w) = (w·x − y)²   = quadratic in w, minimum at w* = y/x = 3.
def loss(w, x, y):
    return (w * x - y) ** 2

def grad(w, x, y):
    # d/dw [(wx − y)²] = 2x(wx − y).
    return 2 * x * (w * x - y)

x, y = 2, 6
loss(0.0, x, y)        # → 36     (way off)
loss(3.0, x, y)        # → 0      (perfect)
grad(0.0, x, y)        # → −24    (loss decreases as w grows)
grad(3.0, x, y)        # → 0      (no signal at the minimum)

How far? — the learning rate

The direction is settled. The size still isn’t. We multiply the gradient by a positive number $η$ (the learning rate · notation across fields·4) and subtract:

w  ←  w  −  η · L'(w)

Different $η$ shapes the trajectory. Try $η = 0.04$ in the widget: each step shrinks the distance to w_ by a constant factor (about 0.68 here), so convergence is geometric but slow. $η = 0.12$ is near the Newton-optimal $η = 1/L''(w_) = 1/8 = 0.125$ — convergence in two or three steps. $η = 0.27$ is past the boundary; we’ll see what happens to it next.

Iterate — the descent loop

The whole method is just step → step → step until either the gradient is small enough (we’re near a minimum) or a budget runs out:

for _ in range(steps):
  w = w − η · L'(w)

Five lines of code. That’s gradient descent. The widget runs exactly this loop when you press auto. Press it with $η = 0.12$ and watch $w$ converge to $3$. Press it with $η = 0.27$ and watch the iterates leave the screen.

# The descent loop, in five lines. The shape that scales to neural nets.
def descent(w0, lr, x, y, steps=20):
    w = w0
    for _ in range(steps):
        w = w - lr * grad(w, x, y)
    return w

descent(0.0, lr=0.04, x=x, y=y, steps=20)   # → 2.95   slow but stable
descent(0.0, lr=0.12, x=x, y=y, steps=20)   # → 3.00   fast, near-Newton
descent(0.0, lr=0.27, x=x, y=y, steps=20)   # → ~10⁵  diverged

# The single requirement for stability on a quadratic with second
# derivative c is η < 2/c. Here c = L''(w) = 2x² = 8, so η < 0.25.
# Beyond that, every "step" overshoots more than it corrects, and
# the iterates explode geometrically.

Diverge — when η goes too far

For our quadratic loss, write u = w − w_. The update rule becomes $u ← (1 − η · L''(w_)) · u$ — the distance to the minimum gets multiplied by $r = 1 − η · L''(w_)$ at each step. Stable convergence requires |r| < 1, i.e. 0 < η < 2/L''(w_). For our specific loss, $L''(w) = 8$, so any η < 0.25 converges and any η > 0.25 diverges.

Above $0.25$, each step overshoots the minimum by more than it corrected for. The signs of $u$ alternate (we land on opposite sides) and the magnitude grows. The widget makes this picture: at $η = 0.27$, the iterates ping-pong and balloon. The fix in real ML is never “trial and error” alone — it’s measuring local curvature (second-order methods, Adam-style adaptive rates) and using a schedule that shrinks $η$ as the optimum gets closer.

Why this is everywhere in ML

Real ML doesn’t have one parameter; it has millions or billions. It doesn’t have one example; it has millions of them. It doesn’t have a clean parabolic loss; it has a high-dimensional, non-convex landscape full of saddles and plateaus. None of that changes the recipe — it just changes what each symbol stands for.

• One parameter → vector. $w$ becomes a parameter vector $θ$; the gradient becomes a vector of partial derivatives. Vector subtraction in place of the scalar update — the same two operations the vectors module walks through (add, scale).

• One example → mini-batch. Sum (or average) the loss over a small random sample of examples each step. The gradient is now noisy — that’s $SGD$. The noise is a feature: it helps escape bad local geometry.

• Clean parabola → real loss surface. Cross-entropy on a deep network is non-convex. There is no single optimum; we settle for “good enough” local minima , found by the same descent loop.

• Hand-derived gradient → autograd. The chain rule is mechanical. Modern ML frameworks build a computation graph during the forward pass and walk it backward to produce $∇L$ automatically. The derivative we computed by hand for the toy is the same operation, scaled.

The bridge to Confidently Wrong: the squared loss here is the warm-up. The cross-entropy loss there has the same “compute gradient, take a step” geometry — just with $−\log p_true$ instead of $(\hat{y} − y)²$. Same descent algorithm, different landscape.

# Same loop, real ML.
# Replace the toy parameter w with a parameter VECTOR θ.
# Replace the toy gradient with the partial derivatives ∂L/∂θᵢ.
# Replace the single example with a SUM (or mini-batch average) over data.
#
# Pseudocode for a one-layer linear classifier with cross-entropy loss
# (the loss from /ml/confident-wrong) — same descent, more axes:
#
# for batch in data_loader:
#     ŷ = softmax(W @ batch.x)            # forward
#     L = cross_entropy(ŷ, batch.y)       # loss
#     g = autograd.grad(L, W)             # ∇_W L  via reverse-mode autodiff
#     W = W - lr * g                       # one step
#
# autograd ≠ a new idea; it's *bookkeeping for the chain rule* applied
# to the same gradient operation arc 3 derives by hand. Walk downhill,
# but with millions of axes and a clock.