Why does Google show those results in that order?

Counting matches ranks docs by length

Search $"the cat sat"$. Every doc has $the$. The longest doc has it most. Counting matches ranks documents by how many words they contain — not by how relevant they are. We need a score that knows which words mean something.

TF — count, divided by length

Term frequency : $tf(t, d) = count(t, d) / length(d)$. The fraction of words in $d$ that equal $t$. Without dividing by length, a long doc would win every comparison just by repeating every word more often. Length normalization is “level the playing field” — same evidence per word, regardless of how many words there are.

# tf — count, normalized by doc length.
docs = [
    "the cat sat on the mat",
    "the dog ran fast",
    "cat and dog are friends",
    "the quick brown fox",
]
toks = [d.split() for d in docs]

def tf(term, doc):
    return doc.count(term) / len(doc)

# 'cat' in each doc — note doc 0 is longer, so its tf is smaller
# even though both contain 'cat' once.
[round(tf("cat", d), 3) for d in toks]
# → [0.167, 0.0, 0.2, 0.0]
#
# That length normalization is the whole reason: doc 2 "wins" between
# the two cat-containing docs because its sentence is shorter.

IDF — rarity is signal

Inverse document frequency : $idf(t) = \log₂(N / df(t))$, where $N$ is the corpus size and df(t) counts how many docs contain $t$. Walk the extremes:

• $df = N$ (every doc has it): $\log₂(1) = 0$ bits. Stopword. Zero contribution.

• $df = 1$ (only one doc has it): $\log₂(N)$ bits. Maximum signal. The query word singles out that doc.

In the 4-doc widget above: $"the"$ shows up in 3 of 4 docs, so its IDF is $\log₂(4/3) ≈ 0.42$ bits — almost nothing. $"fox"$ shows up in 1 of 4, so its IDF is $\log₂(4/1) = 2$ bits. The same word “fox” is worth 5× more to a query than “the”, and nobody hard-coded that.

from math import log2

N = len(toks)

def df(term):
    return sum(1 for d in toks if term in d)

def idf(term):
    d = df(term)
    return log2(N / d) if d else 0.0

[(t, df(t), round(idf(t), 2)) for t in ("the", "cat", "fox")]
# → [('the', 3, 0.42),    # 3 of 4 docs → barely any signal
#    ('cat', 2, 1.00),    # half the corpus → 1 bit
#    ('fox', 1, 2.00)]    # rarest → 2 bits
#
# A query word that lives in every doc gives 0 bits — the score
# learned to ignore stopwords without anyone hard-coding them.

Why log? — the same log as in entropy

The log is not a smoothing trick. Treat $df/N$ as “the probability that a random doc contains $t$”. Then

That is exactly the self-information from the entropy module: the bits of surprise you’d feel if a random doc contained that term. A “surprising” word — appears in few docs — carries many bits. A “boring” word — appears in nearly all of them — carries near zero. IDF is just self-information wearing a different hat.

The frame is heuristic — it pretends “random doc” is uniform over the corpus and that presence is binary. Real corpora are messier. But the intuition the frame delivers is rigorous, and it’s the reason the log isn’t going anywhere: rarity logarithmically beats frequency, every time.

Why product (tf × idf)? — borrowing from log-probability

Independent events have probabilities that multiply; their logs add. So if a doc had a probability for each query term independently, the log of the joint probability would be a clean sum: $Σᵢ −\log P(tᵢ)$. TF-IDF imitates that shape:

But the imitation is not the thing. $tf$ is a count, not a probability. Multiplying it by $idf$ is not literally taking the log of a joint probability. TF-IDF is a useful ranking score that borrows the “logs of independent probabilities add” shape — not a real likelihood. Treat it as a heuristic motivated by information theory; that is exactly the right amount of trust to give it.

# The score: tf · idf, summed over query terms.
def score(query, doc):
    return sum(tf(t, doc) * idf(t) for t in query.split())

ranking = sorted(
    range(N),
    key=lambda i: -score("the", toks[i]),  # try also "fox", "cat dog"
)

# Compare to the raw-count baseline.
def raw(query, doc):
    return sum(doc.count(t) for t in query.split())

# Query "the": raw count picks doc 0 (uses 'the' twice, longest doc).
# TF-IDF picks... barely anything — every score is near zero, because
# log2(4/3) ≈ 0.42 bits. The trap of "popular doc wins" is closed.
[round(raw("the", d), 3)   for d in toks]   # → [2, 1, 0, 1]
[round(score("the", d), 3) for d in toks]   # → [0.139, 0.104, 0, 0.104]

The vector view — cosine, not Euclidean

Pin a column for every word in the vocabulary. Each doc becomes a vector of TF-IDF weights — mostly zeros, a few non-zeros where the doc actually uses those words. (This sparse-vector representation is called bag of words : it keeps frequencies but throws away order.) The query is the same kind of vector. Ranking docs becomes “find the doc-vectors closest to the query-vector”.

Why cosine similarity , not Euclidean distance? Because doc length should not move the score. Take a doc $d$ and its concatenation with itself, $d ++ d$: the second has every component doubled — a vector pointing the same direction, twice as long. Cosine of the angle between the two: $1$ (identical direction). Euclidean distance: $\|d\|$ (non-zero). Doubling a doc shouldn’t change its meaning; cosine is the metric that agrees.

Why this became the paradigm — and what came after

TF-IDF won three races at once. It was fast (sparse vectors, inverted index), scalable (web-sized corpora fit in commodity hardware), and interpretable (every score has a per-term breakdown — exactly what the widget shows). It also produced rankings that were hard to embarrass: the model has one strong assumption (term independence) and a probability-flavored intuition, and the failure modes were familiar.

BM25 = TF-IDF + two empirical patches. (a) Saturation: the 5th occurrence shouldn’t count as much as the 1st; $tf$ is replaced by $(k₁+1)·tf / (k₁+tf)$, saturating around $k₁ ≈ 1.2$. (b) Length normalization: docs longer than the average get their tf damped further. Both patches were tuned on TREC evaluations in the 1990s and stayed.

The bridge to ML: TF-IDF was the first feature vector for text. Naive Bayes classifiers, logistic regression on text, latent semantic analysis — all stand on this representation. Modern dense neural retrievers (DPR, ColBERT, the dual-encoder family) replaced sparse TF-IDF vectors with dense embeddings, but they still benchmark against BM25. A new method that doesn’t beat the 30-year-old baseline on a real retrieval task is not a new method. The log is still there.