apply-bm25-ranking

Use when building or tuning full-text search relevance, scoring documents against a keyword query, or building the sparse/keyword leg of a hybrid RAG retriever — including deciding between BM25, raw T

SKILL.md

GitHub

Apply BM25 Ranking

Score documents against a keyword query using BM25's term-frequency saturation and document-length normalization — the default relevance function behind Elasticsearch and Lucene, and the keyword half of hybrid RAG retrieval.

Why This Is Best Practice

Adopted by: Apache Lucene (default Similarity implementation since 6.0) and Elasticsearch (default since 5.0) — meaning BM25 ranks search results for most production full-text search deployments built on either engine, including Wikipedia search (via Elasticsearch/CirrusSearch) and countless e-commerce and enterprise search products. It's also the standard sparse-retrieval baseline in IR research (TREC) and the keyword leg of hybrid retrievers in production RAG stacks (LangChain, LlamaIndex, Pinecone hybrid search).

Impact: Robertson & Zaragoza (2009) show BM25 consistently outperforms raw TF-IDF across TREC ad-hoc retrieval benchmarks because it caps the reward for repeated query terms and correctly normalizes for document length — two failure modes that make raw TF-IDF over-rank long, keyword-stuffed documents. In hybrid RAG systems, adding BM25 alongside dense/vector retrieval measurably improves recall on queries with exact-match terms (IDs, error codes, proper nouns, acronyms) that embeddings routinely miss.

Why best: Raw TF-IDF has no ceiling on term-frequency reward, so a document repeating the query term 50 times outscores one using it twice with better context — BM25's saturation term (k1) fixes this. Pure dense/vector search captures semantic similarity but is weak on exact lexical matches (SKU numbers, error codes, rare proper nouns) and requires an embedding model plus a vector index; BM25 needs neither training data nor a model, is cheap to compute, and is exactly why hybrid retrieval (BM25 + dense, merged via Reciprocal Rank Fusion) consistently beats either alone.

Sources: Robertson & Walker, SIGIR 1994; Robertson & Zaragoza, "The Probabilistic Relevance Framework: BM25 and Beyond," Foundations and Trends in Information Retrieval (2009); Apache Lucene BM25Similarity docs; Elasticsearch "Practical BM25" guide.

Steps

1. Understand the formula

For query Q with terms q1...qn, score for document D:

score(D, Q) = Σ IDF(qi) · ( f(qi, D) · (k1 + 1) ) / ( f(qi, D) + k1 · (1 - b + b · |D| / avgdl) )
  • f(qi, D) — term frequency of qi in D
  • |D| — length of D (in terms); avgdl — average document length in the corpus
  • IDF(qi) — inverse document frequency, down-weights terms that appear in most documents
  • k1 — controls term-frequency saturation (typical range 1.2–2.0)
  • b — controls document-length normalization strength (typical range 0.5–0.9, default 0.75)

2. Tune k1 (term-frequency saturation)

k1 controls how quickly additional occurrences of a term stop adding score:

  • k1 = 0 — term frequency ignored entirely (binary term presence)
  • Higher k1 — each additional occurrence keeps contributing more (approaches raw TF-IDF behavior)
  • Lower k1 — score saturates fast after the first few occurrences (recommended when documents are prone to keyword stuffing)

Only change the default (Lucene/Elasticsearch: k1 = 1.2) against a labeled relevance evaluation set — tuning blind on a handful of manual queries reliably overfits.

3. Tune b (length normalization)

b controls how much a document's length relative to the corpus average penalizes its score:

  • b = 0 — no length normalization; long documents aren't penalized for containing more terms
  • b = 1 — full normalization; score is scaled entirely relative to average document length
  • Default b = 0.75 is a reasonable middle ground for most corpora

Lower b when document length varies for legitimate structural reasons (e.g., some docs are naturally longer reference pages, not keyword-stuffed); raise b when longer documents in the corpus tend to just repeat terms more without adding real relevance.

4. Decide when to combine with dense/vector retrieval

Use BM25 alone when queries are keyword/exact-match heavy (IDs, codes, names, short factual queries) and there's no budget for an embedding model or vector index. Combine BM25 with dense retrieval (hybrid search) when queries mix semantic/paraphrase intent with exact-match terms — merge the two ranked lists with Reciprocal Rank Fusion (RRF) rather than a raw weighted-score blend, since BM25 and cosine-similarity scores aren't on comparable scales.

Dimension BM25 Raw TF-IDF Dense/vector search
Exact-match / rare terms Strong Strong Weak
Semantic/paraphrase queries Weak Weak Strong
Training data / model required None None Embedding model + index
Term-frequency saturation Yes (k1) No — unbounded N/A
Length normalization Yes (b) Inconsistent across implementations N/A
Compute cost Low Low Higher (embedding + ANN search)

5. Evaluate against a labeled set, not by eyeballing results

Build a small labeled query/relevant-document set before tuning k1/b or deciding whether to add hybrid search. Measure with standard IR metrics (NDCG, MRR, recall@k) before and after any change — perceived relevance from a few manual queries is not a reliable signal for whether a parameter change actually helped.

Rules

  • Never tune k1/b without a labeled relevance evaluation set — changes that look better on a handful of manual queries frequently regress on the full query distribution.
  • Don't blend raw BM25 scores with dense/vector similarity scores directly — the scales aren't comparable; use rank-based fusion (RRF) instead.
  • Don't treat a BM25 score as a probability or a bounded relevance percentage — it's an unbounded relative ranking score, useful only for ordering within one query's results.

Common Mistakes

  • Treating BM25 score as an absolute quality measure. Scores are only meaningful for ranking documents within the same query — they aren't comparable across different queries or usable as an absolute relevance threshold.
  • Ignoring document-length normalization on a corpus with highly variable document lengths. Skipping or misconfiguring b lets long documents dominate purely by repeating terms more often, independent of actual relevance.
  • Applying IDF weighting on a very small or narrow corpus. With few documents, IDF statistics are noisy and can produce unstable rankings — consider disabling or dampening IDF, or growing the corpus, before trusting the scores.
  • Tuning k1/b by manual inspection of a handful of queries. This overfits to those specific queries; use a labeled evaluation set and IR metrics instead.

When NOT to Use

  • Queries are predominantly semantic or paraphrase-heavy with little lexical overlap between query and relevant documents — pure dense/vector retrieval will outperform BM25 alone; consider hybrid or dense-only.
  • The result set is already small enough that exhaustive or manual scoring is cheaper and simpler than building a scored index.
  • The corpus is too small or too homogeneous for IDF to produce meaningful term discrimination (see Common Mistakes).