Join free

kits

The retrieval scorer

Forty lines of Python that score your retrieval against your own questions, so a chunking change stops being a matter of opinion and starts being a number.

Scores how often your retrieval actually fetches the right document for your own questions, and shows you every question where it did not.

Run it

  1. 1. Check your tools. You need Python 3.10 or newer. Run: python3 --version. Nothing else needs installing yet, and there is no database and no Docker in this kit on purpose.
  2. 2. Make a folder and an isolated environment. mkdir retrieval-scorer && cd retrieval-scorer python3 -m venv .venv source .venv/bin/activate pip install voyageai numpy On Windows PowerShell, activate with .venv\Scripts\Activate.ps1 instead of the source line.
  3. 3. Set an embedding key. We use Voyage here because the free tier is enough to finish this kit. export VOYAGE_API_KEY=your-key-here On Windows PowerShell: $env:VOYAGE_API_KEY = "your-key-here"
  4. 4. Put your documents in a folder called docs. Plain text or markdown, anything from twenty to two hundred files. Use real documents, ideally the ones your system will have to answer from. A tidy sample corpus will give you a high score and teach you nothing.
  5. 5. Save this as harness.py. import glob, json, os, sys import numpy as np, voyageai CHUNK_CHARS, OVERLAP, MODEL = 1000, 200, "voyage-3.5" vo = voyageai.Client() def chunks_of(text): step = CHUNK_CHARS - OVERLAP return [text[i:i + CHUNK_CHARS] for i in range(0, max(len(text), 1), step)] def embed(texts, kind): out = [] for i in range(0, len(texts), 128): out += vo.embed(texts[i:i + 128], model=MODEL, input_type=kind).embeddings m = np.array(out, dtype="float32") return m / np.linalg.norm(m, axis=1, keepdims=True) paths = sorted(glob.glob("docs/**/*.*", recursive=True)) if not paths: sys.exit("No files under docs/. Put your documents there first.") chunks, sources = [], [] for p in paths: for c in chunks_of(open(p, encoding="utf-8", errors="ignore").read()): if c.strip(): chunks.append(c) sources.append(os.path.basename(p)) print(f"{len(paths)} files, {len(chunks)} chunks") M = embed(chunks, "document") questions = [json.loads(l) for l in open("questions.jsonl", encoding="utf-8") if l.strip()] Q = embed([q["q"] for q in questions], "query") hits, misses = {1: 0, 3: 0, 5: 0}, [] for q, qv in zip(questions, Q): got = [sources[i] for i in np.argsort(-(M @ qv))[:5]] for k in hits: if q["must_cite"] in got[:k]: hits[k] += 1 if q["must_cite"] not in got: misses.append((q["q"], q["must_cite"], got[0])) n = len(questions) print(f"\nchunk={CHUNK_CHARS} overlap={OVERLAP} model={MODEL} n={n}") for k in (1, 3, 5): print(f" recall@{k}: {hits[k]}/{n} = {hits[k] / n:.0%}") if misses: print("\nmissed entirely (fix these, not the average):") for q, want, got in misses: print(f" {q}\n wanted {want}, top hit was {got}")
  6. 6. Write questions.jsonl, one JSON object per line, like this: {"q": "How do I rotate an expired service credential?", "must_cite": "credentials.md"} {"q": "What is the retention period for audit logs?", "must_cite": "logging-policy.md"} Write twenty of them. This step takes most of the thirty minutes and it is the step that decides whether any of the rest is worth anything. Write the questions someone would really ask, including the badly worded ones.
  7. 7. Run it: python harness.py
  8. 8. Change exactly one thing, CHUNK_CHARS or OVERLAP or MODEL, run it again, and put the two results side by side. One variable at a time, or the number tells you nothing.

What you get

A table showing recall at 1, 3 and 5 across your question set, and every question that missed entirely with the file it wrongly retrieved instead. Recall at 5 is the ceiling on your whole system: a chunk that never gets retrieved cannot be cited, however good the model reading it is. Read the misses rather than the average. They almost always cluster around one document format or one kind of question, and that cluster is your next fix. It is usually chunking, not the model.

Take it further

  • Swap the model name and re-run against the same chunks and the same questions. One variable, one number, and the cheapest embedding-model comparison you will ever run.
  • Add a reranker over the top twenty results and check whether recall at 3 actually moves. Often it does not, and learning that for the price of an hour is a good trade.
  • Replace the in-memory matrix with pgvector once the corpus outgrows a laptop, and keep this harness pointed at it so the old scores stay comparable to the new ones.
  • Add an answer-quality check on top, but keep the two numbers apart. Retrieval failures and generation failures need different fixes and different people.