newsletter
Measure the simple version first
Before you make an AI system cleverer, measure the plain one. Twenty questions and one number will tell you more than a month of tuning.
This week's idea is small and it will save you a month. Before you make an AI system cleverer, measure the simple version. Almost nobody does, because measuring feels like admin and tuning feels like progress. So people spend weeks improving something they cannot prove got better, and then defend it in a meeting on instinct. We are going to fix that in about an hour, with a question set and one number.
Start from a finding. Researchers at Stanford published a paper with a deliberately unglamorous result: a plain retrieve-then-read pipeline, one that keeps the retrieved passages in their original document order and does nothing else at all, matched or beat two considerably more sophisticated systems across several long-context benchmarks. No reranker. No summarising tree. Retrieve, keep the order, read.
That is worth sitting with. The instinct when a retrieval system returns a wrong answer is to add something: a reranker, a cleverer chunker, a bigger model. Sometimes that is right. But if you never measured the simple version, you cannot say whether what you added helped, hurt, or did nothing while quietly costing money on every single request.
Here is the hour that fixes it. Write twenty questions someone would genuinely ask of your documents, and for each one write down which file the answer lives in. That list is the hard part and it is the whole asset. Then run your retrieval over those twenty and count how often the right file comes back in the top five. That number is a ceiling on your entire system, because a passage that never gets retrieved cannot be cited, however good the model reading it is.
Now change exactly one thing. Chunk size, or overlap, or a reranker. Run the same twenty. Write down what happened, including when it got worse. Especially when it got worse.
You will find two things. Some of your clever additions do nothing at all. And the failures cluster, usually around one document format or one kind of question, which tells you precisely where the next hour goes.
That is the loop. It is not sophisticated, and that is exactly why it works.
Three links
- Stronger Baselines for Retrieval-Augmented Generation. The paper above, in full. The simple baseline that matched the complicated pipelines. https://arxiv.org/abs/2506.03989
- The retrieval scorer. Our kit, about forty lines of Python, runs the loop above over your own documents in under thirty minutes. No database, no Docker.
- Hype check: does 95 percent of enterprise AI really fail? The number everyone quotes, and the much narrower thing it actually measured.