Ragas, RAG metrics that start fast and need calibrating

It names the thing everyone argues about. Moving from vibe checks to a systematic evaluation loop is the exact conversation you have in week three of a deployment, and Ragas hands you the vocabulary for it. Crucially, it keeps separate scores for whether retrieval fetched the right context and whether the answer stayed inside the context it was given. That split is worth more than any individual number, because it converts the useless sentence, the bot is wrong, into one of two useful ones: retrieval missed, or the model wandered. Those have different fixes, different costs and often different owners, and knowing which one you have saves days. Several metrics run without ground-truth answers, so you can measure a pipeline against the client's real questions before anybody has built a labelled set, and the missing labelled set is usually the thing blocking evaluation entirely. Custom metrics are supported and it drops into the same test run as everything else.

The metrics are judged by a model, and that is not a footnote. Scores move between runs on identical inputs, so a number reported without a variance estimate is decoration, and the documentation is noticeably quieter about this than it is about the benefits. Every evaluation run is a second inference bill, and across a large test set that bill is not small, with nothing warning you before it arrives. Several of the more useful metrics still want reference answers, so the labelled set you were hoping to avoid comes back through a side door. Worst of all, the scores are seductive. A faithfulness score of 0.82 reads like a measurement and is in fact one model's opinion, uncalibrated against your client's domain, and someone who does not know that will paste it into a slide. Until you have checked it against human judgement on your own data, it is a direction, not a number.

Spend an hour on it if you have a retrieval pipeline and no way to tell whether last week's chunking change helped or hurt. Skip it if you have not yet written twenty real questions from the client's own domain, because the metric is only ever as good as the question set and Ragas will not tell you your set is too easy. Calibrate against human judgement before any score reaches a slide with a client's logo on it.

Promptfoo with deterministic assertions wherever the answer has checkable properties, which is more of your cases than you would expect.