Promptfoo, the eval harness you can put in CI on day one

Configuration is a file, execution is a command, and both fit in a pull request. You describe prompts, providers and test cases, then run a matrix across all three from the CLI, as a library, or inside CI. It runs locally, which sounds like a detail and is actually the reason it survives a client's security review: the client's prompts and the client's data never leave the client's machine. The assertion types start deterministic, so contains, regex, JSON schema and a plain function get you real coverage before you ever pay a model to have an opinion. Providers span OpenAI, Anthropic, Azure, Google, Hugging Face and local models, so a comparison across two vendors is a config change rather than a project. Red teaming and vulnerability scanning live in the same tool, which matters in week two when the client asks about prompt injection and you would rather answer with a run than a paragraph. Zero to a failing test in one afternoon is a realistic expectation, not a marketing one.

The YAML gets away from you. Past roughly fifty cases with providers and variables, the config becomes a program written in a format that is not a programming language, and the usual response is to generate it, which means you now maintain a generator nobody asked for. Assertions graded by a model inherit that model's variance, so a red build can simply be the judge having an off run, and you discover this at six in the evening while the client waits. Cost is unmetered by default: a matrix across four providers is four bills, and nothing in the tool stops you or warns you first. The web viewer is built for a person reading one run, not for a system tracking many: there is no first-class history, regression tracking or ownership across runs unless you buy the enterprise tier or build that layer yourself. And red teaming produces findings rather than fixes, with a plugin list that grows faster than any team's ability to triage what it returns.

Spend an hour on it if you are the person who will be asked whether the output got worse after a prompt change, and you currently have no answer. That is every forward deployed engineer eventually. Skip it if your outputs are free-form prose with no checkable properties, because you would be paying a model to have opinions about another model. Whatever you do, run human review alongside it for the first month before trusting any grader.

Ragas if the question is specifically about retrieval quality. For everything else Promptfoo is the cheaper first move.