Braintrust, evals and observability in one product

It treats evaluation and observability as one problem, which they are, and most of the market still sells them as two. Traces from production, datasets built by annotating those traces, a playground for trying a change against them, and experiments comparing runs all live in the same place, so the loop from a bad answer in production to a test case that would have caught it is short enough that people actually complete it. Human feedback and annotation are first-class rather than bolted on, which matters because the honest answer to most quality questions still involves a person reading outputs. The command line tool is built for repeatable scripted runs rather than clicking, so evaluations belong in your repository and your pipeline. And the free tier is unusually generous on the axis that normally hurts: unlimited users, projects, datasets, playgrounds and experiments, so nobody is priced out of collaborating while you are still deciding whether it works.

Fourteen day retention on the free tier undermines the thing you adopted it for. Regression tracking is a claim about time, and a tool that cannot tell you what last month looked like cannot support that claim; thirty days on the paid tier is better and still short against a quarterly engagement. Self-hosting exists only on the enterprise plan, which for forward deployed work is frequently decisive: when a client's data may not leave their network, the answer is not a sales conversation, it is a different tool. The metering compounds too, running on processed data, on scores and on model credits at once, so forecasting the cost of a large evaluation sweep before you run it takes real work and the number you quote a client can be wrong in several directions. And the core is not open source, so everything you build on it is a commitment to one vendor's continued terms.

Spend an hour on it if your evaluation practice already exists and is outgrowing scripts and a spreadsheet, and if the client permits their data in a third-party cloud. Skip it when data residency is the constraint, because the tier that solves that is a procurement project rather than an afternoon. If you adopt it on the free tier, export before day fourteen or accept that the history is gone.

Promptfoo plus Langfuse when the data must stay inside the client's network, which is most forward deployed work.