career bridge
The regression net
Systems like this do not break loudly, they get worse. Build the net that notices, and watch it fail once on purpose.
You have a system that is fast enough and affordable enough. Now the quiet risk. A prompt edit, a library bump, or a provider quietly updating a model can degrade it, and nothing will tell you. Systems like this rarely break loudly. They get worse.
What you do
Build the net that catches it. Write a test set from real usage rather than imagination: pull a hundred actual requests out of your logs, spread across the shapes you found in the last stage, and record what a good response contains for each. Start with deterministic checks wherever the output has a checkable property, because they are free, stable and never have an opinion. Valid JSON. Required fields present. A cited source that genuinely exists in the corpus. No forbidden string. Inside a length bound. Reach for a model-graded check only where nothing deterministic will do, and when you do, run it several times and record the variance, because one grade from a model is a sample rather than a measurement. Wire the whole set into CI so it runs on every change to a prompt, a model name or a dependency. Pin the model version explicitly, so a provider update becomes a deliberate decision you make rather than a Tuesday you cannot account for. Then break something on purpose and confirm the net catches it. A suite nobody has ever seen fail is not evidence of anything.
Done when
- A test set of a hundred real requests exists, drawn from logs and grouped by shape.
- Deterministic checks cover every output property that can be checked without a model.
- Model-graded checks record variance across repeated runs rather than a single score.
- The suite runs in CI on any change to a prompt, a model name or a dependency.
- You have deliberately broken something and watched the suite fail.
What you end up with
A regression suite running in CI over a hundred real requests, with a recorded run where you broke it on purpose and it caught you.
If you get stuck
The most common failure is grading everything with a model because that is easiest, which produces a suite slow and flaky enough that the team quietly starts ignoring red builds. Push as much as you can into deterministic checks. The second failure is never once seeing the suite fail, which leaves you trusting something you have never actually tested. Break it deliberately, keep the record, and you have proof instead of a claim.