career bridge
Cost and latency
Instrument every request, read the twenty most expensive ones, then change one thing at a time and record what it did.
Your audit has numbers in some rows and unknown in others. The two unknowns that end conversations fastest are the ones nobody asks about until the invoice arrives: what each request costs, and how long a user waits for it. Neither is a model problem. Both are yours.
What you do
Instrument first, then change one thing at a time. Log per request the input tokens, output tokens, model, wall-clock duration and computed cost, and store it somewhere you can query. An aggregate dashboard will not do, because the expensive requests are never the median. Push realistic traffic through it rather than five hand-picked prompts, then look at the distribution: p50, p95, p99, and the twenty most expensive requests by cost. Read those twenty properly. They almost always share a shape, an over-long context, a retry loop, a tool call that fires every time and helps rarely. Now make changes one at a time and record each one. Cache the responses that repeat. Cut the context nobody reads. Route the easy requests to a smaller model and measure whether quality actually drops instead of assuming it will. Stream, so the wait a user feels is shorter than the wait you are paying for. Keep a before and after table for every change, including the ones that made things worse. Finish with a cost per thousand requests and a p95 you would defend in a meeting.
Done when
- Per-request tokens, model, duration and computed cost are logged and queryable.
- The distribution from p50 to p99 is recorded from realistic traffic, not a handful of prompts.
- The twenty most expensive requests are read, and their shared shape is named in writing.
- At least four changes are logged with before and after numbers, including any that made it worse.
- A defensible cost per thousand requests and a p95 latency are written down and dated.
What you end up with
A cost and latency table: the starting distribution, every change you made, and what each one did to both numbers.
If you get stuck
The common failure is optimising the average, which is the single number no user ever experiences. Somebody waits at p99, and that somebody writes the review. Look at the tail. The other failure is refusing to try a smaller model because you assume quality will drop, which leaves you either paying too much forever or shipping something worse without knowing. Measure the swap. It takes an afternoon and it ends the argument.