career bridge
The pipeline that survives
Build one retrieval pipeline over data you did not create, measure it before you tune it, and record the changes that made things worse.
You know what the job is and which gaps you are closing. You may well have built a retrieval demo that answered five questions beautifully. What you have not done is watch one meet a real corpus: inconsistent formats, duplicated documents, tables that carry the actual meaning, and questions nobody thought to anticipate.
What you do
Build one retrieval pipeline over data you did not create, and make it hold. Pick a corpus with genuine mess in it: your employer's public documentation, a regulator's PDF library, a large open dataset with scanned pages in it. Not a clean text file, which teaches you nothing. Ingest it, chunk it, embed it, store the vectors alongside their metadata, and put a query in front. Postgres with pgvector is enough and keeps the whole thing in one place. Then, before you tune a single parameter, write the question set. Twenty to forty questions somebody would really ask, each with a note on what a good answer must contain. This is the step almost everyone skips and the step everything after it depends on. Run the set. Record what fraction retrieved the right passage at all, and separately, what fraction produced a good answer. Those are two different failures with two different fixes, and one combined number destroys the information. Now change exactly one thing. Chunk size, or overlap, or metadata filters, or a reranker. Run the set again. Write down whether it helped, including the times it did not, because those entries are the ones that make the table trustworthy. Three or four honest iterations beat twenty untracked ones. Finish by naming the failures you did not fix.
Done when
- A corpus you did not author is ingested end to end, and a query returns cited passages from it.
- A written question set of twenty or more real questions exists, each with expected answer content.
- Retrieval accuracy and answer quality are recorded separately, never as a single combined score.
- At least three single-variable changes are logged with before and after numbers, including the ones that made it worse.
- A short written list of known failure modes exists, with your best guess at the cause of each.
What you end up with
A repository containing the pipeline, the question set, and a results table showing what each change did, including the changes that made it worse.
If you get stuck
Almost everyone tunes before they measure. It feels productive, it is unfalsifiable, and three weeks later you cannot say whether anything improved. The way out is to stop, write the question set, and accept a bad first score with good grace. The other trap is a corpus that is too clean, which makes everything work and teaches you nothing at all. If your first run scores above ninety percent, either your data is too easy or your questions are too kind. Make them harder and run it again.