Retrieval evaluation.

If the exam stays the same, every new score tells you whether the search got better or worse.

01

The analogy

A teacher who gives the same exam every term can tell whether the class improved. Change the questions each time and a good grade could just mean an easier exam.

Here the student is a search system and the exam is 24 fixed questions whose right passages are known. The slider decides how many passages the search reads per question before it answers.

In the diagram below

the exam
24 fixed questions
the student
the search system
the pages it reads
passages read per question
a question off the syllabus
one with no answer in the documents
02

Try it

Twenty-four questions about a set of policy documents. Twenty-one have their answer somewhere in the documents; three don't.

Your move / Predict, then test

Before you run it, place a bet: reading 3 passages per question, do all 21 answerable questions find their evidence?

Choose your prediction, run the experiment, then reveal the result. You can also explore without predicting.

Each line on the exam sheet is one question, in order. Read more passages and the search finds more, but it also hands more text to whoever reads the answer.

Place your bet and press Run it.

03

Your setting or reading 8

Same exam, same search. The only change is how many passages it reads per question.

04

Where it fits

Worth it

Before changing anything in a search that feeds answers to customers or staff. I picture an internal help desk that answers from policy manuals.

Not needed

For a one-off prototype nobody depends on yet, or when nobody can say which passages are the right ones for each question.

05

What it proves

I didn't trust a good result on one question. I set a fixed exam first, so every change to the search could be measured against the same ruler.

For engineers
  • Hybrid retrieval: BM25 and TF-IDF merged by reciprocal rank fusion over a 16-passage corpus. A question passes when recall at k is 0.5 or more.
  • Three questions have no labelled evidence and are left unscored on purpose, so the exam can't reward an answer that isn't in the documents.
  • Recall, regression diffs, judge agreement and calibration are shared with the Python backend; the retrievers and this exam run are TypeScript only.
  • Stack: Next.js 16, TypeScript, Python, Vitest, pytest.
Source code →