Why I needed evals
prsist turns my AI sessions into short notes. For weeks I judged the prompt that writes them by reading its output and deciding whether it felt better. It mostly felt better. Then I changed the prompt, and a note I cared about stopped appearing, and I only noticed days later.
I needed a number I could trust after every change, cheap enough to run every time.
Labels first
I went through 39 real sessions and wrote down the notes I’d want from each one. That took most of an evening and was the most useful hour of the project. Writing the labels forced me to decide what a good note actually is.
A hash of each session id puts about a third of them in a test split. I never look at those while tuning, so I can’t fit the prompt to them by accident.
A model as judge, for the part I can’t label
The distiller often writes notes I didn’t label but that are still fine. Rather than mark those wrong, Claude Opus judges them: worth keeping, or junk. That gives me two numbers:
- Capture: the share of my labelled notes the distiller wrote.
- Junk: the share of everything it wrote that the judge threw out.
Run it three times
The same prompt on the same sessions gives different scores from run to run. Running each version three times shows how much of a change is real. Between v5 and v6, junk fell from 27.5% to 22.8%, and the ranges across runs didn’t overlap, so I believe it. Capture didn’t move, and I don’t pretend it did.
What I’d tell someone starting
- Label by hand before you build the harness. You’ll learn what you’re measuring.
- Keep a test split you never tune on.
- Repeat runs before trusting a small difference.
- Track the cost. Mine is under a dollar a run, which is why I actually run it.
The full write-up, with the numbers, is in the prsist case study.