Learning log / November 2026

Evals with a model as judge

How I stopped judging a prompt by feel. Hand labels, a judge model, and running everything three times.

Why I needed evals

prsist turns my AI sessions into short notes. For weeks I judged the prompt that writes them by reading its output and deciding whether it felt better. It mostly felt better. Then I changed the prompt, and a note I cared about stopped appearing, and I only noticed days later.

I needed a number I could trust after every change, cheap enough to run every time.

Labels first

I went through 39 real sessions and wrote down the notes I’d want from each one. That took most of an evening and was the most useful hour of the project. Writing the labels forced me to decide what a good note actually is.

A hash of each session id puts about a third of them in a test split. I never look at those while tuning, so I can’t fit the prompt to them by accident.

A model as judge, for the part I can’t label

The distiller often writes notes I didn’t label but that are still fine. Rather than mark those wrong, Claude Opus judges them: worth keeping, or junk. That gives me two numbers:

Run it three times

The same prompt on the same sessions gives different scores from run to run. Running each version three times shows how much of a change is real. Between v5 and v6, junk fell from 27.5% to 22.8%, and the ranges across runs didn’t overlap, so I believe it. Capture didn’t move, and I don’t pretend it did.

What I’d tell someone starting

The full write-up, with the numbers, is in the prsist case study.