What we measure, how, and what it showed. Generated from the recorded results; no number on this page is typed by hand.
Synthetic
invented people and invented records. No real person's data is used in any eval.
This page is regenerated from the recorded results in evals/results by evals/page.py, every time the evals are recorded. Newest results: 20261019T055740-code.json, 20261019T055626-model.json.
How we measure
Code evals are deterministic. They run on every change, and a missed bar fails the build.
Model evals run at least three times. The model proposes and code checks what it proposed. We record the median, the range across runs, tokens in and out, and the time per call. A hash of every file whose text reaches the model is kept; if any of them changes, the recorded run no longer counts until it is run again.
Each bar is set from what 3P promises, not from what it scored. Lowering one needs a written decision.
Cost is shown only from a published price. None has been recorded yet, so it says not computed rather than a guess.
Measured by code, on every change
Recorded · commit 9d5b55d+changes · on macOS-26.6.2-arm64-arm-64bit-Mach-O
When two people meet: speak only when there is a reason, and cite it
When two people meet, does 3P speak only when there is a reason, cite it, and stay silent otherwise, including when the other person forbade the topic? Directed cases in English and Persian, run on real signed cards.
Measure
Result
Must be
Met
precision
1
≥ 0.95
Met
recall
1
≥ 0.9
Met
silent when unrelated
1
≥ 0.95
Met
silent under a never-say rule
1
= 1
Met
leaks past a never-say rule
0
= 0
Met
uncited or invented citations
0
= 0
Met
directed cases
252
precision (English)
1
recall (English)
1
precision (Persian)
1
≥ 0.95
Met
recall (Persian)
1
≥ 0.9
Met
The non-negotiables, each held by its tests
Does each non-negotiable still hold? Each is tied to the tests that fail if it stops being true, and those tests are run.
Measure
Result
Must be
Met
held
1
= 1
Met
The verifier agrees with the shared test vectors
Does the verifier agree with the shared test vectors that the browsers check too?
Measure
Result
Must be
Met
agreement
1
= 1
Met
vectors
13
canonical forms
15
Lose a machine, come back from paper
Can a person lose a machine and come back from a retyped paper sheet, step by step?
Measure
Result
Must be
Met
steps held
1
= 1
Met
steps
12
Persian is read one way everywhere
Do different spellings of one Persian word meet, and do different words stay apart?
Measure
Result
Must be
Met
variant spellings meet
1
= 1
Met
different words kept apart
1
= 1
Met
tags read
1
= 1
Met
Speed on this machine
Is writing an entry instant, and are signing and checking a card quick?
Measure
Result
Must be
Met
write an entry, median (ms)
0.407
write an entry, 95th percentile (ms)
0.485
≤ 50
Met
build and sign a 300-entry card, median (ms)
27.646
≤ 1500
Met
check a card, median (ms)
25.378
≤ 200
Met
Measured with a model, three runs or more
Recorded · commit 9d5b55d+changes · on macOS-26.6.2-arm64-arm-64bit-Mach-O
Faces found from patterns, not declared
Are faces found from patterns, without tags or declarations? An invented person with three selves; code scores each proposal's evidence.
Model claude-opus-5-5, anthropic · 3 runs · 3 model calls · tokens in / out 6129 / 2984 · latency per call, median / 95th percentile 11436 / 11735 ms · cost: not computed: no published price recorded
Measure
Result
Must be
Met
selves found
1
= 1
Met
purity min
1
≥ 0.9
Met
purity mean
1
proposals
3
“What matters now” comes from this week
Does “what matters now” come from this week? An invented week with three planted goals and an old project that must not come back.
Model claude-opus-5-5, anthropic · 3 runs · 3 model calls · tokens in / out 2871 / 392 · latency per call, median / 95th percentile 2730 / 3137 ms · cost: not computed: no published price recorded
Measure
Result
Must be
Met
goals found
1
≥ 0.66
Met
proposals
3
precision
1
≥ 0.75
Met
dropped by code
0
old project leaks
0
= 0
Met
With people
No study with people has been run yet. The study kit is ready; nothing here comes from participants.
What these numbers do not show
Every eval uses invented people. Real records are messier.
The meeting corpus was written by the same author who fixed the code it found problems in, so its scores are not independent evidence. A held-out corpus written by someone else is needed.
The model tasks are easy: three clearly different selves, three clearly planted goals. Perfect scores here show the plumbing works, not that discovery works on a real life.
Speed is measured on one laptop.
Nothing here measures whether people find 3P useful. That needs the study with people.