- Stimuli
- For each Golden Set brand, 8 page variants from one scaffold: 1 faithful, 5 single-axis ablations, 2 multi-axis drifts — 40 total. Stage 1 tests axis sensitivity (discriminant validity), not ecological validity; stage 2 uses real candidate pages.
- Raters
- 8–12 working designers, blind to the canonical variant and to the judge's output; recruited outside the Studio W230 network where possible; familiarity recorded and stratified.
- Task
- Pairwise "which is more on-brand for {brand}?" plus a pass / revise / reject per stimulus. Sessions capped at 20 pairs.
- Reliability ceiling
- Krippendorff's α on verdicts, Kendall's W on rankings — the human–human band the judge is held to.
- Judge–human
- Spearman ρ vs Bradley–Terry human ranking; AUROC of pass/reject vs human majority; per-axis discriminant validity (does ablating axis k move the judge's axis-k score the way humans penalised?).
- Pass criterion
- Judge–human ρ falls within the 95% CI of mean human–human pairwise ρ. A judge far above the human band is reported as a flag, not a success: it would suggest the stimuli are easier for a rule engine than for people.
- Held-out
- At least 5 catalog brands never touched by
learn_weights.py are reserved as a permanent validation set.
- Committed in advance
- If human α is low (< 0.4), the claim becomes "more consistent than designers, and auditable", not "agrees with designers". Logged as a dated amendment to the protocol before any data was collected, so the stimuli cannot be tuned toward a flattering ρ.
- Protocol
- Judgment Validity Protocol v1 — pre-registered 10 Sep 2026, amended 5 Oct 2026, no data collected yet. The report will be published at
docs/VALIDITY-REPORT-v1.md and linked here whatever it shows; if the judge fails the criterion, this page will say so.