# Judgment Validity Protocol v1

| | |
|---|---|
| Status | **Pre-registered — data collection not started** · amended 2026-10-05, before any data collection (see §11) |
| Date | 2026-09-10 |
| Grammar version | 2.0.0 |
| Weights version | `w-2026-05-001` |
| Companion | [grammar-v2-calibration-2026-05.md](grammar-v2-calibration-2026-05.md) — extraction fidelity, a different question |
| | [SELF-LEARNING.md](SELF-LEARNING.md) — consumer of the evaluation records this study produces |
| | [schema/evaluation-record.json](../schema/evaluation-record.json) |

## 0. Why this document exists

Everything tasteHQ has published about the judge so far measures **extraction
fidelity**: how closely the extractor's reading of a brand's own homepage matches
the hand-graded catalog value for that brand (`/eval`, `tools/spike_roundtrip.py`).
That is a test of the parser.

It does not answer the question a user actually asks: *when the judge says a new
page passes against Stripe, would a competent designer agree?* That is
**judgment validity**, and it has never been measured. This protocol is committed
before any data is collected so that the analysis cannot be shaped by the result.

The protocol is deliberately small. A rigorous answer on five brands is worth more
than an impressionistic one on a hundred.

## 1. Questions

| # | Question | Statistic |
|---|---|---|
| Q1 | How well do designers agree with *each other* about on-brandness? | Krippendorff's α (verdicts); Kendall's W (pairwise rankings) |
| Q2 | How well does the judge agree with the designers' consensus? | Spearman ρ (judge score vs Bradley–Terry human ranking); AUROC (judge `pass`/`reject` vs human majority verdict) |
| Q3 | Per axis: when axis *k* is perturbed, do humans penalise it, and does the judge's axis-*k* score move in the same direction? | Sign agreement rate and effect size per axis |

Q1 is the ceiling. Q2 is read against Q1, never alone. Q3 is what decides which
axes belong in a core grammar.

## 2. Stimuli

| Element | Specification |
|---|---|
| Brands | The Golden Set: `anthropic`, `linear`, `stripe`, `acne-studios`, `antimetal` |
| Scaffold | One neutral page template per brand (hero, one content section, one CTA, footer), rendered from the brand's catalog tokens. Same content in every variant; only design axes change. |
| Variants per brand | 8: **1** faithful render · **5** single-axis ablations · **2** multi-axis drifts (3 axes each) |
| Total stimuli | 40 |
| Ablation axes | Chosen per brand from axes the extractor scores ≥ 0.70 on that brand, so the judge *can* see the change. Minimum spread: one palette, one type, one whitespace, one surface, one emphasis axis. |
| Ablation magnitude | Two ordinal steps on the grammar scale, or the nearest nominal alternative. Piloted (see §5) so ablations are neither invisible nor cartoonish. |
| Rendering | Static HTML + CSS, screenshotted at 1440×900 with `api/render.mjs`. Raters see screenshots; the judge scores the HTML. |
| Exclusions | No third-party imagery, logos, or copy. Rights posture follows `DATA-LICENSE.md`. |

## 3. Raters

| Element | Specification |
|---|---|
| N | 8 minimum, 12 target |
| Who | Working product / brand designers, ≥ 3 years' experience. At least half recruited outside the Studio W230 network. |
| Blinding | Raters do not see the judge's output, do not know which variant is the faithful render, and are not told which axis was perturbed. |
| Familiarity | Self-reported familiarity with each brand (1–5), recorded before the task, used as a stratification variable. |
| Session | ≤ 20 pairwise comparisons + ≤ 8 absolute verdicts per sitting. Two sittings maximum per rater. |
| Compensation | Paid at a professional hourly rate. Recorded in the report. |

## 4. Tasks

**Task A — pairwise.** "Which of these two is more on-brand for {brand}?" Pairs are
drawn within brand; each rater sees a balanced incomplete block so every variant
appears in ≥ 4 pairs per rater. Order and left/right position randomised.

**Task B — absolute.** For each stimulus: `pass` / `revise` / `reject` as if
reviewing a pull request for that brand. One free-text line: "the single thing
that most gives it away".

Task A produces the ranking (Q1 W, Q2 ρ). Task B produces the verdict (Q1 α,
Q2 AUROC). The free-text line is coded post hoc against the 30 axes for Q3 and is
exploratory.

## 5. Pilot

Two raters, one brand, all 8 variants. Purpose: check that the faithful render
wins ≥ 75 % of its pairs and that at least 3 of the 5 ablations are detected
(ranked below faithful) by both raters. If not, adjust ablation magnitude and
re-pilot. Pilot data is excluded from the main analysis. The adjustment is
recorded in the report.

## 6. Judge configuration

Fixed for the study and recorded in every `EvaluationRecord`:

- `weights_version`: `w-2026-05-001` (the weights *before* any learning from this data)
- `profile`: default public profile; `JUDGE_RENDER` on; `JUDGE_VISION` off
- Target: catalog grammar for the brand (`target_slug`), not `target_url`
- Viewport: 1440×900 only

The judge is run once per stimulus before rater data is collected, and the outputs
are committed (hashes in the report) so they cannot be re-run after seeing the
human results.

## 7. Analysis plan

1. **Q1.** Krippendorff's α on Task B verdicts (ordinal metric). Kendall's W on
   per-rater Bradley–Terry rankings from Task A. Report with 95 % bootstrap CIs.
2. **Q2.** Fit a Bradley–Terry model to pooled Task A data → one human ranking per
   brand. Spearman ρ between that ranking and judge `score`, per brand and pooled.
   AUROC of judge `score` against human majority verdict (`pass` vs not), and
   separately of judge `verdict` (binary) against the same.
3. **Read Q2 against Q1.** The pre-declared success criterion is:
   *judge–human ρ falls within the 95 % CI of mean human–human pairwise ρ.*
   A judge far *above* the human band is reported as a flag, not a success — it
   would suggest the stimuli are easier for a rule engine than for people.
4. **Q3.** For each ablated axis: fraction of raters ranking the ablation below
   faithful; judge axis-*k* delta (faithful − ablation). Sign agreement and
   Cliff's δ. Axes with sign agreement < 0.6 or where humans did not penalise the
   ablation are flagged for the core/extended grammar decision.
5. **Confounds.** Repeat Q2 stratified by familiarity (≤ 2 vs ≥ 4). Report both.
6. **No exclusions** after the fact except a rater whose Task A choices are
   indistinguishable from random (binomial test, p > 0.5 against 50 %), declared
   here in advance.

## 8. Outputs

| Artifact | Location |
|---|---|
| Stimuli (HTML, screenshots, ablation manifest) | `research/validity-v1/stimuli/` |
| Judge outputs, pre-committed | `research/validity-v1/judge/*.json` + `SHA256SUMS` |
| Rater data (anonymised) | `research/validity-v1/raters.jsonl` |
| Analysis notebook / script | `tools/validity_report.py` |
| Report | `docs/VALIDITY-REPORT-v1.md` — linked from `/eval` |
| Evaluation records for the learning loop | `feedback.jsonl` per `SELF-LEARNING.md`, **after** the report is published |

The report publishes Q1, Q2, and Q3 whatever they show. If the judge fails the §7.3
criterion, `/eval` says so.

## 9. Held-out policy

The 25-brand extended set already informs `catalog_prior` in `api/weights.json`.
From this study onward:

- Golden Set brands are used for *validity* studies and *regression* tests.
- `learn_weights.py` may consume the evaluation records this study produces.
- Five catalog brands, named in `docs/HELD-OUT.md` before the first weight update,
  are never used to tune weights and are reserved for a future replication.

## 10. Known limitations, declared now

- Five brands from a similar cluster (developer-tools and fashion minimalism).
  Validity here may not transfer to dense or maximalist grammars.
- Screenshots at one viewport. Motion, scroll behaviour, and responsive axes are
  out of scope for v1.
- Synthetic stimuli from a shared scaffold. Real agent output is messier; v1
  establishes whether the judge can see controlled differences at all.
- Small N. The study can detect large disagreements between judge and designers;
  it cannot resolve small ones. That is acceptable for a first measurement.

## 11. Change log

| Date | Change |
|---|---|
| 2026-09-10 | Protocol pre-registered. No data collected. |
| 2026-10-05 | **Amendment, before any data collection (none collected as of this date).** The original text above is unchanged; these two commitments are added. (a) *Interpretation of Q1, fixed in advance:* if human–human agreement on Task B is low (Krippendorff's α < 0.4), tasteHQ will not claim the judge "agrees with designers". The claim becomes "more consistent than designers, and auditable". Decided 2026-09-10 and published here so that stimuli, ablation magnitude and analysis cannot be tuned toward a flattering ρ. (b) *Scope:* v1 is stage 1 — controlled ablations, i.e. discriminant validity. It does not test ecological validity on real candidate pages; that will be pre-registered separately as v2 before any v2 data is collected. |