unify docs

Ratification — what the agent-experiment rounds changed

Status: v0.7.0, historical record and argument Role: This document is the narrative and the evidence behind docs/ratification-protocol.md. The protocol is the procedure — how to run a round, how to triage what it finds. This is the account of running it: what was measured, what broke, what the results changed in the documentation, the specification, the implementation, and the product itself. The round-by-round primary source is _notes/ratification-rounds-7-25.md, with round 1 and the pre-registrations beside it.

1. Why an empirical loop existed at all

v0.6.6 shipped 62 test files at a reported 93%+ coverage while unify init exited 1 and scaffolded nothing, layout discovery did not exist, and a page's content was silently deleted when the layout lacked the expected element — under a printed ✅ Build completed successfully!. docs/testing-strategy.md §1 traces the five mechanisms that let a suite that large see none of it. The short version: coverage counts lines entered, and a test that calls a function and asserts the result is an object covers every line of a wrong answer.

The countermeasures in docs/testing-strategy.md fix that for the engine. They cannot touch the other half of the product. docs/authoring-rules.md claims to be the complete authoring surface in 60 lines — learnable in one sitting, pasteable into a prompt. That claim is the whole authoring contract, and no test suite can evaluate it: a suite tests what the build does with markup that already exists, not whether a reader of those 60 lines produces the right markup in the first place.

Reviewing the document does not work either, and the record says so. Three adversarial reviews and a ten-agent design panel (_notes/agent-authorship-panel.md) read the same text and missed defects the first round of authoring agents surfaced in twenty minutes. Later, paper review began injecting defects — roughly three new silent-failure modes introduced while fixing fifteen. You read your own documentation the way you meant it; every ambiguity resolves silently in your head.

So the doc was treated as an empirically testable artifact. Hand an agent nothing but those 60 lines and a plain-English brief, in a sandbox it cannot escape, and see whether it authors a compliant site. If it cannot, the doc is wrong. The agent is a fast, cheap, uncomplaining proxy for the human newcomer who fails the same way and never files an issue.

2. The loop

One round is: regenerate rules.md from the current doc so every sample is provably given byte-identical input; seed N isolated sandboxes (five Haiku plus a Sonnet control is the default); launch one claude -p process per sample with cwd inside its sandbox; collect; judge; triage; amend at most a handful of things; re-run.

Three properties of that shape carry the method.

Isolation is structural, not instructed. In-session subagents inherit CLAUDE.md no matter what the prompt says. Each sample is its own process, with the task, the rules and the report format copied into its directory so no path outside cwd is ever referenced. A correctly configured run answers NO when asked whether it has any project context describing the tool; that check is re-run on every protocol change.

The weak model is the bar and the strong model is the instrument. The rules are ratified when Haiku consistently (4 of 5 or better) produces a compliant site from the rules alone. Sonnet runs alongside not to raise the ceiling but to locate defects: when the strong model succeeds where the weak one fails, the gap is a documentation defect that reasoning papers over, not a capability floor. Round 1 is the canonical instance — Sonnet got the title rule right and 3 of 4 Haikus got it wrong, and both Sonnets reported having inferred it rather than read it. Round 13 is the same signature at spec level: the control escaped a trap by prior knowledge ("og:image has to be a fully-qualified URL"), not by anything in front of it.

The build is the judge. unify build --dry-run --strict, run by the experimenter over every sample, exit code plus diagnostic list. This was not true at the start, and the change is the single most important fact about the record.

The instrument changed at round 7

Rounds 1–6 were judged by reading sample files, and recorded two 6/6 sweeps. The first brief judged by running the real build came back 1 of 5 Haiku, on a document those rounds had passed. Nothing regressed; the measurement improved. Every pre-round-7 clean result should be read as no violation a human reviewer noticed, and any older brief worth relying on should be re-run.

Two costs came with the upgrade. The CLI has to go inside the sandbox (a compiled binary carries no docs, so isolation holds), and briefs must be honest about what the build checks — a reference to an image the agent had no way to create fails the build while telling you nothing about the doc. Rounds 7 and 9 each lost two samples to exactly that artifact before round 13 seeded a real images/ directory.

Two experiments, not one

Blind authoring — the agent can run nothing — measures the document. Authoring with the build measures the document plus the error contract, which is how anyone actually works. Both are legitimate; what is illegitimate is switching a brief from one to the other while measuring a fix, because that changes the environment and the doc together and the result stops being attributable.

3. Rules of evidence, learned the hard way

Every one of these was paid for.

Judge what a sample did from its transcript, not from its report. A report is a claim. Round 15's write-up stated that five of six samples never ran --dry-run; the session transcripts show all six did, and the figure had come from grepping each agent's final report text. Round 16 hit it twice independently — one report naming data-layout= on pages that carry the Markdown layout: spelling, another declaring a site clean of a placeholder string its own files still contained. The self-report is the best evidence available about what a sample found ambiguous, and it is not evidence about what a sample ran or wrote.

Judge in place, with the sample's own command. Re-running a sample's build with -o redirected turns the existing dist/ of a root-authoring sample into source material and manufactures failures. This produced false verdicts twice — round 13's haiku-3 was recorded exit-1 and is clean, and three round-14 samples the same way.

Check exit codes before analysing anything. Thirteen simultaneous cold starts produced a burst of transient proxy TLS failures that killed half a round; at twenty seconds apart, none. A sample that never started looks exactly like a sample that did nothing. Empty a sandbox before re-seeding it, too: a retry that begins on top of a dead run's half-built site is not a sample.

Sweep the site output, not the sandbox. rules.md quotes {{ }} as an example of what not to write, so a naive violation grep reports templating syntax in every clean sample.

Convergence is the strongest signal there is, and it points away from the agents. Independent models do not make the identical mistake by accident. When four of five write the same wrong thing, the rule caused it — and before touching the rule, check what the spec says, because three times in the record the answer was that the agents were right.

4. Triage in practice

Every finding is assigned to documentation, specification, implementation, or outlier, and the fix goes to the layer that is actually wrong. The full decision procedure is docs/ratification-protocol.md; these are four real assignments.

Documentation — the named-slot fill (round 7). The brief compelled exactly one named fill. Four of five Haiku failed: one wrote <slot name="footer"> in the page, one nested a <slot> inside a page <footer>, one wrote <slot slot="footer"> — both spellings at once — and one avoided the mechanism by inventing a third layout. Not contamination, not an outlier, and the spec was right: a <slot> in a page has always been inert. The cause was that the rule showed <slot name="footer">fallback…</slot> and never showed a fill, so the only concrete literal in the rule was the one that got copied into the wrong file. haiku-1's report quotes the sentence, restates the rule correctly, and then writes the wrong thing. The doc now shows the page-side literal (<footer slot="footer">…</footer>) and says outright that a <slot> tag in a page fills nothing. Round 9: 5 of 5 correct fills, zero stray <slot> tags.

Specification — flat og: keys (round 5). Four of six wrote og:image: /card.png where the spec required a nested block. The reflex is to call that invalid frontmatter; it does not survive checking. A colon is only a YAML key separator when followed by whitespace, so the flat form parses to a key literally named og:image, fell through to "every other key", and emitted <meta name="og:image"> — a tag that looks right, builds clean, exits 0, and is ignored by every scraper. The spec was also arguing with itself: P17's own fix advice suggested the flat spelling the table did not accept. The rule was re-keyed off the key's name rather than the YAML shape, which made both spellings work and deleted a trap. Round 6: six of six wrote the flat form, including the control — the nested-only rule would have failed every sample.

Implementation — the og: reference-check hole (round 13). Found while triaging, not by a sample. collectHtmlReferences gathered og:/twitter: content only when it started with /, so under a full-URL --base-url the absolutized values were never reference-checked at all — despite §12's explicit sentence that stripping exists so those values "stay checkable instead of masquerading as external". A broken og:image 404'd silently and REF-02's stripBaseUrl was dead code for the metas it was written for. Round 18 found another by accident: a whole-line # Build settings comment in unify.yaml exited 2, because the trailing-comment strip requires whitespace before the #, so a comment at column 0 fell through to the key/value match and failed it. The round's fixture author had to delete the comment to keep it from swallowing the round.

Outlier — one sample, no pattern (round 7). haiku-5 used data-layout correctly and aimed it at the wrong layout for its override page. That is brief comprehension, not a rules defect. Outliers are recorded and watched, never amended for: the doc is capped, and spending a line on a one-off is how a 60-line file becomes a 200-line file nobody reads.

The ledger

Every finding the loop shipped, by the layer it was charged to. "Tested" means a later round the fix was not written for came back clean; everything else is fitted to the round that found it.

| Finding | Round | Layer | Repair | Follow-up | |---|---|---|---|---| | Layout title separator described, not instructed | 1 | doc | layout-side literal, imperative | tested — 6/6 in r7, 6/6 in r9 | | <meta charset> in pages "forbidden" | 2 | doc | the spec permits it; the doc was wrong | — | | Flat og: keys rejected, silent dead name= meta | 5 | spec | key off the key's name, not the YAML shape | tested — 6/6 flat in r7 and r9 | | Named-slot fill: layout literal copied into pages | 7 | doc | show <footer slot="footer">; A04 → P20 | 5/5 in r9, 5/5 in r16 | | Unquoted colon in a YAML title breaks the build | 7 | doc | "Frontmatter is YAML: quote any value containing a colon" | 0 P18 in r9 | | P12 fix: suggested a destructive repair | 8 | impl (message) | "rename … or merge them into one" | 7/7 preserved in r10; 6/6 in r18 | | P04 printed the Markdown spelling on HTML pages | 8 | impl (message) | spell the fix for the page's kind | 3/6 cited it in r18 | | --base-url unnamed beside --pretty-urls | 11 | doc | name the flag | 5/5 in r12, every link prefixed | | "run first" did not say who runs it | 11 | doc | show the literal command | 0 reports of the ambiguity in r12 | | Path-form --base-url ships dead og: metadata | 13 | spec | full-URL form only; A15 tried, then retired | 6/6 in r14, 6/6 in r15 | | og: values never reference-checked under a full base | 13 | impl | widen the REF-02 collector | unit tests only | | Scaffold taught a placeholder as structure | 16 | scaffold | replace with <main><slot></slot></main> | — | | slot= placement rule wrong in both directions | 16 | doc + engine | doc fixed; fill scope widened to the unwrapped <main> | — | | Cascade + duplicate diagnostics (21 for one fault) | 18 | impl | DEDUP §14.1, cascade suppression §12 | 30 → 13 on r18's fixture | | P04's fix: shipped one hardcoded layout path | 18 | impl (message) | suggest the layout the page would discover | — | | P18 never named the colon | 18 | impl (message) | lead with the quoting rule and an example | — | | Whole-line comment in unify.yaml exited 2 | 18 | impl | fix the comment strip; pin both surviving cases | — | | A03 condemned valid documents, rewarded nested ones | 16/18 | spec | retire the advisory; §14.3 gains two tests | — |

5. The sharpest result: round 18's accidental control

Rounds 8 and 18 test the other half of the documentation — the error contract, which is a far larger surface than the 60 lines, written by implementers and read only in the moment someone is stuck. The design: plant known faults in a working site, hand it over with the CLI, require a clean build without losing anything the site says. Every page carries a unique MARKER-* sentence so content loss is mechanically detectable, because a repair that deletes a page to silence its error looks like success from the exit code alone.

Round 8 (seven-page site, eight problems, two advisories) returned 7 of 7 clean, four of them with no documentation whatsoever. Round 18 (ten pages, a fault set chosen deliberately outside round 8's — P02, P04, P07, P11, P13, P14, P18, P19, A03, A11, A13) returned 6 of 6 clean, zero markers lost, zero pages deleted, 65 of 66 planted-fault repairs correct.

Then the accident. Round 18's first launch inherited a harness default without Bash, so six samples repaired the site from the source alone, never running the tool. The run was preserved rather than discarded: 0 of 6 reached a clean build — 17, 12 and 7 problems remaining in the no-docs arm — every one wrote a confident report, two copied the broken pattern from a neighbouring file, and none found P04, P14 or P18.

Round 8 established that the error contract is sufficient documentation for repair. This is the converse, and it is the stronger claim: without it, the same model on the same site fixes roughly half the faults and cannot tell that it failed. For repair, the diagnostics are not a supplement to the 60 lines — they are the documentation.

That conclusion has a direct consequence for how diagnostics are written, and the loop paid for every clause of it. A message must be located, must carry a fix: line, and that line must name the replacement spelling for the file it is pointing at:

6. What the loop changed in the product

The findings did not stop at prose. Four changes to the product itself came out of this record, and two design rules generalise from them.

A rule that shows exactly one literal will have that literal copied, whether or not it belongs in the file the reader is editing. This is the project's most repeated finding. It explains the named-slot failure (round 7), the P04 misdirection (round 8), the hardcoded layout path (round 18) — and, used deliberately in the other direction, it is what fixed --base-url: when the doc's single literal became the full-URL form, six of six samples in round 14 copied it on their first build and no sample ever saw the advisory written to catch them.

Prefer deleting the choice over warning about it. Whenever a diagnostic exists to compensate for a design the tool did not have to offer, the diagnostic is the wrong repair.

One further finding was not a doc defect at all. Round 16 was the first round to start from unify init, and the scaffold outperformed the prose — 5 of 5 on the named-slot fill the 60 lines had historically failed at, with no sample re-reading the rule. Working code teaches a shape that prose has to argue for. It also taught one thing wrong: the template's sink was <main><p>Page content appears here.</p></main>, four of four Haiku left it in place and three copied it verbatim into a brand-new section layout. Triage: not doc, not spec, not implementation — scaffold. The tell was the control's replacement, <main><slot></slot></main>: the rules doc's own literal, the shape those 60 lines call "the usual", and a shape the scaffold did not contain.

7. The guardrails that kept it honest

An empirical loop with no constraints becomes overfitting with extra steps. These are not negotiable.

7b. Round 26: the slotted-include rule, and two defects it found on the way

Product-spec §6.4.1 made shipping slotted includes conditional on the complete rule still fitting comfortably on the authoring-rules page — the one feature in the product whose ship condition is an authoring trial rather than a test. §32.7 recorded the empirical half as owed until this round ran.

The brief. A public observatory's site, with the same equipment callout in nine or ten places — a bordered panel, an icon, a heading, body text, a muted difficulty line — where only the words differ, volunteers who do not write HTML must be able to add one, and the day the border style changes, it must change in one place, not in ten files. Nothing else in the authoring surface answers that shape. Five Haiku samples and one Sonnet control, mount-namespace isolation, 45-minute cap, staggered launches; isolation re-verified on the machine first (a probe answered NO to the project-context question and a whole-filesystem search for the tool's own documentation returned empty).

The result: six of six. Every sample reached for a slotted include unprompted — nine to fourteen call sites each, a single *.fragment.html holding the panel, named slots for the heading and the difficulty line, the bare slot for the body, and fallbacks written into the fragment. Every sample built clean under --dry-run --strict with zero problems and zero advisories. No fallback and no <slot> reached any output, no draft published, and the mechanism worked from Markdown log entries as well as HTML pages — the case the rule never shows and every sample got right anyway. Against the protocol's bar of 4 of 5, the rule teaches.

The round still found two documentation defects, neither of them in the rule being tested.

Documentation — --pretty-urls and "link the real filename" read as contradictory (3 of 5). The doc stated both facts and never connected them. haiku-1's nav links failed and it had to re-read to find that .html was right after all; haiku-5 wrote /about/ links first for the same reason; haiku-3 never resolved it and inferred a false model — "the server handles the URL masking" — which happened to build clean here and would not have on a site that leaned on it. Three independent samples converging is the protocol's strongest signal, and it points away from the agents. The sentence now says the rule holds under --pretty-urls: you still write /about.html, and the build rewrites it to /about/.

Documentation — "shipping exactly as written" was false (Sonnet control). The named-slot rule said a fill "replaces the slot, tag and all, shipping exactly as written". S3 and S7 remove the consumed slot attribute, so the phrase contradicted the specification on a detail a reader can only learn by reading built output — which is exactly what the control did, then reported that it could not find the sentence saying so. The spec was right and the doc was wrong; it now says the element keeps its own markup and only slot= is dropped. This is the round's clearest instance of the pattern the protocol names: the strong model found a precision defect the weak ones papered over, and precision defects are the ones that mislead quietly.

Two notes on the instrument, recorded so the next round does not re-learn them. judge-round.mjs could not judge one sample because its publish command was written in inline code rather than a fenced block — it was a correct command and the site built clean when run by hand, so a "could not judge" line is not a failure and must not be read as one. And a frontmatter metric configured as "/^---$/m" silently counted zero across all six samples that did write frontmatter: a supplementary metric reading zero is a claim about the config until checked against the files.

7c. Round 27: the 0.8 surface holds, and a diagnostic is the round's one fix

The pre-registered round (_notes/ratification-round-27-preregistration.md, outcome addendum included) briefed six isolated samples to publish a site using the whole 0.8 surface: schema:, the nested og: block, dated Markdown posts by non-HTML-writing volunteers, a subpath deploy with extensionless addresses, and a working feed. Five criteria were fixed before launch.

The 0.8 rules teach. Per criterion, Haiku scored: strict build 5/5, feed 5/5 — four entries each, every <updated> carrying a time, zero A17 in any final build — JSON-LD 5/5, og: on posts 4/5, extensionless addresses 4/5. The registered bar (four of five samples clean on all criteria) was missed at 3/5, by two single-sample, single-criterion misses: one sample put its og: block on the HTML pages and none of its Markdown posts; one never engaged --pretty-urls. Both are outliers under the convergence rule — no rules.md amendment. Both samples also reported the missed thing as verified ("og: tags present" over posts with none; "No .html extensions" twenty lines below its own /workshop/visit.html) — round 18's confident-false-report pattern, twice in one round, caught by judging fresh rebuilds instead of reports.

The convergent finding pointed at a diagnostic. Two of five linked /feed.xml from their chrome before the feed's conditions were met and got P13's standing fix line — check the path spelling and casing — wrong on both counts: the spelling was right, and no source file was missing. One "fixed" it by shipping ../feed.xml; the other invented a build-twice model in its report. §12 now adds a second fix line when the unresolved target is exactly one of the three generated root names, stating the condition this build did not meet (--base-url; --search-index; for the feed, whichever of address/declaration/dated-time actually failed). Two-sided tests pin all three names, including that the line never fires on a build that emitted the file.

The date rule's error contract works. One sample wrote day-only dates, met A17 mid-iteration, and repaired to full timestamps — the pre-registered success path, observed rather than assumed.

The control's report earned its keep even while its site passed everything: it named the one genuinely under-specified sentence on the surface (og:image:width versus "two levels deep is an error" — resolved only by experiment), and independently re-derived round 7's layout-side/page-side slot confusion before backing out to a simpler design. Both recorded as watched, not amended: neither caused a failure, and the budget spends only on failures.

7d. The post-round session review: what was adjusted, and what deliberately was not

Rounds 26–27 were re-read end to end — samples' claims against their artifacts, every friction report against the sentence it names — and each candidate adjustment was decided with the evidence stated, the declines recorded as firmly as the changes.

Diagnostics — one change (already landed), two declines. The P13 second fix line for the three generated root names is round 27's fix. A17's wording stays: the complaint was that it "focuses on reporting rather than inclusion", and its message already says this page is not in feed.xml — inclusion is named. And there is deliberately no new audit finding for an Article post with no og:image (round 27's one V4 miss): verified empirically that nothing fires, and that is the design — §6.3.4's "missing social-image targets" means a declared image whose file or dimensions are missing, because reporting the absence of a tag no rule requires would be the style prescription §6.1 forbids.

Output — no change, for a reason worth keeping. A successful build prints nothing, and the review confirmed that is contract, not accident: stdout is compared byte-for-byte by the determinism fixtures, §14.1's summary sentence assigns a channel rather than mandating content, and the quiet is of a piece with the no-log-theater posture. A generated-artifacts summary line was considered and declined — it would not have prevented either round-27 false claim (h2's problem was a flag never used, not ignorance of what was emitted), and it would repin every stdout fixture on the eve of a release for a speculative benefit.

Authoring rules — one clause, one decline. The clause: line 53 read "(og:image: /card.png; two levels deep is an error)", and both report-reading samples strained against it — the control could not tell whether og:image:width is "two levels deep" and resolved it only by experiment, while the audit's own fix lines recommend exactly those keys. An author who parses the colon as nesting avoids the keys the tool then asks for: the silent-degraded-share-card class round 13 was about. The clause now says a colon inside a name is not nesting. The decline: unify audit stays out of the 60 lines for now — it would not have prevented any round-27 failure (verified: audit is silent on the one V4 miss), every addition displaces, and the honest instrument for "should the rules name the evaluator?" is a round whose brief compels evaluation. That is a candidate round-28 design, not an amendment.

The harness — one addition. Round 27's evidence lived on /tmp and was destroyed by a machine reset hours after judging; the protocol now archives a round — judge output, reports, sample trees — into the repository before triage. Its transcript-capture failure stays an open instrument item for round 28, partially compensated by the archive step.

7e. Round 28: the evaluator needs no clause, and a sharper question replaces it

The round §7d promised: a brief that compels evaluation — every check the tool offers, including the ones that do not block a build, must report nothing — without ever naming the mechanism, with discovery thresholds registered before launch. Discovery was 6/6. Every sample found unify audit from --help alone and ran it two to five times, fixing what it reported (titles, descriptions, share-image dimensions) and re-running to clean. The registered question closes with no amendment: the 60 lines stay silent about the evaluator because the tool advertises it adequately at the moment it matters.

The rest of the registered card: strict builds 6/6, audit-clean on fresh rebuilds 6/6, and — re-measuring round 27's one miss — og:image with real dimensions, absolute, on every post, 6/6. The line-53 clause written after round 27 held universally on first exposure.

What the round surfaced instead, recorded as unregistered and exploratory: three of five shipped .html-visible addresses against the brief, and the transcripts turn this from a doc question into a sharper one. One sample quoted the pretty-urls sentence verbatim, explained it correctly in its own reflection — "an output transformation, not a source-level concern" — and then left the flag out of its final command. The sentence teaches; the flag does not survive into the command line; and nothing in the tool can push back, because extensionless addresses are a per-site deployment requirement no generic check can own. Round 29's registered question, with one design note carried forward: none of the four samples that missed it, across both rounds, wrote a unify.yaml — the mechanism that exists precisely to make a site's flag decisions durable.

8. Running it again, and what is still untested

docs/ratification-protocol.md is the procedure — setup, the harness traps, the triage order, the two amendment rules, and the results table for every round. _notes/ratification-handoff-prompt.md is the self-contained block to hand a fresh agent, including the five gates that must be green before any commit.

Known limits of the method, stated because a method with no acknowledged limitations has not been examined: it reaches only the authoring surface, roughly a fifth of the spec — splice rules, head-merge tables, collision matrices, URL rewriting and publish semantics are unreachable by an agent writing HTML. Samples are correlated, since models share training data. And the bar is a model, so "Haiku gets it right" is a moving target; a doc that passed a year ago is not thereby passing today.

Open work, in the order the record recommends:

  1. The human half. Eighteen rounds, all models. A person who knows HTML but not this tool fails differently — they stop and re-read where a model guesses and moves on.
  2. A Markdown source map. A body-derived diagnostic on a .md page currently omits its line number, because the text the engine holds at that point is converted HTML and any line it computed would be a guess (DIA-13). Omitting is correct and it is not the same as being located; mapping converted offsets back to .md source lines would let those diagnostics carry a real FILE:LINE.
  3. The mis-nested fill. A slot= under a plain wrapper <div> still does nothing, still ships unify's own attribute into output that is supposed to contain no tool vocabulary, and says nothing. Three catalogue slots are free, but the evidence does not yet say whether authors write that markup at all. The experiment that would settle it is a brief compelling a fill from inside a styled wrapper — not a guess.