A written design cannot be run, so nobody can prove it works the way you prove software works. This is a method for grading one anyway — many AI agents in parallel, each ruling on a slice of the cases, each forced to point at the exact line that makes its answer true.
Method written down 2026-09-09, from a live run of it · counts recounted from the verdict files the same day
The problem in one sentence. Before this, “the design passes 564 stress tests” only meant 564 cases had been written down. Nobody had checked whether the design actually answered any of them. This method replaces that sentence with one verdict per case, each carrying a citation you can go and read.
Read this table alone and the rest of the document should follow. None of these words are used loosely anywhere below.
| Word | What it means here | What it is NOT |
|---|---|---|
| design | A set of written documents describing how a system should behave — rules, refusals, field names, invariants. Prose and tables, not code. | NOT software. You cannot run it, so you cannot test it the ordinary way. |
| stress case | One awkward situation written down as a short scenario plus the thing that must be true — “two people are numbered t-1001 in two different accounts; the system must not confuse them.” |
NOT a unit test. Nothing executes. It is a sentence that a human or an agent has to judge. |
| adjudicate | To read the design, read one case, and rule on whether the design as written satisfies it. The ordinary courtroom sense of the word: hear the matter, give a verdict. | NOT to fix anything. An adjudicator never edits the design. It only rules. |
| citation | A pointer to the exact place that makes a verdict true — a file and line number (want.md:216) or a section id (WANT §10.55). |
NOT a summary, and never a reviewer’s opinion. Pointing at somebody else’s conclusion is not evidence. |
| positive control | Something you already know is there, searched for on purpose, to prove the instrument can find anything at all. Borrowed from the lab: you run the test on a sample you know is positive before you trust a negative. | NOT optional, and NOT the same as double-checking a result. It checks the tool, not the answer. |
| provenance | Where a case came from and when it was written relative to the design rule that answers it. | NOT the same as which range or file it sits in. Two neighbouring cases can have completely different provenance. |
| independent case | A case written before the design answered it, or written by an outside reviewer trying to break the design. A verdict on one of these is real evidence. | NOT necessarily an old case. Independence is about the order of authorship, not the date on the file. |
| derived case | A case written in the same editing round as the rule that satisfies it — often written from that rule. It passes because it was built to. | NOT worthless, but NOT proof of correctness either. A derived PASS proves the document agrees with itself. |
| circular | The condition of a case that cannot fail, because the thing it tests and the thing it tests against were written by the same hand at the same time. | NOT cheating and NOT a mistake by anyone in particular. It happens quietly, which is why it needs measuring. |
| swarm | Many agents launched at once on different slices of the same job. They read the same source files and write different output files, so they never collide. | NOT agents talking to each other. Each one works alone and never sees another’s answer. |
| Verdict | Meaning | Requirement |
|---|---|---|
| PASS | The design satisfies the case. | Must carry a citation. No citation means FAIL, not PASS. |
| FAIL | It does not. | One line saying what is missing. |
| PARTIAL | The machinery exists but the case’s specific claim is not covered. | Cite what exists; name what does not. |
| N/A | The case is about shipped software, not about the design. | Say so. |
A unit test works by running the code and comparing what came out to what should have come out. A design has nothing to run. So a catalogue of stress cases written against a design sits in an uncomfortable place: it looks like a test suite, it is counted like a test suite, and it has never actually been executed against anything.
Adjudication is the substitute. It cannot prove the design is correct. What it can prove is much narrower and still worth a great deal: for each case, either a specific line of the design answers it — and here is that line — or nothing does.
One agent per range of cases. Ranges of roughly forty to sixty cases each. All agents launched at the same time. Each one reads the design and its own slice of the catalogue, rules on every case in the slice, and writes a table to its own file.
| Role | What it does |
|---|---|
| The driver | One session that owns the whole effort. Picks the ranges, writes the briefs, launches the agents, keeps a live state file, and assembles the results at the end. It does not adjudicate anything itself. |
| The adjudicator (many) | One agent, one range. Reads the design, rules on each case, writes one table. Never edits the design. Never reads another adjudicator’s output. |
| The reviser (later) | A separate agent, run only after triage, and given only the genuine design gaps. Adds rows to the design. Never sees the raw verdict list. |
| Source | Why |
|---|---|
| Read the case catalogue | The cases themselves. Find yours by id. |
| Read the target-state document | The rows and refusals. Most PASS citations come from here. |
| Read the full design record | Both are needed. In the pilot the agent first failed three cases on the target-state document alone; re-checking the full record moved all three to PARTIAL. |
| Read the human rulings file | A case that contradicts a ruling the founder has already made is a failure of the design, not of the ruling. |
| Never read the reviewers’ own reports | This is the load-bearing exclusion. Those files contain the reviewers’ conclusions. An adjudicator that reads a conclusion inherits the answer instead of reaching it, and then cites the reviewer rather than the design. If you find yourself citing a report, stop — that is not evidence. |
Each agent writes one file. One table. One row per case. Nothing else — no essay, no summary, no recommendations.
| ST-id | verdict | citation, or what is missing |The narrowness is deliberate. An agent asked for prose writes prose and hides the verdict inside it. An agent asked for a table produces something you can count.
This is the reusable part. Substitute the values in double braces and hand it to an agent. Everything else stays fixed, on purpose — the wording of the rule and the warning about passing everything are what keep the results honest.
Eight placeholders are marked here so the brief travels to another project. In the live run only {{RANGE_START}}, {{RANGE_END}} and {{OUTFILE}} changed between agents; the four source paths and the do-not-read list were written out in full and identical in every brief. The real paths are in §10.
Adjudicate a range of architecture stress cases against the design, and return a
verdict table. You are one of several agents doing this over different ranges;
yours is ST-{{RANGE_START}} through ST-{{RANGE_END}}.
## What you are doing and why
A software design cannot be executed, so these cases cannot be unit-tested. They
can be adjudicated: read the design, read a case, and rule on whether the design
as written satisfies it.
## THE RULE THAT MAKES THIS REAL — read it twice
A PASS must cite the row, refusal, invariant or section that makes the case true
— a file and line, or a section id like `WANT §10.55` / `DF §4.2`. A verdict with
no citation is a FAIL, not a PASS.
Without that rule, an agent who has just read a persuasive design rubber-stamps
everything, and the exercise is worthless. You are not here to agree with the
design. You are here to find out whether a specific claim is actually written
down somewhere.
Allowed verdicts:
- PASS — the design satisfies the case. Must carry a citation.
- FAIL — it does not. Say in one line what is missing.
- PARTIAL — the mechanism exists but the case's specific assertion is not covered.
- N/A — the case is about shipped code rather than the design.
Some cases in your range are expected to FAIL. Cases marked [OPEN] exist to make
a record gap visible. If you return PASS on everything, that is itself evidence
your method is broken. Returning FAIL where FAIL is right is the most valuable
thing you can do here.
## Sources — read in this order
1. {{CATALOG}} — the cases. Find yours by id.
2. {{WANT}} — the target rows. Most PASS citations come from here.
3. {{RECORD}} — the full design record.
4. {{RULINGS}} — the founder's rulings; a case contradicting a ruling is a FAIL
on the design, not on the ruling.
Do NOT read {{REVIEWER_DIRS}}. Those hold the reviewers' own conclusions; an
adjudicator who reads them inherits the answer instead of reaching it. If you
find yourself citing a reviewer's report rather than a design row, stop — that is
not evidence.
## Output — write it, do not just report it
Write {{OUTFILE}}. One table, one row per case, nothing else:
| ST-id | verdict | citation, or what is missing |
Cover every id in your range that exists. Skip ids that do not exist — do not
invent cases. One line per citation. A table to scan, not essays.
## Report back
How many PASS / FAIL / PARTIAL / N/A, the file path you wrote, and whether
anything about the design surprised you, in at most three lines. If you could not
reach a verdict, say which and why rather than guessing.The reason is uncomfortable and worth stating plainly. An agent that has just spent twenty minutes reading a careful, persuasive design agrees with it. Ask that agent whether the design handles a hard case and it will say yes, warmly and at length, because everything it has just read sounded right. That is the authors-marking-their-own-homework problem wearing a new costume.
A citation breaks the spell, because a citation is falsifiable. The agent has to go and find a line. If there is no line, there is nothing to paste, and the verdict changes on its own.
Both of these are real rows from the run, shortened.
| Case | Verdict | The row as written |
|---|---|---|
ST-60 |
PASS | want.md:216 + want.md:456 — the minting call takes an explicit organization and membership must exist, never first-found; the change log carries issuer, capability and timestamp per organization, so a wrong-side write is valid and attributable. |
ST-75 |
FAIL | Nothing in either file makes unitId optional or names a common-area work order; no split between “unit unknown” and “unit required” (zero hits in both files) and the work-order shape still carries unitId (want.md:724). |
Notice that the FAIL row is just as specific as the PASS row. It names the files searched and reports the zero. “Zero hits” with the search named is a finding; “I could not find anything” is not.
Every batch must contain cases you already know should fail. If the agent returns PASS on one of them, that agent’s whole batch is void — not just the one row. You have learned that this adjudicator says yes to things, and you cannot tell which of its other yeses were real.
| Form | How it works | Strength |
|---|---|---|
| Planted known-failures the standard |
Take cases from an outside list of things the design is known not to answer — an independent reviewer’s leftover findings — and seed them into the batch without telling the agent which they are. | Strong the agent has no way to recognise them |
Catalogue [OPEN] markerswhat the pilot actually used |
The catalogue itself marks some cases as open gaps. The pilot checked that one of these did not come back PASS. It did not — it came back PARTIAL, and the batch was accepted. | Weaker a marker in the file the agent is reading is a hint the agent can see |
Use the first form. The second is reported here honestly because it is what the live run used, and it is better than nothing — but a control the subject can spot is a weak control.
All three of these happened on the single day this method was designed. None of them raised an error. Each returned a clean, confident, wrong answer.
| Instrument | What it said | The truth |
|---|---|---|
A code search run against a named snapshot of the repository (git grep -o against a revision) |
0 matches. | The same word appears 23,666 times. The search form simply does not work against a revision in this setup. Zero from it means nothing at all. |
| A chat search for a sentence that had been said in the very conversation being searched | No match. | Replies inside a thread are not indexed by that search. The sentence was there. The search could not see that part of the room. |
A quick status check trimmed to its first line (git status -sb | head -1) |
A clean branch line. | There were modified files. They were on the lines that got trimmed away. |
The same logic transfers exactly to an agent. An adjudicator that returns nothing but PASS may be reading carefully, or may have stopped reading. From the outside those look the same. The known-failure is how you tell.
Several agents, working separately and none of them asked to look for this, reported the same thing: large parts of the catalogue could not fail, by construction. The cases had been written in the same editing round as the design rows that satisfy them. In several ranges each case names a rule and that rule names the case back.
“Every case in these sections was authored in the same round as the section that names it, and each row cites its own case back — these cases were derived from the rows, so they cannot fail by construction.”
— the agent adjudicating ST-461–488, unprompted
“The contracts’ own benches … near-zero refutation power.”
— the agent adjudicating ST-341–400, which returned 60 PASS and no failures at all
Split the ranges by provenance and the pattern is not subtle. Every figure below was recounted from the verdict files on 2026-09-09, after all eleven had landed.
| Range | Result | Non-PASS | Provenance |
|---|---|---|---|
| ST-001–059 | 56 PASS / 20 PARTIAL / 0 FAIL | 26% | Round-one core the design was built around these — see the refinement below |
| ST-060–100 (pilot) | 23 PASS / 16 PARTIAL / 3 FAIL | 45% | Independent early sequence, predates the rows |
| ST-101–160 | 34 PASS / 23 PARTIAL / 3 FAIL | 43% | Independent three of the seven FAILs came from here |
| ST-161–220 | 37 PASS / 22 PARTIAL / 1 FAIL | 38% | Independent early sequence |
| ST-221–280 | 57 PASS / 3 PARTIAL / 0 FAIL | 5% | Derived |
| ST-281–340 | 59 PASS / 1 PARTIAL / 0 FAIL | 2% | Derived rows authored the same day, for these cases |
| ST-341–400 | 60 PASS / 0 PARTIAL / 0 FAIL | 0% | Derived “near-zero refutation power” |
| ST-401–460 | 59 PASS / 1 PARTIAL / 0 FAIL | 2% | Derived |
| ST-461–488 | 25 PASS / 3 PARTIAL / 0 FAIL | 11% | Derived “cannot fail by construction” |
| ST-900–910 as the catalogue rewrote them | 10 PASS / 1 PARTIAL / 0 FAIL | 9% | Was meant to be the control — see below |
| ST-900–910 in the reviewer’s original words | 8 PASS / 3 PARTIAL / 0 FAIL | 27% | The only genuinely independent test in the whole run |
The oldest range of all, ST-001–059, was expected to have the highest failure yield and returned no failures at all. That breaks the simple story of “older means more independent,” and the agent that ruled on it explained why: those cases are the round-one core that the design was written around. Every one of them has some mechanism in the record, because the record was built to answer them.
| Direction | What happened | Effect on evidence |
|---|---|---|
| Case shaped the design round-one cases | The design was authored to answer these cases. It therefore answers them. | Passes are near-certain. Little refutation power. |
| Design shaped the case late cases | The cases were written from the rows and restate them. | Passes are guaranteed. No refutation power. |
| Neither the middle sequence, and an outside reviewer | Written against a design that had already been decided and by someone trying to break it. | This is the only real test. |
One block of cases — ST-900 to ST-910 — had been designated the independent control. An outside reviewer had written them, on a different day, specifically to break the design. They should have been the hardest cases in the whole set. They returned 10 PASS and 1 PARTIAL.
The agent found why, in the catalogue’s own introduction to that block: the assertions had been “derived from the governing WANT rows”. During ingestion, the reviewer’s original wording had been rewritten into the design’s own vocabulary.
The repair is simple and everyone should copy it. Go back to the reviewer’s file, take the assertions as originally written, and adjudicate the same eleven cases again against the same design. Then count how many verdicts moved.
The three did not all soften the same way. Each is a separate move worth recognising in your own ingest:
| Case | The softening move | What it did |
|---|---|---|
ST-900 |
A fact was added | The restated case gives one party a permission the reviewer’s original scenario never granted. That single added clause turns an unconditional demand the design cannot meet into a conditional one it can. |
ST-906 |
The assertion was inverted | The reviewer demanded that a building’s own calendar come back as the default. The restated case asserts “no automatic redirect” — the design’s answer, copied out word for word, as the thing to be proved. |
ST-910 |
A requirement was dropped | The reviewer asked for two things. The restated case keeps the one the design satisfies and quietly loses the one it does not. |
| Check | Tell |
|---|---|
| Does the design row name the case id? | If the row says “this closes ST-473” and the case cites that row, they were written together. Derived |
| What does the version history say? | Compare the editing round that added the case with the round that added the row. Same round means derived. |
| Does the case use the design’s private vocabulary? | An outsider writing a case says “two tenants with the same number.” The design’s own author says “a claim sentinel without an account dimension.” House vocabulary in a case is a warning sign. |
| Does a preamble say the assertions were normalised? | Take it literally. That sentence is the whole finding. |
| Kind | What it means | Who fixes it | Real examples |
|---|---|---|---|
| 1. Design gap the real bug |
No row anywhere answers the case. The hole is genuine. | Hand to a reviser to add rows to the design. Additive only — never let a reviser condense. | ST-75 no work order without a unit · ST-81 no recording-consent rule at all · ST-87 two people numbered t-1001 in two accounts collide · ST-180 no cap on how many organizations one view may select |
| 2. Stale case fix the case |
The design deliberately answers the opposite way, on purpose, and the case was never updated to match. The design moved; the catalogue did not. | Edit the case. Touching the design here would reverse a decision somebody made deliberately. | ST-83 the case demands no writer-stop window; the design deliberately closes writes during a fenced run · ST-70, ST-73 assert gaps the design has since closed · eleven of the twenty-two PARTIALs in one range were this |
| 3. Decision owed ask a human |
The mechanism is fully written, but it rests on a product choice nobody has made yet. There is no engineering answer because the question is not an engineering question. | Raise it with the person who can decide. Not a code change and not a design change. | policy_authority_matrix, still listed as an open decision in five places that are now stale · booking_confirmation_promise, which a passing case already depends on · prewall_cohort_logins |
Failures arrive scattered and are usually not scattered. Two examples from the same run:
| Looked like | Actually was |
|---|---|
Three separate PARTIALs in the pilot — ST-61, ST-64, ST-96 | One unsettled contract, at a single line of the design. One work item, not three. |
| Twenty-two PARTIALs and one FAIL spread across a sixty-case range | One thin layer: everything to do with a human looking at and editing settings across many buildings at once. The design is deep on rules and refusals and thin exactly there. One coherent work item, not twenty-three. |
| Report it as | Because |
|---|---|
| Independent cases: N pass, N partial, N fail | This is the refutation evidence — the part that could have gone the other way. |
| Derived cases: N pass, N partial, N fail, labelled as internal consistency | A PASS here proves the document agrees with itself. That is coverage evidence, which is worth something, but it is not correctness. |
| Unreachable: which ranges have not landed | An absent range is not a passing range. |
The driver’s running state file is written by hand as batches land, and hand-kept tallies drift. Recounting all eleven verdict files on 2026-09-09 gave 528 verdicts: 428 PASS, 93 PARTIAL, 7 FAIL — of which 517 are first-pass rulings (420 PASS / 90 PARTIAL / 7 FAIL) and 11 are the control block judged a second time. Three differences from the state file, all small, all worth knowing about:
| Claim in the state file | Recount from the verdict files | |
|---|---|---|
ST-197 listed among the design-gap FAILs | Its verdict file records it as PARTIAL | Contradicts |
| ST-161–220 at 33% non-PASS | 23 of 60 = 38% | Understated |
| An interim total of 333 PASS / 43 PARTIAL / 5 FAIL over eight files | The same eight files: 330 / 47 / 4 | Drifted |
One more counting trap, since this is a document about instruments that lie. Two of them, in fact. Counting rows by searching for the row marker also counts the table’s header row, so every file reads one higher than it is. And one file wrote its ids in bold — | **ST-900** | — so a search for the plain form found four rows in a file that holds eleven. Both mistakes return a confident number. Verify the count on one file you can count by hand.
DRIVER.md stating the goal, the citation rule and the control requirement; an AGENT-PROMPT-TEMPLATE.md holding the brief in §4; a STATE.md updated as batches land; and a verdicts/ folder. Anyone picking the run up mid-flight reads those two files and continues.verdicts/ST-161-220.md. The file name is the only index you need, and a missing file is immediately visible as a missing range.These are the real paths from the run described here, so the {{…}} values in §4 can be filled in without guesswork. They are written as text, not links — they are on one machine, not on the web.
| What | Path | Role |
|---|---|---|
| The run directory | ~/arch-stress-run/ | Everything the driver owns. |
| — the goal and the rules | ~/arch-stress-run/DRIVER.md | Goal, citation rule, control requirement. First thing a successor reads. |
| — the brief | ~/arch-stress-run/AGENT-PROMPT-TEMPLATE.md | §4 of this document, with the placeholders. |
| — live state | ~/arch-stress-run/STATE.md | Updated as batches land. Hand-written, so recount it before quoting — see §9. |
| — the output | ~/arch-stress-run/verdicts/ | One file per range: ST-161-220.md, and so on. |
| The design set (base) | …/docs/planning/portfolio-architecture/how/in the worktree wt/fable-arch-review | Everything an adjudicator reads lives under here. |
— {{CATALOG}} | notes/stress-catalog.md | The cases. |
— {{WANT}} | notes/want.md | The target rows. Most PASS citations come from here. |
— {{RECORD}} | design-final.md | The full record. Must be read too — see the both-files rule in §9. |
— {{RULINGS}} | notes/decided-2026-09-09.md | The human decisions. A case contradicting one is a design failure, not a ruling failure. |
{{REVIEWER_DIRS}} — never read | ~/astra-review/rounds/~/fable-review/ | The reviewers’ own conclusions. An adjudicator that reads one inherits the answer. |
| The outside reviewer’s original wording | ~/portfolio-independent-stress-2026-09-08.md §4 | The most valuable file in the set. Written before the design absorbed it, and the only thing the rewritten control block could be checked against. |
| Guidance | Why |
|---|---|
| 40–60 cases per agent | Both sizes worked. Much larger and the agent starts summarising instead of adjudicating. |
| Cut ranges along the catalogue’s own numbering, not along topic | Numbering usually tracks the order cases were written, which is what provenance depends on. Cutting by topic mixes independent and derived cases inside one file and destroys the split you most need. |
| Give the independent ranges their own agents | They are the only refutation evidence you have. Keep them separable in the output. |
| Expect ids that do not exist, and lettered sub-ids that do | Tell agents to skip missing ids and never invent a case. One 59-id range produced 77 verdict rows because cases split into ST-02a, ST-02b and so on. |
| Item | Status |
|---|---|
| The method itself — citation rule plus positive control | Validated in the pilot: control held, all 23 PASS rows carried a citation, zero uncited passes |
| All eleven ranges adjudicated | Landed 2026-09-09 — 528 verdicts across eleven files |
| Re-adjudicating the control block against the reviewer’s original wording | Done — 8 PASS / 3 PARTIAL, and the source file was confirmed byte-identical to the reviewer’s copy, so all the drift is in the catalogue conversion |
| How much ingestion softened the outside review | Measured: 3 of 11 — three verdicts differ, all harsher, by three different mechanisms (a fact added, an assertion inverted, a requirement dropped) |
| Whether the same softening happened in the ordinary catalogue | Unmeasured — only the eleven control cases have an original wording preserved to compare against. For the other 500-odd there is nothing to check them against, which is the whole argument for keeping originals |
| Whether the design gaps, once revised, actually close | Pending — requires a fresh re-run of the affected ranges, by a new agent, not the one that ruled first |