Retro: why we made basic mistakes and iterated slowly — PO pivot live-test day
retro · 2026-07-28
first live-test day, vendor PO work
Written by Fable after the first live-test day of the vendor PO work.
why we are making basic mistakes and its taking so long to iterate.
Fede, verbatim — the question this page answers.
The short answer
Three things compounded:
- Our safety gates were words, not mechanisms. We said "hold this PR for review" — but the repo's auto-merge arms itself the moment a PR opens, and it merged seven PRs past their gates today, including the ADR that was explicitly never supposed to merge without Fede. A rule that isn't enforced by a machine is a suggestion.
- We tested every part and never the whole. The database layer was green. The prompt evals were green. The sync was green. And the very first real phone call failed — because the failure lived in the connections: seed data missing an index key, a prompt that never read the variables being sent to it, a voice platform that silently drops what the prompt doesn't reference. Nobody's checklist said "now make one real call."
- We used a shipping pipeline for a design loop. Getting a prompt tweak to Fede's ear took PR → review → merge → sync: 30–45 minutes per lap. Feel iteration needs a lap time of minutes. Fede called this mid-session ("can we just work on a branch on eleven labs — lets do some quick iterations"); the bench (branch-fed test agent on its own phone number) went live 90 minutes later. The mistake was that the shipping pipeline was the default at all — branch-based iteration should have been the starting setup for a feel-testing day.
The mistakes, in plain terms
| What happened | Why it happened | What changed |
|---|---|---|
| First live call: Clara had no PO info at all | Demo data was seeded without the index key the lookup reads. Every layer had been tested alone; the seam never ran until a human dialed | Mandatory pre-flight probe (one curl) that proves the call pipeline end-to-end before anyone dials. CI test queued |
| Second failure: vars arrived but Clara ignored them | The prompt that talks to vendors never referenced the new variables — and ElevenLabs silently drops unreferenced variables. The evals covered a different prompt than the one vendors actually reach | Real robo-calls (the inbound battery) are now the proof step; a drift test guards the variable-wiring class; evals must target the agent the lane actually runs through |
| A regression shipped to prod overnight | The merge gate said "evals must pass first," but the eval was blocked (metered-key guard) and auto-merge merged anyway | Evals now run on subscription auth (no human unblock needed); holds are enforced with a label applied at PR creation — the only brake auto-merge respects |
| The ADR merged without Fede's review | Same auto-merge race | Same fix; plus repo-level change proposed: auto-merge should require the review verdict, not just green CI |
| Robot tests passed while the basic flow was broken | Battery scenarios were written from the feature we just built, not from what vendors actually call about (scheduling is #1 in the real data and had zero coverage) | Standing rule: test scenarios derive from the production call corpus, by frequency. The multi-turn gauntlet is corpus-shaped and caught 4 real bugs in its first run |
| Clara leaked internal accounting posture to a vendor | Every grader policed making things up; none policed volunteering internals | New disclosure floor in all vendor-facing suites: never volunteer internal process/accounting state |
| Fede dialed the wrong number and tested the old version | Two numbers in play and my messaging switched between them; nothing on the call itself says which version answered | Bench runbook consolidates: one bench number, persona switcher, and the transcript check that verifies which agent answered. The bench itself was Fede's mid-session call, live 90 minutes later — it should have been the day's starting setup, not a mid-afternoon addition |
| Vendor-outbound agent was 1,012 characters behind the repo in #4786's dry run — and 3,756 characters behind by the time the sync actually applied, since #4782 merged in between | Its prompt could only be updated by hand-pasting into a dashboard — so merged work silently never reached it | Safe per-agent sync built (#4786); drift is now measurable with a read-only script; fleet-wide nightly drift check queued |
| "Staging" test calls were secretly answered by the prod agent | A stale environment variable pointed at a branch of the wrong agent | Found and repointed; the lesson is the same as above: every mirror (staging, preview, bench) must be verified faithful, not assumed |
What the slowness actually was
Almost none of today's elapsed time was writing code. It was:
- Re-work from premature merges (fix-forward PRs for things review would have caught: 5+ extra PRs).
- Discovering seam failures one phone call at a time instead of one probe at a time.
- Waiting on merge-grade pipeline for feel-grade changes until the bench existed.
- Chasing unfaithful environments (staging answered by prod, preview resolving phantom data).
What's now true that wasn't this morning
- A pre-flight probe proves the pipeline before any human dials.
- Real robo-calls exist for inbound (3 scenarios, rebuilding corpus-shaped) and outbound (rig existed, now PO-aware).
- A multi-turn simulated-conversation suite grades the actual thing Fede cares about — does the vendor get helped without a transfer — and prints a transfer rate.
- A full-corpus replay (66 of the 69 labeled calls — the 3 others are 2 emails + 1 SMS — vs. the new prompt, call-by-call better/same/worse) is running.
- The bench gives minutes-scale prompt iteration without touching prod.
- Evals run without waiting on a human to unlock spend.
- Holds are mechanical (label-at-creation), and the deeper repo fix (auto-merge waits for review) is proposed.
The one-sentence version
We built and verified parts, trusted the assembly, and let an over-eager merge robot ship the
assembly unverified — the fixes are: probe the assembly, test what users actually do, and give design iteration a
loop that runs in minutes.
Corrected 2026-07-29 after a source audit.