Ask Clara · Overnight build report — 2026-08-01
Ask Clara went from timing out on the founder's own question to answering all seven — and now starts answering in a few seconds
Everything below was measured on production by the same graded seven-question owner battery, run before and after. The whole rebuild is merged and live. Judge: Claude Sonnet 4.6, five dimensions (correctness, breadth, specialization, precision, actionability), 1–5.
1 · The trigger
In Friday's standup Sean asked Clara for the top 5 problems I have as an owner — and it timed out in front of everyone. The investigation that followed found the failure was not flaky, it was deterministic: that question returned a platform timeout 3 out of 3 times at roughly 60 seconds, and so did every other broad, portfolio-wide owner question. The same question, hinted to use fewer lookups, came back fine in 42 seconds — so the intelligence was never the problem, the budget was.
Two quieter failures were sitting underneath it. Some questions returned a successful response with a completely empty body — the screen showed a blank Clara bubble, no error, no log line. Others were cut off mid-word and shipped as if finished, so the action items at the end of the answer — the whole point — were silently lost. And nothing streamed: the entire progressive-typing apparatus in the interface was inert scaffolding waiting on a server that sent one buffered chunk at the very end.
Eight verified root causes were catalogued in total, from the hard 60-second wall clock and the unchecked answer cap down to a poisoned chat history that stuck in one browser forever once an empty answer was saved into it.
2 · The before scoreboard
Seven owner questions, run against production the night of Jul 31.
4
hard failures (timeouts)
1
of 7 gradable — scored 4.4
≈ total
time to first word
Time-to-first-word was identical to total time on every single question: you stared at nothing for a minute, then either got everything at once or got an error.
3 · What was rebuilt overnight
Making an answer possible at all
- Extended time budget with a graceful wrap-up — Clara now has room to think, and when she's running out of it she closes out with what she has instead of dying at the wall.
- Eight times the answer budget, and truncation is handled — a cut-off answer is now detected and continued rather than shipped as if it were finished.
- Error copy in Clara's voice — every distinct backend failure used to collapse into one opaque "I had trouble analyzing the data"; each now says what actually happened.
- A retry affordance and self-healing chat history — a failed turn can be retried in place, and a poisoned empty turn no longer sticks in that browser forever.
Making it feel like a product
- Server-rendered boot — faster first paint and a single settings read instead of a client round-trip.
- Voice dictation in the composer — ask by talking.
- A ChatGPT-class interface, now live on production.
- Real token streaming with progress narration — words appear as they're produced, and Clara narrates which part of the portfolio she's reading while she works.
Making her smarter — the brain layer
- A multifamily-analyst persona, replacing the generic assistant framing.
- She can read the nightly ranked portfolio signals. The product already computed, every night, a deterministic ranked answer to "where do I need to pay attention" — and Clara had no way to read it, so she was re-deriving it from raw records on every question. Now she just reads it.
- Conversation themes aggregated across calls, texts and emails, so what residents are actually saying becomes an input.
- Real Denver market benchmarks — HUD FY2026 Fair Market Rents plus Census ACS — with every figure carrying its source and vintage.
- Test traffic excluded from demo numbers, per-request read memoization, end-to-end specs pinning all three failure modes, and a repeatable benchmark harness — the one that graded everything on this page.
The rebuilt interface, live on production.
Voice dictation in the composer.
4 · The after scoreboard
Same battery, same judge, same production URL. The streaming release merged during the session, so the headline numbers below are from the final battery run against live production with everything shipped.
Live on production, everything shipped
4.66
overall quality (1–5)
2.3–6.7s
time to first word, every question
4.1s
median time to first word
Sean's question — "top 5 problems I have as an owner" — now starts answering in 2.3 seconds and grades 4.8 out of 5. Before, it was a 60-second wait ending in an error, three times out of three.
An honest note on total time: median end-to-end is about 64 seconds, in the same range as before. What changed is not how long Clara thinks — it is that you now see the first words in 2–7 seconds and watch her narrate which part of the portfolio she's reading while she works. The wait didn't shrink; it stopped being a blank screen.
Question by question
| Question |
Before |
Prod, wave 1 |
Live grade |
Live first word |
| Portfolio health |
timed out |
5.0 (rerun) |
4.8 | 5.5s |
| Top 5 owner problems Sean's question |
timed out |
4.8 |
4.8 | 2.3s |
| Where to pay attention |
timed out |
5.0 |
5.0 | 4.0s |
| Rent per square foot |
empty answer |
4.6 |
4.4 | 6.7s |
| Vendor and maintenance themes |
answered but truncated · 4.4 |
4.2 |
4.8 | 2.9s |
| Delinquency |
empty answer |
4.6 |
4.6 | 5.1s |
| Drill-down follow-up |
timed out |
4.4 |
4.2 | 4.1s |
| Overall |
1 of 7 gradable · 4.4 |
7 of 7 · 4.6 |
4.66 | 4.1s median |
Breadth — how many parts of the portfolio one answer reaches across — was the dimension that stayed soft all night, and it is the one the brain layer moved most: 3.7 on the wave-1 run, 4.4 live. Specialization and actionability are now a clean 5.0 across all seven questions.
Supporting evidence — the streaming release's preview
Before it merged, the streaming release was benchmarked on its own live preview: 7/7 answered, 0 empty, 0 truncated, overall 4.46, first word in 1.3–7.4s (median 3.2s), median total 43s. That run is what justified shipping it; the production numbers above supersede it.
Two independent checks
- The automated behavior-eval sweep passed on CI — and for the first time it was scoring the real thing. The eval had been grading a stale paraphrase of Clara's prompt rather than the prompt production actually runs; that was fixed tonight. The dashboard-chat gate passed, and every domain that finished scored a 100% pass rate before a superseding push cancelled the tail.
- Overnight error monitoring came back clean. Zero errors on any surface this work touched, across every deploy in the window. The only new issue in that window was an unrelated tour-notification test gate, which is ticketed.
5 · Also fixed, from the same standup
- Rent per square foot now recalculates off the adjusted lease rent, so an adjustment actually moves the per-square-foot number instead of leaving a stale one on screen.
- Vendor callbacks are no longer misfiled as dead air — the class of miss that swallowed the blinds installer's return call.
6 · Where it landed, and what's still open
All twelve pull requests from this session are merged, and every Trello card that came out of the standup is Done. The whole rebuild — time budget, answer budget, error handling, retry, boot, dictation, interface, streaming, persona and brain layer — is on production.
Deliberately left open:
- The Opus-vs-Sonnet model comparison is deferred on purpose. Running it means forcing a model tier through a mechanism gated behind a spend control, and that needs Fede's explicit override rather than a subagent's. Opus stays the default, matching Fede's prior.
- Follow-up tickets filed: an org-scope audit of the sibling data tools, uncited legacy benchmark constants still sitting in the code, consistency between the theme reader and its callers, a guard so CI cannot go silent when a run is cancelled by a conflicting push, and the tour-notification test-gate noise.
7 · 2026-08-02: Model decision — Sonnet 5 at low effort
The Opus-vs-Sonnet comparison deferred above was run the next morning on an in-process harness (subscription-judged, same seven-question owner battery), followed by an effort sweep and a fresh live production re-measurement.
Tier comparison, against the shipped prompt
| Metric |
Opus 5 |
Sonnet 5 |
| Overall | 4.66 | 4.49 |
| Correctness | 4.57 | 4.57 |
| Specialization | 5.00 | 4.86 |
| Precision | 4.57 | 4.14 |
Correctness tied exactly. The overall gap is 0.17, mostly precision (4.57 vs 4.14). Latency between runs was noisier than the quality gap: Sonnet measured 0.76× Opus in one run and 1.16× in another.
The pre-agreed quality gate — overall ≥ 4.5 — technically failed by 0.01 against the shipped prompt. Fede chose Sonnet with the numbers in hand, for cost and latency. It is recorded in code as an explicit override, not a measured win.
Effort sweep on Sonnet 5 — low vs medium
Two repeats per leg.
| Metric |
Low |
Medium |
| Overall | 4.36 | 4.30 |
| Correctness | 4.36 | 4.14 |
| p50 latency | 29.8s | 46.2s |
| Slowest question | 59s | 136s |
Correctness was higher at low effort in both repeats; quality overall is labeled unchanged within variance. Latency is not: p50 0.64× and the slowest question 0.43×. CHAT_EFFORT is set to low.
Latency changes shipped the same morning
- First-turn portfolio snapshot prefetch (3s budget, deadline-capped).
- Message-tail caching on wrap-up.
- A no-preamble-before-tools rule.
- Parallelized DB reads in three tool handlers.
- Adaptive round-trip estimation.
- Cache-hit telemetry.
Live production re-measurement — same seven-question battery, 2026-08-02
30s
median total (was 64s — 0.47×)
3.1–8.5s
time to first word
Plainly: the 4.2 vs 4.66 quality dip is a single run and sits within the run-to-run variance already observed. The rig re-runs in about 10 minutes if answers feel thinner in real use.
A precision-focused prompt variant was also tested and rejected — it raised precision by +0.29 but cost breadth on both models.