Clara's conversation loop: where the 21 seconds go

2026-08-02 · stage-by-stage latency budget for inbound → reply · 475 real production SMS traces (Jul 26 – Aug 2) + read-only prod probes + a 3-model bench on the test property. No production code changed; no PRs opened.

Strategy first: what we are actually optimizing

The goal is a fast reply, and "fast" here means total wall-clock from inbound webhook to outbound send — not time-to-first-token. Nobody watches an SMS stream in. Two hard constraints sit on that number: the Twilio webhook budget, and the fact that our compliance gates run after the model and before delivery, so every millisecond anywhere in the loop lands on an already-serialized tail.

That framing decides where to look. The instinct — and the framing of the sibling research dossier — is that an agent loop's wall clock is dominated by model rounds, tool serialization and prefill. For Clara today that instinct is wrong by a factor of two. Every LLM call, every tool, and every compliance gate in a median production turn adds up to about 6.4 seconds. The turn takes 21.5. The missing 15 seconds are spent in one unbroken block of database work between two trace steps, before Claude is ever called.

The stage budget

Every production Clara turn writes an AgentTrace row to DynamoDB with a per-step duration ledger. I pulled every trace from the entityType-createdAt-index GSI on propflow-prod for Jul 26 – Aug 2 (1,077 rows), kept the 475 SMS turns that ran the real production code path, and reconstructed the wall clock — including the gaps between steps, which is where the answer turned out to live.

Two LLM calls are not in this ledger because neither is wrapped in tracedClaudeCall: the pre-loop Haiku intent classifier in classify-message.ts (runs in the router, before the agent loop) and the fair-housing Haiku gate (measured separately above via its own fair_housing_check step). A median SMS turn therefore makes four or more serial LLM calls, not one.

Culprit 1 — context assembly (15.0 s, 70%)

Between the agent:identity_resolution and agent:capability_composition trace steps sits roughly 1,000 lines of conversation-manager.ts containing 50 awaits, 15 dynamic import()s, and only 3 Promise.alls. Nothing in it is instrumented, which is why a year of latency work never saw it.

It is not identity-dependent — the gap is essentially identical for verified tenants (14.4 s), verified prospects (15.1 s) and unknown callers (14.6 s), so the cost is in the unconditional spine, not the enrichment branches.

What is actually slow, measured

I timed the individual data-layer calls on this path directly against propflow-prod, read-only, three runs each:

These probes ran from a laptop (≈50 ms round-trip to us-east-1 vs ≈5–15 ms from Vercel), so the per-call numbers are an upper bound and the fan-out costs are inflated more than the point reads. The shape is what matters: one N+1 that scales with a property's work-order history, sitting in front of ~12 point reads that could be one wave.

Culprit 2 — the system prompt misses its own cache on every turn

Prompt caching inside a turn is working perfectly. Across turns it is almost entirely failing, and the reason is a fixable ordering bug.

Median tokens per call. This directly refutes the dossier's hypothesis #3 (20-block lookback exhaustion): there is no cache falloff by round — zero calls at any round position returned cache_read_input_tokens = 0 after the initial call. The lookback window is not our problem.

The problem is the first call of every turn, and it is a prefix-ordering defect. Anthropic renders tools → system → messages and a cache breakpoint caches the prefix from byte 0. Clara's system prompt is 101,484 characters ≈ 25,400 tokens, and its section order is:

  line   2  ## Security Rules            <- static
  line  23  CLARA'S VOICE ...            <- static
  line  45  PROPERTY INFO:               <- VOLATILE (per property, per conversation)
  line  68  LEASING WORKFLOW:            <- static
  ...       ~530 more lines of static policy, rules, examples
  line 601  ## ACTIVE CAPABILITIES

Because the per-conversation PROPERTY INFO block sits at line 45, ~24,000 tokens of completely static policy text live behind a volatile byte and are re-written to cache on essentially every turn instead of being read from it. The code comment at conversation-manager.ts:1026 already names this exact fix as known-and-unshipped ("reordering getUnifiedSystemPrompt so its volatile sections land last… held for a follow-up PR"). The trace data is the missing evidence for how much it is worth: 10.92 M cache-creation tokens across 475 turns, billed at the 1h-TTL write premium instead of the 0.1× read rate.

Culprit 3 — an entire wasted round on 18% of turns

The first-turn tool gate (added after the 2026-04-08 hallucinated-price incident) checks whether a leasing turn 1 called any tool. If not, it pushes a reminder and re-runs the whole call with a forced tool_choice. In production this fires on 87 of 475 turns (18%) at a median 3,251 ms — 625 ms amortized across all turns, plus a second full ~42k-token prefill.

The retry already knows the answer: it forces tool_choice: {type:'tool', name:'get_available_units'}. Setting that on the first call for a fresh leasing turn removes the failure mode rather than correcting it, at zero behavioral risk — the retry path proves the forced choice produces the desired turn.

Culprit 4 — the fair-housing gate is a serial post-hoc LLM call

checkFairHousing() is a blocking Haiku call that runs after the reply text is final and before delivery, on 93% of turns, costing 1,313 ms median. It cannot be parallelized against the model call that produces its input. It is a real compliance control and I am not proposing to delete it — but its position and its per-turn scope are product decisions worth surfacing, and a Haiku-tier rewrite gate on every turn (including tenant maintenance replies, where fair-housing risk is negligible) is a broader net than the risk requires.

Culprit 5 — serial tool execution (small; the dossier over-ranked it)

The research dossier ranks concurrent execution of parallel tool_use blocks as opportunity #1, and the mechanism is real: conversation-manager.ts has exactly one await dispatchToolCall site and it sits inside for (const block of response.content), so a 3-tool fan-out costs t1+t2+t3 instead of max(t1,t2,t3). But measured against production:

• 103 of 475 turns (22%) contain at least one multi-tool round.
• Batch sizes: 194 rounds of 1 tool, 53 of 2, 50 of 3.
• Serializing costs 241 ms median on the turns that have a multi-tool round, and 58 ms amortized across all turns.

The reason it is small is that our tools are fast: get_available_units 100 ms, check_availability 261 ms, get_tour_slots 395 ms. It is still worth fixing — it is cheap, it is correct, and it protects against tools getting slower — but it is a 0.3% fix, not a top-three one. This is the main correction this measurement makes to the dossier.

Per-model loop latency

Production A/B (strongest evidence)

Production has been running a natural split — 274 SMS turns on claude-opus-5 and 201 on claude-sonnet-4-6 in the same window, same code, same table:

Controlled bench, 3 models, test property

I drove the real loop through runPipeline against the bench property appfolio-45 (The Willows) with a stress-synthetic sender, 5 runs × 2 scenarios per model, CLAUDE_MODEL overriding all AGENT-tier call sites. Credential verified as [anthropic/client] credential=bearer — subscription, never metered. Outbound is unconditionally suppressed for +1206555xxxx senders by stressSuppressionFor(), so nothing reached a carrier.

Two structural signals from the bench matter more than the model column:

• Context assembly is 4.9–5.1 s regardless of model — as expected for DynamoDB work, and it is still half the turn even in the best case: a long-lived local process with warm memoized reads, on a property whose data the process has already touched. Production pays 15 s for the same code on a cold serverless invocation. That gap is the size of the prize.
• A warm prompt cache makes the turn 6–12× cheaper. Bench turns read 33k–47k cached tokens and wrote only 80–102, costing $0.0007–$0.0014. The equivalent production turn writes 24,486 cache tokens and costs $0.0084. Same model, same prompt, same code — the entire difference is whether the static prefix hit. This is the strongest available evidence for fix 3 below.

Bench caveat, stated plainly: scenarios were replayed on a persisting conversation, so runs 2+ settle into single-round, zero-tool turns (Clara correctly answers "already reported"). Conditions were identical across all three models, so the comparison is fair, but these are steady-state simple turns — not a substitute for the production distribution above.

The fix package: can we cut total reply time by half?

Yes — and the two items that get us there are both in context assembly, not in the model. The waterfall below stacks residually: each fix is applied to what the previous one left, so savings on overlapping stages do not double-count. Every row is labelled measured or estimate.

Honest read of the stack-up: items 1 and 2 carry 85% of the win and both are estimates, because I could not safely re-run the production path with the fix applied. They are estimates with measured floors underneath them (the probe timings, and the bench's demonstrated ~5 s assembly on identical code), not guesses — but the package's claim to −57% rests on them, and the first thing to do is to prove item 1 with a before/after on the real path, which is cheap.

If items 1 and 2 underdeliver — say assembly only drops to 9 s instead of 4.5 s — the package lands at roughly 14.0 s, a 35% cut, short of the bar. A model change would not rescue it: the measurement above scores that at zero. What would close the remaining gap is architectural, not incremental:

• Extend ADR-0114's deterministic-first routing from inbound email to SMS, so the common cases — a "thanks", a tour confirmation, a STOP — never enter the agent loop at all. 231 of 475 production turns (49%) used one Claude call and zero tools; a meaningful slice of those never needed the 25k-token prompt or the 15 s of context. Tradeoff: a deterministic pre-router is a new class of misroute risk, on the channel where misroutes reach tenants fastest.
• Precompute the context bundle asynchronously when a conversation opens (or on a schedule per property) instead of assembling it on the inbound edge. Tradeoff: staleness — Clara would answer from a snapshot, and the freshness contract would have to be written down per field.

Both are real projects, which is why they are the fallback and not the plan. The plan is that items 1 and 2 are ordinary database hygiene against code that nobody has profiled.

Cost interplay — how this maps onto metered spend −80%

The north star is $3,541.87/month (July) → ≤$708/month. Latency and spend are the same problem here more often than not, because both are driven by tokens prefilled per turn:

• Fix 3 (cache prefix order) is by a distance the largest token lever in this report. 24,486 cache-creation tokens per turn are being written where a read would do. Overall prompt-cache hit rate across the sample is 63% (20.51 M read / 10.92 M written / 0.97 M fresh); nearly all of the 10.92 M written is the same static policy text, re-written per turn. The bench quantifies the ceiling: the same loop with a warm cache costs $0.0007–$0.0014 per turn against production's $0.0084 — a 6–12× reduction from cache alone. Even capturing part of that is the single biggest move available toward −80%, and it is a cost fix that happens also to be a latency fix.
• Fix 4 (forced tool_choice) deletes a whole duplicate prefill on 18% of turns — roughly 15% of agent-loop input tokens on the leasing path.
• Model change is −20% per turn in the prod A/B ($0.0099 → $0.0079 for Sonnet 4.6), with no latency cost either way. That makes it a clean cost decision.
• Fixes 1, 2 and 5 are latency-only — they cut DynamoDB reads and wall-clock, not Anthropic tokens. Worth stating plainly so nobody double-counts them against the spend target. (They do cut the DynamoDB bill: fix 1 alone removes ~1,300 reads per turn.)
• Unrelated leak found while tracing credentials: both non-loop Haiku calls — classify-message.ts and fair-housing.ts — import the raw anthropic client directly rather than going through clientForProperty(). They therefore bill the metered key even for bench and test-property traffic, which the 2026-07-31 policy puts on the subscription. Two one-line import changes; no latency effect; pure spend.

Median LLM cost per SMS turn today is $0.0084 (475 traces, llmCostUsd). The turn-level spend story is dominated by cache-write tokens, not by output — median output is 88–106 tokens against ~25k of prompt.

Methodology & provenance

What is production data

• Every timing in the stage budget, the per-round cache table, the tool table, the retry rate, the per-model A/B and the cost figures — 1,077 AgentTrace rows queried from propflow-prod via the entityType-createdAt-index GSI, Jul 26 – Aug 2, filtered to the 475 SMS turns whose source is production (unset or production). Gaps between steps were reconstructed from startedAt + durationMs per step against the trace's own startedAt/endedAt.
• Cohort caveat, stated plainly: 466 of those 475 turns are stress-harness synthetics on +1206555xxxx numbers against The Willows — real production code, real production table, real production infrastructure, synthetic identities. Only 9 turns in the window are live humans (7 Camellia, 2 Willows). Those 9 show the same shape with smaller numbers (context assembly 2.6–8.4 s, totals 6.6–21.1 s) because Camellia's data volumes are smaller than the stress corpus's. Read the 475-turn medians as the code path's cost at Willows-scale data, and the 9 real turns as confirmation that the same gap dominates real traffic.

What is probe data

• Per-call DynamoDB timings were measured by a read-only script against propflow-prod, 3 runs per call, no writes. Laptop-to-us-east-1 round-trip inflates every number by roughly 50 ms and fan-outs proportionally more; treated as an upper bound throughout.

What is bench data

• The real loop driven through runPipeline against appfolio-45, 5 runs × 2 scenarios × 3 models (30 turns total). Safety: stress-synthetic sender (stressSuppressionFor() refuses +1206555xxxx unconditionally, so no carrier was reached), bench property only, no real vendor or tenant touched, no prod mailbox written. Credential confirmed [anthropic/client] credential=bearer; ANTHROPIC_API_KEY never exported.
• A sibling POC agent is measuring a before/after on the serial-tool-execution fix specifically; those numbers land separately and are not folded in here.

What is an estimate

• Fix 1 and Fix 2 savings, and Fix 3's latency (not its token) saving. Each is tagged est in the waterfall with its anchor stated.

Relationship to prior work

• Prod Latency Audit & Fix Sprint (2026-07-24/25) fixed the operator page waterfall — full-table scans, cache eviction, polling hygiene. None of its ten PRs touched the agent loop, and its central diagnosis ("reads whose cost is O(everything ever written)") turns out to describe the Clara loop too. This report is the agent-loop half of that same problem.
• ADR-0114 deterministic-first inbound email attacked round count on the email path. The email channel today runs at a 25.0 s median with a 14.2 s context-assembly gap — the same bottleneck, so the deterministic-first work and this work are complements, not substitutes.
• The sibling research dossier's ranked practices are folded in above: #1 (parallel tools) is confirmed as a real defect but measured small; #2 (cache-prefix hygiene) is confirmed and promoted to the top token lever with production evidence; #3 (20-block lookback) is refuted by the per-round cache data; #5 (defend effort: low) is endorsed unchanged.