Cost & Infrastructure · 2026-07-31

Who is burning Opus 5 on the metered key

Goal: the metered Anthropic key should carry only the core Clara loop — live email, phone, and post-call transcript processing. Everything else belongs on the subscription bearer. This investigation attributes the ~$34/day of Claude Opus 5 that appeared on metered billing after the Jul 30 spend-reduction work, so the remaining non-core consumers can be named and moved or downgraded deliberately.

Verdict. All of it is the production app itself. 100% of the Opus 5 spend rides the propflow-prod key (admin usage report, per-key attribution, Jul 29–31). GitHub Actions, Agent Smith / Trinity, Slack bots, and local daemons were swept and are clean. The driver is AGENT_MODEL_DEFAULT = 'claude-opus-5' (#4688, restored 2026-07-28): every background job that rides the repo-wide agent tier in a production runtime now bills Opus on the prod key — and ~85% of the Opus tokens burn between 1–4 AM Denver, when no tenant is talking to Clara. The core loop is the minority of the Opus bill.

1 · Measured baseline (admin API, per-key)

Day (UTC)	Opus 5 cost	Opus 5 tokens	Key	Shape
Jul 29	$37.94	9.2M	propflow-prod (100%)	Mostly 1h-cache writes + cache reads, tiny output (Jul 31: 0.25M uncached / 2.51M cache-write-1h / 4.62M cache-read / 96k out) — batch classification over big transcripts, not agentic coding traffic
Jul 30	$33.71	8.1M	propflow-prod (100%)
Jul 31	$33.82	7.9M	propflow-prod (100%)

Hourly shape (identical both days): 07:00 UTC ≈ 1.2M tokens, 09:00–10:59 UTC ≈ 5.5M tokens — together ~85% of the day. Evening 19:00–22:00 UTC (~1.1M) is real afternoon traffic plus the 22:00 UTC digest. The small Opus 4.8 line ($0.44 on Jul 31) is the conversation-review digest's verifier (VERIFIER_MODEL = 'claude-opus-4-8').

2 · Named consumers of the overnight burn

Window (UTC)	What runs (verified)	Where	Core loop?
07:00–08:00	25 `maintenanceWorkflow` + 22 `walkTurnoverCohortWorkflow` starts (Temporal list, Jul 31)	ECS renewal-worker + Vercel crons (`outreach-reconcile` 07:00)	No
09:00–11:00	24 `conversationTopicStampWorkflow` + 24 `conversationRationaleWorkflow` + 24 `conversationGraderWorkflow` (enrichment sweep re-driven by the 09:00–09:45 Vercel reconcile crons; starter identity is a Vercel function), plus 41 turnover-cohort walks, 19 renewal, 16 prospect-outreach, 11 tour workflows	ECS renewal-worker (log volume triples exactly here) + Vercel	No — enrichment/QA
22:00	`conversation-review-daily` Temporal schedule — Opus 5 reviewer over every conversation of the day + Opus 4.8 adversarial verifier, digest to #agent-smith	ECS renewal-worker	No — QA digest
scattered, business hours	Live Clara agent loop (email/phone/post-call) on `AGENT_MODEL`	Vercel + Lambdas	Yes — the sanctioned metered consumer

The topic-stamp, rationale, and grader paths all import AGENT_MODEL directly (message-topic-classifier.ts:404, message-rationale.ts:757, daily-review/llm.ts:30) — full-transcript input, one-line output, which is exactly the measured token shape.

3 · Layers swept and cleared

Layer	Result
GitHub Actions (all 11 org repos)	Clean.
No `ANTHROPIC_API_KEY` secret exists in any repo (Jul 30 deletions confirmed). All Opus 5 review/mention lanes run on the OAuth reviewer token pool. The one declared metered lane (`intent-evals.yml` trace-replay, allowlisted) is Haiku-only, 12 runs over two days. Timing also excludes CI: review runs spread across the day; the burn doesn't.
Agent Smith / Trinity / Slack	Clean.
Both daemons are OAuth-only by construction (`llm.py` passes `auth_token=` explicitly); tiers are Sonnet 5 / Opus 4.8 / Fable — no Opus 5 reference anywhere. Worker near-idle Jul 30–31; the deliberate metered path (`morpheus_nightly`) did not fire. Trinity is a display identity for the clara seat, same process.
AWS Lambdas (agent-runtime, inbound-processor)	Idle in the burst window (CloudWatch).
Local machine	No metered-key daemons; nothing running in the window.

4 · Latent risks found on the way (not currently billing)

Reviewer token pool is not type-checked — the rotation health probe accepts any credential returning 200; a metered key slipped into the 6-slot pool would bill every Opus 5 xhigh review. Not the current leak (wrong key, wrong timing) but worth a one-line sk-ant-oat prefix assertion in rotate-reviewer-tokens.sh on the mini.
Global ~/.claude/settings.json sets CLAUDE_CODE_SUBAGENT_MODEL: claude-opus-5 — any harness that runs claude with a metered key in env bills metered Opus 5 through every subagent.
Three handoff docs sanction exporting the metered key locally (clara-identity-ground-truth-handoff.md and siblings) to dodge subscription 429s — with AGENT_MODEL now Opus 5, following that instruction is a metered-Opus session.
Agent Smith child_env.py falls back to ANTHROPIC_API_KEY when no OAuth token resolves — unreached today (roster healthy), but it is exactly the leak shape.
Stale doc: CLAUDE.md still claims nightly.yml runs on NIGHTLY_ANTHROPIC_API_KEY; the workflow on main is bearer-only.

5 · Resolution — shipped 2026-08-01

Policy locked (Fede, 2026-07-31): the metered key carries only the core Clara loop (live email, phone, post-call transcripts); everything else runs on the subscription bearer.
Code lever merged (PR #5213): the three enrichment lanes (topic stamping, message rationales, nightly review) now ride backgroundClient() — a scoped bearer client led by ANTHROPIC_BACKGROUND_AUTH_TOKEN, with the Claude Code identity gate and account failover. Fail-open by design: until the token is provisioned the lanes stay on today's metered credential (loud warn), so the merge itself changed nothing. Scope is drift-guarded in both directions; ANTHROPIC_FORCE_METERED=1 and an unpopulated REPLACE_ME secret stub both degrade safely to the default client.
The billing flip is the provisioning act, held for Fede: pick the funding account (whose weekly subscription cap absorbs ~8M tokens/day), mint a long-lived token, create the propflow-anthropic-background-auth-token secret, land the task-def/bootstrap follow-up for the renewal worker, and add the env var to Vercel prod. Proof: the prod key's Opus line in the daily cost report drops to core-loop-only the next day.
Still open as a further lever: these lanes are extraction/QA-shaped — a cheaper tier (Sonnet 5 / Haiku) would also cut their token draw on whichever subscription funds them; eval-gated follow-up.

6 · The true burner, found and fixed (2026-08-01)

The morning after the flip, the 09–10h prod-key Opus burn recurred (3.27M tokens) with zero background-client constructions and idle enrichment (16 topic/rationale workflows, nothing to classify). A full transitive call-graph audit of the worker plus Temporal history then named the real burner: Agent Smith's nightly maintenance eval (maintenance-eval-smith-*, daily 07:03 UTC, ~3 hours) — 17 scenarios driven through the real Clara agent loop against the bench property, on the metered production key. The fingerprint is conclusive: the 1-hour prompt-cache writes in the token mix come from exactly one module repo-wide, conversation-manager.ts. Ruled out along the way: enrichment (idle), collections legal review (Haiku), collections/renewal/turnover composition (pure functions or no LLM), maestro (pinned Sonnet 4.6). §2's original timing-based attribution and the earlier §6 composition theory were both wrong; this section supersedes them.

Fix (PR #5244, merged): the agent loop now selects its billing per conversation — clientForProperty() routes bench-property traffic (the eval) to the subscription bearer and real tenants to the metered key, using the same narrow test-property predicate the grader and send guards use. A real tenant riding a personal subscription is unreachable by construction (test-pinned). The eval-only narrator moved to the subscription client outright.

End state: metered = Clara's real email/phone/post-call work plus the declared cents-a-day intent-evals exception; subscription = enrichment, QA, nightly review, the nightly maintenance eval's bench traffic, CI, evals, and bots. Proof lands in the daily cost report: prod-key Opus should drop from ~$34/day to the real-traffic residual from Aug 2 (the funding account's weekly cap resets 2026-08-02 00:00 UTC; until then bench calls 429 and retry, which is the correct outcome for synthetic traffic).

Sources: Anthropic admin cost/usage API (per-key, per-token-type, hourly), prod Temporal Cloud workflow listings, CloudWatch Logs Insights, and repo/workflow audits of all 11 org repos — pulled 2026-07-31 evening. Dollar projections are labeled estimates.