ADR-0085 — Shard the renewal-saga GSI4 partition key (kill the single-partition hot key)

• Status: Proposed (scoped 2026-07-10)
• Deciders: Gera
• Relates: ADR-0035 D2 (re-keyed the saga off the legacy tenantId onto the canonical Person spine personId — created GSI4 renewal-saga-person-index), ADR-0047 (saga retained as the durable read-model), ADR-0025 (renewal on Temporal).
• Execution handoff: docs/planning/renewal-saga-gsi-hot-partition.md — the dedicated-session build plan (backfill, dual-read cutover, drift guards, verification).

Context

Production throws ThrottlingException: Throughput exceeds the current capacity of your table or index … check if you have a hot key on the renewal detail route (GET /leasing/renewals/[leaseId]), on a DynamoDB Query. Sentry issue 7500098427: 4,481 events over ~6 weeks (first 2026-05-22, last 2026-07-08), bursty, not constant. It is user-facing — a PM's renewal page intermittently errors.

Root cause — a single-partition hot key by construction. The renewal-saga-person-index (GSI4) uses a constant partition key: GSI4PK = "RenewalSaga" for every saga row; GSI4SK = "{personId}#{openedAt}". That constant PK was a deliberate trade-off so getAllOpenSagas() / getAllSagas() can enumerate every saga with one partition-wide Query (the reconcilers/sweeps rely on it). The cost: all saga reads and writes converge on one physical DynamoDB partition, which is hard-capped at ~3000 RCU / 1000 WCU.

The partition is small — 179 rows — so this is not a size problem, it is a concurrent-throughput convergence problem. On the PAY_PER_REQUEST (on-demand) propflow-prod table, a single partition can't exceed that per-partition ceiling and its adaptive capacity lags a sudden burst. Everything below hits that one partition:

• The renewals LIST page (src/lib/data/dynamo/leasing.ts:1284) fans out getSagasByPerson per tenant — N queries to the same partition on every load — and the page polls every 3–5s.
• Four+ periodic sweeps each getAllOpenSagas() (a whole-partition read of all rows): workflow-health-sweep, retry-failed-prepare (auto-cure cron), stuck-saga-sweep, the agent-metrics snapshot.
• Every saga WRITE from the Temporal renewal workflow stamps the same GSI4 partition (state transitions, cohort-walker opens).
• Clara's inbound paths (conversation-manager, NTV handler, signal-bridge, voice personalization) each getSagasByPerson per message.

When a sweep + a polled list-page load + a workflow-write burst coincide, aggregate throughput to the one partition exceeds the ceiling → throttle, surfacing on whichever query lands during saturation. The detail route is the observed victim, not the cause.

Decision

Shard the GSI4 partition key across a fixed fan-out of N partitions, deterministically by personId.

• GSI4PK = "RenewalSaga#<shard>" where shard = hashToShard(personId, N) — a stable, deterministic hash of personId modulo N. GSI4SK is unchanged ("{personId}#{openedAt}").
• getSagasByPerson(personId) computes the same shard from personId and queries one shard partition: GSI4PK = "RenewalSaga#<shard(personId)>" AND begins_with(GSI4SK, "{personId}#"). Same cost as today (one partition Query), now spread across N partitions instead of one.
• getAllOpenSagas() / getAllSagas() fan out N parallel queries (one per shard) and merge — enumeration is preserved, now reading N partitions' worth of capacity in parallel.

N is a small fixed constant (proposed 16) — enough to lift the aggregate ceiling ~16× (≈48k RCU / 16k WCU spread) while keeping the all-sagas fan-out cheap (16 bounded parallel queries over a tiny dataset). N is a code constant, not stored per-row (the shard is recomputable from personId), so it can only change with a re-backfill — documented as a one-way choice.

Alternatives considered

• Re-key GSI4PK = personId directly (each Person is its own partition — the ideal for the per-person read). Rejected as the sole change: it makes getAllOpenSagas impossible via Query (no shared PK to enumerate) — it would need a Scan with entityType = 'RenewalSaga' (full-table, eventually-consistent, its own cost/throttle risk) or a second "all sagas" GSI. Sharding keeps both access patterns first-class. (If the enumerate-all pattern is ever retired, personId-keying becomes the cleaner end state — noted for a future ADR.)
• Interim: cut the amplifiers — batch/cache the renewals-list per-person fan-out, stagger the sweep crons so their getAllOpenSagas calls don't pile onto the same partition simultaneously. Reduces pressure but leaves the single partition — the hot key remains one coincidental burst away. Worth doing as a fast follow but not a fix.
• Provisioned capacity / raise throughput. Doesn't help — the ceiling being hit is the per-partition cap, not table-level; more table capacity can't lift a single partition past ~3000 RCU / 1000 WCU.
• Retry-with-backoff band-aid on the read path. The AWS SDK already retries throttles with backoff; the exception surfacing means retries were exhausted — the partition is genuinely saturated. Masking, not fixing.

Consequences

Positive

• The hot key is gone: per-person reads spread across N partitions; the enumerate-all pattern reads N partitions in parallel; saga writes spread across N GSI partitions.
• getSagasByPerson cost is unchanged (still a single-partition Query). The throttle class on the renewal surfaces disappears.

Costs / risks

• Backfill required. GSI4PK changes on every saga row (open + archived). This is a GSI-attribute re-stamp via UpdateItem (the base-table PK=RSAGA#{id}/SK=META is untouched; DynamoDB re-indexes the GSI). The dataset is tiny (~179 open rows + archived), so the backfill is fast, but it touches a load-bearing renewal index and needs the careful cutover in the handoff doc.
• getAllOpenSagas becomes N queries instead of 1 — more code + a bounded latency add (parallel, so wall-clock ≈ one query). Acceptable; the callers are sweeps/snapshots, not hot user paths.
• Transitional dual-read during migration: readers fan out over the N shards and the legacy "RenewalSaga" partition until the backfill completes, then the legacy read is removed in the closing PR (mirrors the spine-drain pattern — a transitional dual-read, never a permanent dual path; removed in the same arc, drift-guarded).

Verification (the bar for "done")

• After cutover: zero saga rows carry GSI4PK = "RenewalSaga" (drift guard); getSagasByPerson + getAllOpenSagas return identical result sets to pre-migration for a sampled set of persons; the Sentry ThrottlingException on GET /leasing/renewals/[leaseId] stops recurring over a watch window.

See the handoff doc for the shard helper, the exact backfill script shape, the read-cutover sequence, the drift guard, and the rollback path.