PropFlow.ai PropFlow.ai Docs
Proposedmarkdown source

0110 — Continuous anonymized prod→stage mirror (stream replicator)

Context

Preview testing keeps hitting stale, gappy data. ADR-0097's anonymized prod-subset backfill made stage realistic, but as a manual, windowed batch it drifts the moment it finishes: a call that lands in prod right now has no transcript in preview until someone re-runs six CLIs. The goal raised 2026-07-23: prod and preview should look the same every day — a call comes in, the transcript is visible in both — so anything validated in preview transfers to prod.

The obvious-sounding alternative — dual-writing from the app layer — is not viable here. There is no universal writer seam: the spine covers ~8 identity-bearing entities, conversation-writer covers conversations (ADR-0079), and several Lambdas (org-drift-detector, agent-runtime jobsRepo, concession-watcher) use their own DDB clients. An app-layer hook would miss writers, add latency/failure coupling to the prod hot path, and risk double-firing side effects (Temporal, SQS, Twilio, SendGrid, AppFolio L4). The table stream is the only choke point that sees 100% of writes, and prod already runs Streams (NEW_AND_OLD_IMAGES) with live consumers (tour-schedule CDC, conversation-events bridge). Stream consumers fire no app side effects, so replication at this layer is effect-free and adds zero latency to prod writes.

PII stance (decided in-session): anonymization stays in-flight and mandatory — the ADR-0097 transform runs per record in the replicator. Raw-copy-now-PII-later was considered and rejected: it reverses the documented SOC2 decision, and mirrored real contacts in stage mean one careless preview test can SMS/email a real tenant. Deterministic pseudonyms with entity IDs preserved keep preview structurally identical to prod.

Decision

Two lanes, both built on the ADR-0097 transform (extracted to src/lib/anonymize/):

  1. Nightly batch (scripts/anon-backfill/run-nightly.ts) — unattended orchestration of the existing five stages for the incremental window [latest-manifest-day, today UTC). Runs on the trusted ops box (raw export PII never enters CI); run dir scrubbed in a finally; Slack alert on failure. Bounds staleness at ≤24h on its own and remains the independent verifier (scan-stage-pii) + disaster-recovery reload path after the stream lane ships.

  2. Stream replicator (lambda/stage-replicator/) — an ESM on the propflow-prod stream mirrors every classifiable change into propflow-stage within seconds:

    • Deny list = classify(), the same policy table the batch uses — never ESM filter patterns. skip kinds drop; unknown kinds drop and alarm (UnclassifiedKind — the streaming census). Windowing collapses: a record arriving now is in-window by definition.
    • In-memory sweep dictionary (Person/Claim/legacy Tenant/Prospect rows): cold-start/TTL (15 min) prod spine scan + same-batch incremental fold. Never persisted — no key material at rest.
    • Scrub-then-write (amended 2026-07-24 after the first live hour — PR #4472): after the dictionary sweep, scrubResidualContacts rewrites every remaining non-sanctioned contact shape with a fake seeded by the contact VALUE (deterministic, same +1000 / @example.test namespaces) — covering contacts no spine entity can ever declare (the platform's own TFNs, vendor/staff contacts inside AppFolio enrichment, office numbers). The original park-on-violation contract dead-ended exactly those records in the DLQ (~800 parks/15min; retries can never succeed when the contact can never enter the dictionary) and the core kinds never mirrored. verifyRecordContacts (the dictionary-independent "no non-sanctioned contact literal survives" assertion) still runs LAST as the gate; park remains as defense-in-depth only — NACK → bisect/backoff → metadata-only DLQ + first-message alarm, keys only in logs, never row bodies. Trade-off, accepted: during the new-person race a value-seeded fake differs from the person-seeded fake the dictionary would mint (self-heals on the row's next touch; mirror-wins). Residual, documented: the value-seeded pseudonym is an unkeyed 32-bit hash — unreachability is the guarantee, not unlinkability; thread a shared salt through both lanes if that ever changes.
    • LWW put conditional on mirroredFromProdAt <= :ts (redrive-safe); provenance stamp anonBackfillRunId: 'stream-replicator'. REMOVEs propagate conditional on the stamp — structurally unable to delete stage-native fixture/eval rows — with PII-in-key strings swept first.
    • Prod safety: the IAM role has stream-read + spine-scan read on prod and write on stage ONLY (no prod write permission exists); code belts assert the target table is never *prod* and the source ARN is the prod table.

Row-level policy on stage (three provenance populations): synthetic bench fixtures and eval-only objects are untouched (disjoint ID spaces); mirrored rows are read-mostly scenery where the mirror wins — a preview test's mutation of a mirrored row can be overwritten at any moment. Tests needing durable mutations use bench fixtures or self-created entities.

Cutover (no stage wipe): merge → deploy replicator (ESM created DISABLED by the provisioning script) → enable ESM → run the batch pipeline once with a widened window for full history (its wipe stage clears pre-pipeline prod remnants; overlap with the live stream is benign because both paths emit byte-identical deterministic output). Enable the stream BEFORE the bootstrap export so the 24h stream retention covers the gap.

Consequences

Alternatives considered