Podcast · Architecture & decisions
A brand-new company, zero properties of its own, logged in and saw Camellia's real tenants. What the bug actually was, how it sat unfixed for five months, and the two-step fix that makes it structurally impossible instead of merely repaired.
Companion to the multi-tenant isolation architecture decision — the adversarial audit, tenancy map, and research this episode is built from live there.
ARTIFACT-STYLE.md § Where the audio comes from). Once the recording exists it goes to artifacts/audio/multi-tenant-isolation-podcast.m4a, gets uploaded to R2, and this player picks it up with no other change. See episodes/README-multi-tenant-isolation-podcast.md for the exact steps.The mechanism first: five of about twenty record types even carry a field saying which company they belong to, only 27 of 292 data-reading functions take a company id as an input at all, and every check that exists today runs after the data has already been fetched, as an in-memory filter — never as a bound on the database query itself. That is why a company with zero properties of its own could see another company's real tenants, revenue, and conversations on seven pages.
How it was found: not by accident, but by building a genuinely non-staff test login specifically because every prior walkthrough had used a PropFlow staff address — and staff are supposed to see everything, so a broken page and a working page looked identical from the inside.
How it got in: a gap named correctly, in writing, on May 4 — then partially closed, then explicitly deferred on July 30 with a reason but no owner and no date, then invisible for five months because until September 1 there was effectively one customer, so there was nothing to leak between companies.
What real multi-tenant systems do about this — silo, pool, and bridge; AWS's per-request LeadingKeys credential and its one documented hole; Postgres row-level security; and the code-discipline middle path — and the two-step decision on the table: make forgetting impossible first, then make the database itself refuse, even if the code has a bug.
This is a product-bug writeup, not a security disclosure. No customer was attacked and no credentials were stolen — a query fetched the wrong rows. No real external customer has ever actually been exposed: every time this bug was observed, it was a deliberate, staff-run test account. The audio is a NotebookLM Audio Overview generated from the source doc and outline below, and the transcript (once it exists) is machine-made from that recording — proper nouns get fixed by hand before anything is quoted, per ARTIFACT-STYLE.md.
Chapter list is the outline's suggested chapters (see the Outline tab). It becomes a real, clickable chapter list off the audio's own cue points once the recording exists.
episodes/README-multi-tenant-isolation-podcast.md for the exact steps once Fede hands over the .m4a.The brief this episode is generated from. It is the input to NotebookLM, not a transcript of what the hosts said.
# The Query That Never Filtered by Company *Source brief for a two-host technical deep dive. Audience: PropFlow's own team and anyone who wants to understand how a brand-new customer briefly saw another customer's real data — what the bug actually was, how it was found, why it sat unfixed for five months, and what PropFlow is doing so the fix cannot be half-done or forgotten again. This is a product-bug writeup, not a security disclosure: the vocabulary throughout is "bug," "gap," and "check," never "exploit" or "vulnerability," because that is what it was — a query that fetched the wrong rows, not anyone breaking in.* --- ## 1. What happened On September 7, 2026, a brand-new company admin account — created that morning, invited into a company called Western Slope Property Management, with zero properties of its own — logged into PropFlow and saw someone else's tenants. Not a placeholder. Not a demo row. A real list: 173 real people, with real names, real unit numbers, real phone numbers, spread across three real properties, including Camellia Apartments. The same login also saw a real dashboard — 80.4% occupancy, $319,274 in monthly revenue, 457 live prospects — belonging to a company it had no relationship with. It saw real conversation threads: a caller's name, a live voice-call summary, a maintenance topic. It saw another company's renewal terms, unit by unit. None of it was this account's data. This account owned nothing. That is the headline, and it is worth sitting with before anything else: a company that had just signed up, using the product exactly as intended — log in, look at your dashboard — was shown another company's tenants. Not because anyone attacked the system. Because a handful of pages, when asked "what should I show this admin," answered "everything in the database" instead of "everything this admin's company owns." ### How it was found It was found by testing, on purpose, with an account built to fail loudly if something was wrong. The team logged in as a fresh company admin — no history, no prior test residue, an email address that had never touched PropFlow before, and specifically *not* a PropFlow staff email — and clicked through the product the way a real customer would: Dashboard, Leasing, Maintenance, Tenants, Conversations, Settings. That last detail — not a staff email — turned out to be the whole reason this had never been caught before. PropFlow's own team signs in with `@propflowai.co` addresses, and those addresses are deliberately treated as staff, which means they are *supposed* to see every company's data. Every internal walkthrough before September 7 used one of those addresses. So every time someone from the team clicked through these pages to check that things looked right, the pages looked right — because staff are allowed to see everything, so a page showing "everything" looked exactly like a correctly-working page. The bug was invisible from the inside because the people checking it were, by design, allowed to see what a real customer should never see. The first attempt to reproduce the bug, on September 6, actually used a staff login by accident and concluded — wrongly, on the page's first write-up — that this was just staff doing what staff do. That conclusion got corrected the next day, once a genuinely non-staff login showed the identical symptom. It is worth naming plainly, because it is a small case of the same larger lesson: if the account you're testing with is allowed to see everything, you cannot tell "correctly scoped" from "the wall isn't there" by looking at the screen. You have to test as the person who is supposed to be fenced in. Once that test ran with a real customer-shaped login, the pattern held up everywhere it was checked. Seven pages leaked: Dashboard, Leasing overview, Leasing → Prospects, Maintenance overview, Maintenance → Routine Maintenance, Tenants, and Conversations. Every one of the *detail* pages behind them — Renewals, Turnovers, Work Orders, Vendors, Calendar — correctly showed empty states the whole time. So this was never "the app has no walls at all." It was a specific, repeatable shape: the overview and list pages that summarize "everything" forgot to ask "everything belonging to whom," while the narrower detail pages happened to ask the right question. ## 2. The numbers, once someone went looking everywhere Finding the bug on those seven pages was the beginning, not the end, of the investigation. The next question was the one that matters more: is this seven isolated mistakes, or is it how the whole product is built? A full audit answered that question, and the numbers are the reason "fix seven pages" is not the plan. **Five of about twenty record types even record which company they belong to.** Tenants, work orders, turnovers, renewal records, tours, leases — none of those rows carry a company id at all. If you have the row in hand, there is no field on it that tells you whose company it is. The only way to find out is to walk backwards: look up the property that row belongs to, then look up that property's company — a second lookup that depends on someone remembering to do it. **Twenty-seven of 292 functions that read data from the database take "which company is asking" as an input.** That's about nine percent. The other 265 — including the functions behind fetching a property, fetching a tenant, fetching a conversation, fetching a work order — will hand back the row for anyone who asks, regardless of who they are. And even some of the twenty-seven that *do* accept a company id treat it as optional: one core function's own code comment says, in effect, "leave it out and you get the whole platform; supply it and you get just that company." Nothing forces a caller to supply it. The safety only exists for the caller who remembers to ask for it. **Every check that stops a company from seeing another company's data runs after the data has already been fetched.** This is the structural heart of the problem. The pattern, everywhere it exists, is: read everything, then filter down to what this person should see, in application code, in memory. Nothing about how the database itself is asked the question is bounded to the right company. It's a lock on the door of a room you've already walked into and are already holding everything from. **Across the whole product, 662 different places can read data** — API routes, pages, background jobs, the tools Clara uses to look things up. When the team logged in as an outside company and tried 25 of the highest-value ones for real, **12 handed back another company's data.** Not 12 out of 662 — 12 out of the 25 they specifically checked, meaning roughly half of what got tested, failed. Put those numbers together and a picture forms: the database itself has no idea which rows belong to which company. The code that reads the database mostly doesn't ask. And the few places that do ask, ask *after* they've already gotten the answer to a bigger, unscoped question. This is not "twelve buggy routes." This is a product built, from the ground up, on the assumption that there is basically one company using it — because for most of its life, there was. ## 3. How it got in, and how it sat there for five months None of this happened overnight, and none of it is new. The trail goes back four months, and it is worth walking slowly, because the interesting failure isn't the original gap — it's what happened after someone found it and wrote it down. **May 4, 2026: companies become a real thing in the product.** Before this, PropFlow effectively had one customer. The very first code change that introduced "companies" as a concept came with its own warning, written into that change's own description: a company admin's "see everything" view needs a second check — is this actually my company's property, not just any property — and that check does *not* ship yet. The team knew, on day one, that this piece was missing. It was explicitly deferred, on purpose, to be handled later. **May 5: a partial rollout, on purpose.** Eight of eleven known pages get the company check. Three more are named and set aside, deliberately, as a separate decision for later. This is a reasonable way to ship — prove the pattern on eight pages before doing all eleven — but "later" needs to actually come back around, and it's the first thread that eventually gets dropped. **May 19: an incident writeup explains why none of this had bitten anyone yet.** Every real customer was still on one shared default company. There was, quite literally, nothing to leak between companies, because there was only one company. The gap was real but invisible — like a fence with a hole in it around a yard with no animals in it yet. **July 4 and July 6: the correct fix gets built, and the scope of the problem gets written down.** A proper version of the check — "is this my company's property," not just "is this my property" — gets built and wired up, but only for one narrow area (email logs), not everywhere. Two days later, a full internal audit puts a number on the gap in writing: roughly 40 pages use only the narrower, wrong check and skip the company-level one entirely for company admins. **July 30: the two biggest remaining gaps get fixed, except two.** Most of those 40 pages get cleaned up. The two hardest ones — Dashboard and the Leasing overview — get explicitly left alone, and the reason is written into the code itself: fixing them means touching shared code used across dozens of pages at once, so it "needs its own change, not a drive-by." This is a completely reasonable engineering call. Big changes deserve their own dedicated pass instead of being bolted onto whatever else is happening that day. **And this is where it goes wrong** — not in the decision to defer, but in how the deferral was recorded. The note said what was wrong and why it was being left alone. It did not say who owns coming back to close it, and it did not say when. A written-down gap with a name and a date attached gets picked back up on schedule. A written-down gap with neither just becomes background noise — technically tracked, practically invisible, indistinguishable from a hundred other comments in the codebase that nobody is actively worried about. **Then, for five months, nothing forced the question**, because of the same fact from May 19: there was still, for all practical purposes, one real customer. "Everything in the system" and "everything my company owns" were the same list. A gap that leaks data between companies cannot be observed when there is only one company to leak between. **September 1, 2026: Camellia Apartments and Yale 25 Station go live as separate, real companies.** For the first time, PropFlow has more than one real customer with real data on the platform. The fence with the hole in it now has animals in the next yard over. Nobody's account had actually been exposed yet — the only people who had signed into a second company's admin role between then and September 7 were PropFlow's own staff, running deliberate tests — but the condition for a real leak now existed, for the first time, and it sat there for six days before anyone specifically went looking with the right kind of test account. **September 6–7: the right test finally runs**, the way described in Section 1, and the five-month-old, correctly-anticipated, explicitly-named gap turns out to be exactly as real as the May 4 comment said it would be. Stack the causes up and a pattern falls out that is bigger than this one bug. A known risk, correctly identified in writing, sat open for five months because (1) the note that deferred it had no owner and no date, (2) the automated test suite that checks for this class of problem covered Clara's phone, text, and email conversations but had no equivalent for the dashboard and list pages a human clicks through, and (3) every person who ever looked at those pages to check they were fine was, by the nature of how PropFlow's own team logs in, exempt from the very rule being tested. Three different kinds of "nobody would have caught this," stacked on top of each other. ## 4. What real multi-tenant systems do about this PropFlow isn't the first product to serve more than one customer out of one shared database, and the industry has a name and a shape for each way of solving it. Understanding the landscape is what turns "we should fix this" into "here's specifically what fixing this means." Amazon Web Services, which is what PropFlow runs on, describes three basic models for keeping one customer's data away from another's when everyone shares the same infrastructure: - **Silo** — each customer gets their own separate table (or database, or account). Perfect isolation, because there's physically nothing to leak into; the cost is that you now run and maintain N copies of your storage instead of one, and any view that spans customers (which PropFlow's own staff need constantly) has to stitch results back together from N places. - **Pool** — one shared table serves everyone, and the wall is enforced at the level of each individual row, using the record's own structure — specifically, making the company id the leading part of how each row is organized and found. This is the default shape for most software at PropFlow's size, and it's the direction the proposed decision (Section 5) points. - **Bridge** — a mix: some layers pooled, some siloed, chosen per-layer based on where the tradeoff makes sense. For the pooled model on the specific database PropFlow uses — DynamoDB — Amazon's own documented pattern is to make the company id the *leading* part of every table's key and every index's key, then hand out a short-lived, temporary credential per request that is cryptographically restricted, by Amazon's own permission system, to only ever touch rows whose leading key matches that one company. This is called `LeadingKeys`. It is not a check the application code performs and could get wrong — it's Amazon's own infrastructure refusing the request at a level the application never even gets a chance to mess up. It has one well-documented, important hole: it does not cover a "Scan" — the database operation that reads every row in a table regardless of key. Amazon's own guide says so directly: Scan isn't covered because Scan returns everything no matter what the leading key says. PropFlow's own audit used exactly this kind of full-table read to do its counting, and at least one existing feature (a fallback path on the dashboard) uses one too. Any protection built purely on `LeadingKeys`, without also banning raw table-wide reads, would leave that door open. The relational-database version of the same idea is Postgres's **row-level security**, where a rule attached to a table filters every query automatically, based on which company the current database connection says it's acting for. It's the default recommended pattern for most Postgres-based multi-tenant products at PropFlow's scale. It has two footguns worth knowing about even though it isn't the path PropFlow is taking: a shared-connection-pool setup can accidentally let one company's setting bleed into the next request's connection if it isn't set correctly inside each individual transaction, and any database login with elevated privileges can simply skip the rule entirely. Moving PropFlow onto Postgres to get this would mean rebuilding the entire way the product stores and reads data — modeled as a project measured in months, not weeks, for the size of the engineering team available — which is why it was considered and set aside (see Section 5). Short of a database-enforced wall, there's a middle path several teams use: a **mandatory, code-enforced tenant context**. In practice this means every single function that touches the database is written so that it *cannot run at all* unless something has told it which company is asking — not a parameter you can forget to pass, but a value that has to be present in the surrounding request or the code simply won't execute — combined with an automated rule that blocks anyone from adding a new database call that bypasses this layer, plus a battery of tests that log in as one company and check, on every single page and endpoint, that zero rows from any other company ever come back. This is a wall made of discipline enforced by tooling, not a wall made of infrastructure. It closes exactly the kind of bug found on September 7 — a page that forgot to filter — but it has one honest limitation: it only protects code paths that go through it. A new script, or a new background job, that reaches the database directly and skips this layer is still unprotected. It needs a database-level backstop behind it to be a true hard floor, not just a much stronger house rule. ### What each approach actually costs | Approach | What enforces it | What it doesn't cover | Rough effort | Who uses this at PropFlow's size | |---|---|---|---|---| | Mandatory code-level company check + automated tests | Code discipline, enforced by tooling | A new script or job that skips the required layer entirely | 2–4 weeks | The default first step for most software companies before they invest further | | Per-request scoped credentials on the existing database (`LeadingKeys`) | The database provider's own permission system | Full-table scans; anything that needs to see across companies still needs a separate path | 6–10 weeks, and requires re-organizing how every record is stored and looked up | Standard in Amazon's own reference designs for this exact database | | Separate table per company | The database provider's account/permission boundary | Nothing, for isolation — but any staff view across companies now has to fan out across every company's table | 3–5 weeks migration, plus ongoing per-company setup work | Used by vendors serving a handful of large, regulated customers | | Separate cloud account per company | The cloud provider's account boundary itself | Nothing — but every shared service (background job runner, the phone system, email) now needs cross-account access | Months, realistically a re-platform | Reserved for enterprise/regulated software; essentially never used at PropFlow's scale | | Move to a different kind of database with built-in row rules | The database engine itself | A database login with elevated access still bypasses it; a shared connection pool has to be configured carefully or it leaks | Months — a full rewrite of the data layer | Extremely common for products *built* on that kind of database from day one — not for one already built on something else | ## 5. The proposed decision: a wall in two steps Given all of that, the proposal on the table is not to pick one item off that list and declare victory. It's two steps, done in order, each one closing a different part of the gap. **Step one — make forgetting impossible.** Every function that reads data gets required, not optional, to be told which company is asking — no default value, so code that skips this simply doesn't run. An automated check blocks anyone from writing a new database call that skips it. The way data gets fetched also changes shape: instead of "get everything, then filter afterward," a request turns "which company is asking" into "which specific properties that company owns," and the database is only ever asked about those specific properties — never the whole table. Reading an entire table at once gets banned outright as a coding pattern. PropFlow's own staff, who legitimately need to see across every company sometimes, get a separate, clearly-logged "all companies" mode for that purpose — never a plain read that simply forgot to narrow itself. And the manual, one-time test that found this bug — log in as an outside company, check that nothing comes back — becomes a permanent, automated check that runs on every future change to the product, forever, instead of something a human has to remember to re-run. This step is estimated at three to four weeks. **Step two — make the database itself refuse, even if the code has a bug.** Every record gets its company id filled in, worked out from the property it belongs to, and that company id becomes the leading part of how every table and index is organized — the `LeadingKeys` approach described above. Every request gets its own short-lived, temporary database credential that Amazon's own systems restrict to that one company's rows — meaning even a bug in PropFlow's own code, a mistake nobody caught in review, physically cannot reach another company's data, because the credential in hand doesn't have the keys to that door. Staff keep a separate, logged credential for legitimate cross-company work. And the one gap `LeadingKeys` itself doesn't close — full-table reads — is already banned by step one, so the two steps together cover each other's blind spots. This step is estimated at six to ten weeks and starts only after step one is done. **What got rejected, and why.** A separate database table for each customer, and a separate cloud account for each customer, were both considered and set aside — not because they don't work, but because once PropFlow is serving fifty customers instead of a handful, both create real, ongoing operational cost, and staff still need ways to see across every customer for support and reporting, which becomes its own significant piece of new infrastructure under either approach. Moving the whole product onto a different kind of database, to get Postgres-style row rules, was rejected for the same reason discussed above: it would take a full rewrite to arrive at roughly the same protection step two already gets on the database PropFlow already runs. ## 6. The policy that holds until step one ships Here is the part that matters most in the meantime, because a fix that takes three to four weeks still leaves a gap open for three to four weeks. Until step one ships, nobody outside PropFlow gets the company-admin role — including Western Slope's own Jay, who would ordinarily be the very first person to receive it. This isn't a workaround dressed up as a policy. It's the actual rule while the real fix gets built. As of today, nobody outside PropFlow holds that role, so the exposure is theoretical — the bug is real, but there is no real external account positioned to be harmed by it right now. That stops being true the instant a real outside person is given that role, which is exactly why the role isn't being handed out until the structural fix is in place. ## 7. How this gets proven, not just claimed The last thing worth being specific about is what "fixed" actually means, because "we fixed it" and "we can prove we fixed it, and prove it stays fixed" are different claims, and only the second one is worth acting on. The plan is to run the automated test suite described in step one against two real environments — a company called The Willows and PropFlow's own internal sandbox — using login accounts that have genuinely never touched either of those companies before, the same shape of test that found this bug in the first place. The report that comes back says, plainly, what was fixed, shows the wall actually holding in each case, and explains why the same kind of leak structurally cannot happen again — not just that it didn't happen on this particular test run. Nobody has to click through the product by hand to confirm this; the test is built to fail loudly and specifically if the wall isn't there, the same way this whole investigation started.
The focus guidance. Paste this into NotebookLM alongside the source doc — it answers “what should the hosts focus on in this episode?”, and it is what stops a two-host format from wandering into summary.
# The Query That Never Filtered by Company — focus guidance for the hosts *Paste this into NotebookLM alongside the source brief. It answers "what should the hosts push on in this episode?" and it is what stops a two-host format from wandering into a table-of-contents reading of the brief. It carries no facts of its own — every number and claim below is already in the source doc; this file is direction, not content.* Audience: PropFlow's own team, plus anyone technical who wants to understand a real cross-customer data bug and how PropFlow is closing it. Tone: two colleagues talking honestly about a real mistake, including the parts that are uncomfortable — not a security-drama retelling, and not a blame session. This is engineers explaining an engineering problem to other people who build software. ## 1. Open on the moment, not the mechanism Start with the concrete image: a brand-new company, September 7, zero properties of its own, logs in for the first time — and sees 173 real people's names, phone numbers, and unit numbers that belong to Camellia. Say it plainly and let it land before explaining anything about why. The reason this episode exists is that sentence, not the architecture that follows it. Do not open with DynamoDB, partition keys, or any technical vocabulary. The first two minutes should be legible to someone who has never touched the codebase. The technical detail earns its place once the stakes are established, not before. ## 2. Say clearly, early and more than once: this is a product bug, not a break-in This is the single most important framing rule for the whole episode, and it needs to be stated directly rather than implied. Nobody attacked anything. No credentials were stolen. No one bypassed a login. A logged-in, legitimate customer clicked a legitimate page, and the page answered a different question than the one it should have — "show me everything in the system" instead of "show me everything my company owns." Use the word "bug." Use "gap." Use "a query that never filtered by company." Never use "exploit," "vulnerability," "breach," or "hack," even casually, even as a joke. If a host reaches for security-thriller language, the other host should redirect back to the plain description: a query that fetched too much. ## 3. Explain how it was found, and dwell on the one detail that makes the retro land The find matters as much as the bug. It wasn't an accident — a fresh, deliberately non-staff login was built specifically to test this, because someone realized that every previous check had been run by a PropFlow staff account, and staff accounts are supposed to see everything. That's the crux: if the account doing the testing is allowed to see everything, a broken page and a working page look identical on screen. You cannot tell "correctly walled off" from "there is no wall" by looking at a screen with permission to see through it. Spend real time on the detail that the very first attempt at this test, on September 6, used a staff login by mistake and concluded — wrongly — that the whole thing was a harmless staff quirk. That conclusion got corrected the next day once a real non-staff login showed the identical symptom. This is worth including specifically because it's a small, honest example of the larger lesson, and it happened inside the same investigation, not in some other unrelated incident. ## 4. Walk the numbers as a chain, not a list The numbers land hardest in sequence, each one explaining why the previous one is dangerous, rather than as an unordered pile of statistics. - Start with: only 5 of about 20 kinds of records even have a field recording which company they belong to. If the row doesn't say whose company it is, everything downstream depends on someone doing a second lookup correctly. - Then: only 27 of 292 data-reading functions take "which company" as an input at all — about 9%. The other 91% will hand back a row to anyone who asks, no matter who's asking. - Then: even some of that 9% treat the company id as optional, not required — leave it out, and you get the whole platform back. - Then, the closer: every check that exists runs *after* the data has already been fetched, as an in-memory filter, never as a bound on the database query itself. - Land on the real-world consequence: 662 different places in the product can read data, and when the team actually tried 25 of them live as an outside company, 12 came back with someone else's real data. Emphasize that's 12 out of the 25 actually tested, not 12 out of 662 — the denominator matters and should be said out loud, not glossed. Push back gently on any temptation to round this up to "half the product is broken." The finding is precise: a specific class of page (overview/list pages that summarize "everything") failed consistently; detail pages behind them did not. Keep that distinction alive rather than flattening it. ## 5. Make "held, with no owner" the retro's real lesson This is the section to slow down on, because it's the one with a transferable lesson for how the team writes down deferred work in general, not just this one bug. Walk the timeline in order — May 4 (companies introduced, the gap explicitly named as "not shipping yet" in the very first commit), May 5 (a deliberately partial rollout, 8 of 11 pages, 3 more deferred on purpose), July 4 and July 6 (the real fix built narrowly, then the full scope of the problem documented in an audit), July 30 (most of it cleaned up, the two hardest pages explicitly left for later with a reason written into the code). Then stop and name the actual failure: it isn't that the fix was deferred. Deferring hard work on purpose, with a documented reason, is a normal and often correct engineering decision. The failure is that the deferral note said *what* and *why* but never *who* and *when*. A gap with an owner and a date gets revisited on schedule. A gap with neither becomes invisible — technically tracked, practically forgotten, indistinguishable in the codebase from a hundred comments nobody is worried about. State the rule this produces plainly, because it's the most reusable takeaway in the whole episode: "known gap, held deliberately" is not a complete sentence anymore. It needs a name and a date attached, every time. ## 6. Explain why single-customer reality hid it, without treating that as an excuse May 19's incident note said it outright: there was effectively one customer, so there was nothing to leak between companies. That's a real, structural reason nobody got burned for five months — not carelessness, just the honest fact that a wall with nothing on either side of it can't be tested by waiting for something bad to happen. Say this clearly, but immediately follow it with the point that this is exactly the kind of gap that should have been closed *before* the second customer arrived, precisely because it can't be observed until then — waiting for a second customer to prove the wall doesn't exist is not a real safety strategy. September 1, when Camellia and Yale 25 Station went live as real companies, is the moment the theoretical risk became a real one; September 7 is just when someone happened to specifically go looking for it. ## 7. Give the industry landscape its own clean section, and keep it concrete Once the incident and its retro are fully covered, pivot to "how do other systems actually solve this" as a separate, distinct segment — don't blend it into the incident narrative. - Name AWS's three shapes plainly: **silo** (a table per customer — perfect isolation, real ongoing cost), **pool** (one shared table, wall enforced per-row), **bridge** (a mix). Say which one PropFlow is closest to today (pool, in principle) and which one the proposal moves toward (pool, done correctly). - Spend real time on `LeadingKeys` — the idea that a request gets a temporary credential that Amazon's own systems physically restrict to one company's data, so even a bug in PropFlow's own code can't reach the wrong rows, because the key in hand doesn't open that door. This is the single best "aha" moment available in the episode: a wall enforced by infrastructure that PropFlow's own code cannot accidentally talk its way around. - Immediately follow that with its one real hole: `LeadingKeys` does not cover a full-table "Scan," by Amazon's own documentation, because Scan reads everything regardless of key. Note, pointedly, that this audit's own method of counting the problem used exactly that kind of full-table read — so the hole isn't hypothetical, it's already in active use somewhere in the product today. This is why step one's ban on full-table reads matters as much as step two's database-level wall; neither one alone closes the whole gap. - Cover Postgres row-level security as the relational cousin of the same idea, briefly — it's the industry default at PropFlow's size for products built on Postgres from day one, but moving PropFlow onto it would mean rebuilding the entire way it stores and reads data, which is a rewrite measured in months, not a tweak. Mention its own footguns (a shared connection pool leaking one customer's context into the next request if not configured carefully; certain elevated database logins bypassing the rule entirely) so the comparison doesn't read as "the other option is perfect and free." - Cover the code-discipline middle path — every database call required to be told which company is asking, enforced by tooling rather than habit — as what step one actually is. Be honest about its limit: it protects everything that goes through it, and nothing that doesn't, so a new script or job that skips the required layer is still a hole. That's exactly why it needs a database-level backstop behind it, which is what step two is for. - The cost table in the brief is meant to be talked through, not read aloud verbatim. Hit the shape of it — cheap-but-partial (code discipline) vs. real-infrastructure-wall-but-slow (per-request credentials) vs. clean-but-expensive-at-scale (separate tables or accounts) vs. correct-but-a-rewrite (switch databases) — and let the numbers (weeks vs. months) do the work of explaining why the two-step plan picks what it picks. ## 8. Present the two-step decision as one plan, not two options to choose between Step one and step two are sequential and both required — never frame them as alternatives or as "step one for now, maybe step two later if we feel like it." Step one closes the code-discipline gap and makes it permanently checked by automation instead of a habit. Step two closes the one thing step one can't close on its own — a bug in the code that step one's own rules failed to catch — by making the database itself physically incapable of returning the wrong company's rows, even then. Say explicitly: step two is the real floor. Step one is necessary and valuable on its own, but it is still a wall made of code discipline, which is exactly the kind of wall that failed here in the first place, twice, over four months. Step two is what makes "even if the code has a bug" true. Be specific and a little unglamorous about the cost of step two: a full re-key of every table and index in the product so the company id leads everywhere, which is not a quick migration — it's touching the shape of essentially the entire database. That's the price of the real floor, and it's worth saying so rather than making step two sound like a small addition to step one. Cover the rejected paths briefly and move on — separate table per customer, separate cloud account per customer, and moving to a different database entirely were all considered and set aside because, at the scale PropFlow is heading toward, each one trades a real cost (ongoing operational overhead, or a full rewrite) for a protection the two-step plan already achieves more cheaply on the infrastructure PropFlow already has. Don't relitigate these; state the reason once and keep moving. ## 9. Give the interim policy its own beat, and be honest about what it is Until step one ships, nobody outside PropFlow gets the company-admin role — a specific, concrete rule, not a vague "we'll be careful." Say plainly that this makes the current exposure theoretical rather than real, because nobody outside PropFlow holds that role today, and say equally plainly that the policy exists precisely because that stops being true the moment someone outside PropFlow is handed the role. Don't let this section sound like a footnote — it's the thing standing between today and step one shipping. ## 10. Close on what "proof" means, and make it concrete End on the verification plan, because it's the most reassuring part of the episode and it deserves to land last. The test that found this bug — log in as a company that owns nothing, confirm zero rows from anyone else come back — becomes a permanent, automated check that runs on every future change, against two real environments, using logins that have never touched them before. The report that comes back says what was fixed and explains why the same leak structurally can't happen again — not "it passed this one time." Nobody has to click through the product by hand to believe it's fixed. That's the whole point of building the test the same shape as the bug that was found. ## What to avoid - No security-thriller language anywhere — no "exploit," "hack," "breach," "attacker," "vulnerability." Say "bug," "gap," "leak" (as in "the page leaked another company's rows," a plain description of what the code did, not a security term of art), "the wall," "the floor." - No blaming any individual, team, or past decision as careless. The May 4 deferral was reasonable; the July 30 partial fix was reasonable. The failure is specifically the missing owner-and-date on the deferral note, not that anyone was lazy or careless. - No jargon dropped without a one-sentence gloss the first time it's used — "partition key," "GSI," "row-level security," "STS," "LeadingKeys" all need a plain-English handle before the technical name, every time, even if it feels repetitive to a technical listener. - Don't read the cost table row by row as a list. Talk through the shape and the tradeoff; the table is a reference, not a script. - Don't let step one and step two blur into "the fix" as a single undifferentiated thing — the whole value of this section is that they close two different holes, and a listener should come away able to say what each one specifically does. - Don't speculate about what a real customer might have seen beyond what the brief states. The brief is explicit that no real external customer has ever actually been exposed — every instance of the bug being observed was a deliberate, staff-run test. Keep that fact intact; don't dramatize it into "customers were exposed." - Don't resolve the episode with reassurance beyond what's earned. The interim policy is a real, working mitigation, not a fix — say that the fix is still three to ten weeks of real engineering work away, in two parts, and let that stand as the honest ending rather than smoothing it into "so everything's fine now." ## Suggested chapters 1. The Login That Saw Someone Else's Tenants 2. Not a Break-In — a Query That Never Filtered by Company 3. How You Test for a Wall You're Allowed to Walk Through 4. The Chain of Numbers: 5, 27 of 292, 662, 12 of 25 5. May 4: The Gap Was Named on Day One 6. "Held, Deliberately" Is Not a Complete Sentence 7. Why One Customer Hides a Multi-Customer Bug 8. Silo, Pool, and Bridge: How Everyone Else Does This 9. The Key That Physically Can't Open the Wrong Door — and the One Thing It Can't Cover 10. Two Steps, Not One: Make Forgetting Impossible, Then Make the Database Refuse 11. What Got Rejected, and Why It Was the Right Call 12. The Rule Until Step One Ships 13. What Proof Actually Looks Like