Mutation controls, exit 137: what the memory actually did

From the issue: “runControls (scripts/check-mutation-control.ts:241) runs them sequentially; memory appears to build across 56 double vitest runs (inference, not confirmed).” If that were the mechanism, two things would follow: a bigger selection would die sooner, and the same selection would die every time.

Sampling the last 150 ci.yml pull-request runs and reading the mutation step out of each CI Checks job log (141 logs; 113 ran the step): nine runs selected every control and finished.

Every one of those is the same single npx tsx process, the same for … of loop, the same accumulating lines[], on the same 31,486 MB / zero-swap machine class as the job that died — and each ran four to five times longer than the 7m 28s pass that did. Diff-mode passes of 138 and 139 controls complete on the same shape too.

Positive control on the sweep: the same scanner, run over the dead job’s own log (120198320103), returns sel=('diff','56') exit=137. A zero therefore means “no other 137”, not “the scanner did not look”. Across the 113 sampled jobs that ran the step there is no second exit 137.

Run 39887146455 on propflowai#21970 has two attempts. Both landed on r8gd.xlarge, 4 cores, 31,486 MB, Swap: 0.

Identical input, identical machine class, one second apart in duration, opposite outcome. A deterministic build-up cannot produce that. It also means attempt 1 was killed at the very end of its pass, within about two seconds of where attempt 2 finished, not “mid-way”.

Measured locally by replicating only the import half of loadControls — importing all 759 scripts/mutation-controls/*.control.ts in filename order, sampling process.memoryUsage(), executing no run() and spawning no vitest:

/usr/bin/time -l peak maximum resident set size: 838,975,488 bytes (839 MB), process exit code 0, 37.97 s real. Two things follow. The checker’s own footprint is under a gigabyte on a 31 GB box, so the parent is not what fills the machine. And the growth that does exist is in the load phase, which is unconditional: all 759 modules and all 759 control-plus-guard source texts are held whatever the selection, so narrowing the selection does not move this number at all.

• Only 87 of 759 controls spawn vitest at all, and 86 of those 87 call execFileSync('npx', ['vitest','run',GUARD], …) with no maxBuffer, so Node’s 1 MiB default applies and a louder run fails ENOBUFS rather than growing. Worst case retention for a 56-control pass is 56 × 2 arms × 1 MiB ≈ 112 MB. The other 672 controls call the subject in-process and return a short string.
• killed[] holds MutationControl objects that loaded already holds for the life of the process — it retains nothing new.
• lines[] holds formatted verdict text. Attempt 2’s full 56-control report is a few tens of kilobytes.

So the specific thing #21973 names — accumulation inside runControls — is capped around a hundred megabytes before you measure anything. The measurement agrees with the arithmetic.

• “killed mid-way through guest-card-reply/own-card-echo-treated-as-new-lead.” No. That control’s two log lines are stamped 02:03:51.313Z; the kill is at 02:10:50 — seven minutes later. It is simply the last control that writes to stdout: every vitest-spawning control runs with stdio: ['ignore','pipe','pipe'] and prints nothing. Which control was running when the process died is not recoverable from the log at all, and that is the real gap.
• “each control’s run() does a double vitest run.” Only 87 of 759 do. The one named above calls intakeSyndicatedGuestCard directly, in-process, in milliseconds.
• “the merge-queue train runs the full set in one process.” It does not run it at all. ci.yml:1708-1714 gates the step on github.event_name == 'pull_request', with a comment saying the queue must not run it, pinned by src/__tests__/mutation-controls-pr-gate.drift.test.ts. There is no train exposure to fix.

Exit 137 is SIGKILL. The step runs with NODE_OPTIONS=--max-old-space-size=6144, so a V8 heap exhaustion in the parent would abort at 134 with a message, not die silently at 137. That leaves a kernel OOM kill on a 31,486 MB box with Swap: 0 — no elasticity whatever, so any transient peak over the line is an instant kill rather than a slowdown — or the instance being taken away.

The timing signature says the instance. scripts/ci/checks-reclaim-signature.sh’s own header names two runs AWS recorded as Server.SpotInstanceTermination that showed nothing but a bare 137 on the job. Reading both of their logs next to this one:

Three for three, the same ten seconds, and in the two AWS-confirmed cases the victim was a step twenty-five seconds old — a step with no memory history to accumulate. The repo’s own classifier agrees on the one we care about: run it over this job’s annotations and it answers reclaimed=true · why=exit 137 with no test failure: treated as a reclaim (negative control: a synthetic test-failure annotation correctly returns reclaimed=false). That is why the step kept going afterwards — steps 26 to 28 completed in the ten-second gap, which a slow systemd shutdown allows and an OOM kill also allows, so it separates nothing on its own. The ten seconds do.

1. Do the instrument, in ci.yml (the step at line 1708). Lift the tsgo step’s sampler onto the mutation-control step in its read-only form — sample and print, no VERDICT=KILL arm — plus a per-control progress line so the log names which control was running when a step dies. This touches no file the mutation-selector work touches, and it is the only change here that buys a fact nobody has — the next 137 on this step gets classified from its own log instead of from three other runs’ timestamps.
2. Do not build the per-control memory bound #21973 proposes. A fresh process per control, or an explicit pool teardown in runControls (scripts/check-mutation-control.ts:241-275), is the fix for a build-up this measurement says is not there — and it would cost a Node start-up plus a 759-module import, about 36 s, on every control. It would also land in the file a parallel lane is editing. Revisit only if the sampler shows a rising trajectory.
3. The cheap half of the real exposure is the box, not the loop. 31 GB with zero swap turns any transient peak into a SIGKILL. If 137s recur after the sampler lands, the lever is the runner shape or a capped maxForks for the controls’ vitest children — both one-line, neither inside runControls.

• gh api reads of 150 run manifests and 141 CI Checks job logs (158 MB downloaded to a scratch dir, 36 s wall at 8-way parallelism). No writes.
• One local npx tsx process importing 759 control modules and calling no run(): 37.97 s real, 839 MB peak RSS, exit 0. It does not execute the refused scripts/check-mutation-control.ts and spawns no vitest; it is the import half only, written as a standalone file in the lane’s scratch dir.
• scripts/ci/checks-reclaim-signature.sh classify on the dead job’s annotations, with a synthetic real-test-failure negative control that correctly returns reclaimed=false.
• No test suite, no typecheck, no build, no branch, no PR.

Sources: propflowai#21973; run 39887146455 attempts 1 and 2 (jobs 120198320103, 120204148798); the nine mode=all jobs listed above; .github/workflows/ci.yml; scripts/check-mutation-control.ts; scripts/lib/mutation-control.ts; vitest.config.ts; scripts/ci/checks-reclaim-signature.sh.