Core policies 1b: Clara reads the Pets answer

Eval evidence for #16982. The rows fail on origin/main and pass on the branch.

What was run

This is the Core Policies pack (evals/promptfooconfig-core-policies.yaml), run through the subscription runner (scripts/eval-core-policies-subscription.ts) on claude-sonnet-4-6, with the same model as the grader.

Every policy-bearing input is read by the production readers over synthetic companies: the bridge writes pets.policy through the registry and Clara reads it through the shared knowledge loader. Every row is multi-turn.

The "main" column is origin/main with only this PR's dataset rows copied in. The branch column is the PR branch at ddd9088, ×3. The one later commit, d0c93c0, only changes a type import, so behaviour is identical.

Results

Rowmainbranch ×3
PETS, sms: a 65 lb dog, so the 50 lb limit and manager approvalfail3 / 3
PETS, voice: a 65 lb dogfail3 / 3
PETS follow-up, sms: how many, and the monthly chargefail3 / 3
PETS, building override, sms: the no-pets buildingfail3 / 3
PETS, building override, voice: the no-pets buildingfail3 / 3
CUSTOM POLICY, sms (regression guard)pass3 / 3
NOTHING SET, sms (regression guard)pass2 / 3, then 5 / 5 on re-run
What main gets wrong
  • For the 65 lb dog, Clara never states the 50 lb limit or that the manager approves exceptions.
  • On the follow-up, she never gives the pet count.
  • At the building whose own answer is no pets, she tells the prospect "cats are welcome", from the company's answer.

With this PR, the one Pets entry carries every part, labelled, and the building's answer replaces the company's.

The one miss on the branch

One "nothing set" run came back as tool calls with no reply text, so there was nothing to grade. It was not a wrong answer. With no Pets answer anywhere, Clara's input is byte-identical to main by construction, because the reader adds nothing. The row re-ran ×5 on the branch: 5 of 5.

PropFlow Docs