Skip to main content

​
The short version

A playbook is the written procedure an agent follows for one kind of work, such as an answer block or a comparison page. Every week we check each playbook that is allowed to run on its own. If it is falling short, an agent writes a new version of its procedure. We test that version in shadow, and it replaces the current one only if it does better. This is house learning: it improves the playbook for everyone. Learning from your own declines and edits stays in your workspace and is described separately.

​
When a playbook gets a new version

A playbook gets a candidate version when either of these is true:
  • Its win rate is provably low. Its published win rate (GET /v1/public/playbooks/win-rates) across every engine over 30 days is under 25%, and its whole 95% interval is under 25% too. If the interval reaches 25%, the difference is inside the noise, so we do not call the rate low and nothing is drafted.
  • Its drafts miss. On the sample businesses, a draft it wrote failed the never-say, claim-source or safety check, or its drafts covered less than 70% of what a useful draft covers.
The candidate comes with a written rationale. The rationale says why the playbook is under its band, names each draft that missed and why, and says what the new version changes. The agent writing it sees the current procedure, those misses and a few drafts that passed. It never sees anything from a customer’s workspace: the drafts belong to the made-up sample businesses, and the win rate is the published one, which never names a workspace.

​
Tested in shadow, never shipped

Both versions write two drafts for each of the same 9 sample businesses, and the same checks score both: never-say, claim source (the same claim tracer that checks your drafts), safety and value. A playbook with no win rate yet is judged on these drafts alone. Within 30 days of a version change, the win rate is not used to draft a new version, because it still mixes changes made under the old one. A shadow draft is never shipped and never shown as your change. It exists only while it is being scored, and nothing is stored but the scores. The candidate replaces the current version only if both of these are true:
  • It is better, beyond chance. Its eval score is at least 0.05 higher. The eval score is the average of its pass rate on the checks and its value score. It must also beat the current version draft for draft (same business, same sample) more often than chance would allow: a sign test at the 5% level. A gain that one or two lucky drafts could explain is inside the noise and promotes nothing.
  • It is no less safe. It misses the never-say, claim-source and safety checks no more often than the current version does.
Otherwise it is rejected, and the reason is recorded. Once a version is promoted, every run follows it from the next run on, in every workspace. Its scored drafts also count as the drafts it is checked on, so a lane does not wait at L3 for the new version.

​
Confirmed, or rolled back if it does worse

After a promoted version has run for 30 days, we compare its win rate with the rate when it was promoted. A version promoted before the playbook had a published win rate is compared with the 25% band instead.
  • If the new 95% interval sits entirely below that line, the version is rolled back. The version it replaced comes back with a fresh 30-day window of its own, and both steps are recorded.
  • Otherwise it is recorded once as confirmed. A smaller drop is inside the noise and changes nothing.
If someone edits a playbook’s file after a version was drafted from it, the edited file wins at once. The version stops running and is recorded as retired.

​
Reading a playbook’s versions

GET /v1/playbooks/{name}, or get on a playbook over MCP, returns these fields: Each entry in versions has these fields: GET /v1/playbooks and list with type=playbook show each playbook’s current version. The procedure text itself is not returned.