Skip to main content
A playbook is one of the house procedures agents follow: auditing a comparison page, pitching a deciding source, writing a how-to. Every change a playbook ships is measured before and after (see Receipts). Pooled across every workspace, those verdicts say how often each playbook works, and agents use them to choose what to try next.

​
The public win rates

Once a night we publish, per playbook, lane, engine and window (14 and 30 days), the share of decided changes after which the brand was named more often. For example: “Comparison pages: 38% named more often within 14 days on ChatGPT (n=212 changes, 31 workspaces)”.
  • One cell per playbook, lane, engine and window. The lane is where the changes worked: own_site, offsite or outreach (see Lanes). The engines are ChatGPT, Perplexity, Google AI Overviews, Claude and Gemini, plus all: every engine together, read “across every engine”. At 30 days the all cell uses the change’s own final verdict; at 14 days, and for each single engine, it uses that change’s +14 or +30 check, decided the same way (enough answers on both sides, enough baseline days, and a win only when the rate went up). An engine a check did not read is left out of that engine’s cell, not counted as a loss.
  • Only decided changes count. A change enters once its +30 check has a final verdict. The early +3 check never counts, and a check that could not be decided (too few answers, too short a baseline) is left out rather than counted as a loss.
  • A floor protects every customer. A rate is published only with at least 20 decided changes from at least 5 workspaces, belonging to at least 5 organisations, so one customer with many workspaces never clears it alone. Below that the cell says “not enough changes yet” and carries no number at all, not even a count.
  • Every rate has its interval. The Wilson 95% interval is shown with it.
  • Nothing names a workspace. Only built-in playbooks and known engines appear; a playbook a workspace wrote itself never does, and no workspace, organisation, domain or name is ever in the payload.
  • A published night never changes. A night is stored once; running it again the same night returns what was published. Our own workspaces are counted, and a cell that includes any says so (includes_own_workspaces: true).
  • Engine-wide changes are left out. See below.

​
Lanes

Every cell names its lane, and each lane is counted from its own changes, under the same rules: a decided +30 verdict, the early +3 never, the same floor, the same Wilson interval and nothing that names a workspace.
  • own_site cells count the changes seen on customers’ own pages (Page changes), exactly as before lanes existed.
  • offsite cells count directory listings, profile updates and Brand Book facts carried to profiles elsewhere, measured from the read that verified them live. A listing whose last read is not live_verified (a mismatch, a rejection, a page that no longer shows it) is never counted.
  • outreach cells count outreach changes with checks of their own. A paid placement counts only once a live read has found its disclosure, and a placement only once it is verified.
An own-site cell’s sentence reads as it always has. Other lanes name theirs: “Directory listings (off-site): 41% named more often within 30 days across every engine (n=24 changes, 7 workspaces)”. The model below learns from every lane’s decided changes.

​
Engine-wide changes

When our control panel flags an engine-wide change, a change whose check compared days inside the flagged fortnight is left out of that window’s cells for the engines the panel moved on, and out of the every-engine cell at that window. A change hit only at +30 still counts at +14 when its +14 check was over before the flagged days began. Each night’s snapshot lists the flagged windows in its reach as engine_shifts: {from, to, engines}, UTC days. The model below learns from the same changes, so it leaves them out too.

​
Read them

All three return the same night. day=YYYY-MM-DD (on MCP, params: {"day": "…"}) reads a past night; leave it out for the latest. lane (own_site, offsite, outreach or social; on MCP, params: {"lane": "…"}) returns only that lane’s cells; leave it out, or send all, for every lane. A night published before lanes existed holds own-site cells only, and reads that way. No credits are used, and the public route may be cached for an hour. The body is day, published_at, min_changes (20), min_workspaces (5), method (wilson_95) and cells. Each cell has: A day that is not a date, or a lane that is not one of those, is a 400. A night with nothing published, including before the first night runs, is a 404. Both use the usual error shape.

​
See them in the app

The Playbooks page (/playbooks, linked from Customize → Skills → Built in) lists every built-in playbook that list(type=playbook) returns, in the same order. Each row shows:
  • Name, lane and purpose. A playbook with no lane reads and reports.
  • Win rate per engine, for the 14- and 30-day windows: the rate, its Wilson 95% interval and n, from the latest published night. A cell under the floor reads “not enough changes yet” and carries no number, never 0%. Before the first night is published, every row reads “not enough changes yet on any engine”.
  • Quality checks, from GET /v1/evals in the playbook’s own lane only: checked on n examples, with each check’s miss rate (for example “Fact check 25% missed”), or not evaluated yet. A run recorded under another version counts as not evaluated, with the reason. The row reports the result as the read gives it and does not call it passing or failing.
  • Held at L3, shown apart from the result, when the read’s gate_level is L3 (see A lane can run lower than you set it). L3 does not by itself mean the checks failed: a playbook whose checks passed is still held when it is not yet allowed to run on its own in that lane, or when its drafts have not been checked under its current instructions. A playbook with a lane that the house does not evaluate is not evaluated and held at L3. A playbook with no lane ships no changes, so no hold is shown.
  • Run. Opens a sheet with the run’s arguments (optional, a JSON object) and the estimate for that call, in credits. Confirming runs playbook with max_credits set to the top of the quote and one idempotency_key per opening, so pressing it twice starts one run, then opens the run’s thread. A workspace whose balance cannot cover the run sees the insufficient_credits message with a link to top up.
The page reads what an agent reads over MCP and REST, and nothing else. When a read fails, the row it fills says it is trying again and retries on its own, so the rest of the page stays on screen.

​
How agents choose

When an agent has more than one playbook that could address your red questions, it ranks them with a model built on the same decided changes, without the ones an engine-wide change hit. It is an input to ranking, never a decision to act: it picks among playbooks the agent may already run, and everything they produce still goes through your approval settings. One rate per playbook and engine, pooled across workspaces. Each decided change is one trial: the brand was named more often, or it was not. For every playbook on every engine the model starts from the rate across all workspaces. Your kind of workspace, shrunk toward everyone’s. Changes are also grouped into segments: brand size band, category, and whether the question started red (not named at all) or amber (named). A segment’s rate moves away from the rest of the workspaces’ rate only as far as its own evidence allows: a segment with 5 changes stays close to the others, one with 500 is close to its own. A segment with no changes gets the overall rate, exactly. How far a segment may move also depends on how much segments really differ: when they all agree, a small segment barely moves. Segments are cut by starting state today; size band and category are read once a workspace carries them. Each change counts once in an interval. The overall rate for a playbook is its changes alone, so 20 changes give an interval as wide as 20 changes allow. A segment’s estimate is its own changes on top of a starting point taken from the other workspaces’ changes, never stronger than those are, so no change is counted twice. Changes planned by a random draw count double. A mission can be split into waves by a random draw, with later waves held back as a control while an earlier one is worked. A change that shipped while its question’s wave was being worked, while a later wave was still held back and no control had been flagged as contaminated, counts twice as much as any other change. Its verdict is still its own before-and-after reading, not the mission’s comparison with its control, so it is not a causal result on its own; the weight reflects that the question was chosen by the draw rather than picked. Ranked by expected wins per credit. Each candidate is scored by its expected win rate × how many of your red questions it would address ÷ its estimated credits, highest first. One choice in ten explores. Nine rankings in ten use each playbook’s expected rate. The tenth draws a plausible rate for each from its own uncertainty instead, so a new or rarely run playbook still gets tried and its rate becomes known. Which rankings explore is fixed by a seed, so the same ranking for the same subject always comes out the same way. With no evidence, the default. When none of the candidates has enough decided changes on that engine, and your workspace has no record of its own for any of them, the ranking says so (fallback: true) and the comparison-page audit leads. Every rate carries its n and its interval. A rate from 3 changes and a rate from 300 can have the same middle; the 95% credible interval says which is which. What you see clears the same floor as the public rates. Anything ranked where you can see the result, the ranking below and Counter it, uses only evidence with at least 20 decided changes from at least 5 workspaces in 5 organisations. A playbook under that floor is treated as if nothing had been decided for it: it cannot move the order, the fallback or an exploring draw, so no ranking reveals how a handful of other customers’ changes went. A segment under the floor is left out the same way, and when what is left out is itself under the floor, no segment of that playbook is shown.

​
What works for your workspace ranks first

The rate above is everyone’s. Your workspace’s own results are blended into it, so a playbook that keeps working for you climbs your ranking, and one that keeps failing or that you keep turning down drops, for your workspace only. Your own record. For each playbook and engine: your decided changes and how many of them won (the same +30 verdicts as above, so no change, inside the noise and a drop are not wins), weighted the same way as above, so a change planned by a random draw counts twice. Added to them is every change from that playbook you declined as wrong or never touch this in the last 60 days, on every engine, each counting as one more change that did not win. That includes the comparison pages and pitches Counter it filed for you. Declining as not now or already done says nothing about the playbook and is not counted. Blended in proportion to how much you have. Your rate is pulled toward everyone else’s in proportion to how much record you have: blended = (strength × everyone else’s rate + your weighted wins) ÷ (strength + your weighted changes + your declines) The strength is fitted from how much workspaces really differ on that playbook: when they all get much the same result, a few changes of your own barely move it; when they differ a lot, your own record counts sooner. It is never below 5, so a handful of changes cannot outweigh everyone else’s rate. The blend weight is your share of the result: (your weighted changes + your declines) ÷ (strength + your weighted changes + your declines). With no record of your own the weight is zero, and your ranking is exactly the ranking every workspace gets. Everyone else’s rate leaves your changes out. The global rate in your blend is every other workspace’s, and it clears the floor above, so your changes never count twice and nothing in your ranking reveals another customer’s results. Nothing crosses workspaces. Your declines and your results move only your ranking. Another workspace’s declines and failures never move yours. A difference inside the noise is never called one. Your rate (your decided changes only, declines aside) is shown beside everyone else’s, with a difference of higher or lower only when it clears the same significance rule as receipts (Fisher’s exact test when either side has fewer than 30 changes, the two-proportion z-test otherwise, at 95%). Anything less is inside_noise: “no difference from other workspaces that we can prove yet”. With nothing to compare, it is none.

​
Every choice is recorded

Each time the model’s ranking leads to work being filed for you (today: Counter it), the choice is recorded once that work is filed, never before and never for a filing that failed. The record names the playbook that was actually filed. When Counter it’s evidence rule filed the comparison page instead of what the ranking put first, the record says so with fallback: true. With it are every candidate’s global rate and n, your own rate, n, wins and declines, the blend weight, the blended rate it was ranked by, and the difference. Each subject is recorded once, so pressing Counter it again records nothing new. Reading a ranking records nothing. Decisions are kept for 90 days. Read them at GET /v1/decisions (?limit=1-100&before=<next_before>), newest first. Over MCP it is analyze with view=decisions. The route and the view arrive together with the release that turns the model on.

​
Counter it

Counter it asks the model to choose between a comparison page and a pitch to a deciding source for the engine the rival gained on, ranked for your workspace with your own record blended in. Both address the one question the alert is about. It files a pitch only when each playbook has evidence to rank by (other workspaces’ changes clearing the floor on that engine, or your own record) and pitches come out ahead; otherwise it files a comparison page. Declining what Counter it filed, as wrong or never touch this, counts against that playbook for your next alerts. The choice is seeded by the alert, so pressing it again or from another door gives the same answer.

​
Read the ranking yourself

GET /v1/playbooks/recommend returns the ranking for your workspace, with each playbook’s rate in the segment you ask for and over every segment, and own: your record, the blend weight, the blended rate and the difference. Once your workspace’s own record is blended in, the segment and overall rates are every other workspace’s: your own changes are left out of both and shown only under own. Over MCP it is analyze with view=playbook_model. The route and the view arrive together with the release that turns the model on.
Nothing here promises an outcome. A win rate is how often a playbook’s changes were followed by the brand being named more often, with its interval; a difference inside the sampling noise is never shown as one.