Skip to main content
Not live yet. The holdout comparison is built but not switched on. No mission builds a control split, no +14 or +30 check runs, and no holdout sentence appears on a receipt, on the scorecard or in a mission’s thread. Neither read is served either: analyze refuses view: "mission_holdout" as an unknown view, and GET /v1/missions/{id}/causal answers 404 on webapp.attensira.com and on api.attensira.com. This page describes how the comparison works once it is live, and the example under Reading it over MCP and the API is the shape both reads will return.
A receipt tells you whether the answers for one change moved. It cannot tell you whether they would have moved anyway: engines update, rivals publish, a whole topic can drift up in a month. A mission answers that second question by keeping a control: questions the mission leaves alone while it works on the others. At the end it reports whether the questions it worked on went green more than the control did. This page explains how the control is built, when the comparison is allowed to speak, and what each verdict says.

​
The report in one line

A mission’s holdout report is one sentence. It appears in three places with the same words: on the mission’s receipt, as a row on the scorecard, and as a message in the mission’s thread at each look.
Questions we worked on went green 18 points more than 20 questions we held back at random at the +30 check (worked +18, held back +0; p<0.001, randomized order, n=3 samples a day).
Every number in it is in points of green rate: the share of answers in which you are named and linked, on a 0 to 100 scale. worked and held back are each group’s own change, so a difference is never mistaken for a direction. Worked questions that fell less than their control did not rise, and the sentence shows both numbers so you can see that. n is how many times a day each question is read on each engine. It depends on your plan: 1 on Starter, 3 on Growth and Business. A comparison read three times a day can see a smaller difference than one read once a day.

​
The two designs

When a mission starts, it decides which of its questions are worked and which are the control, writes that split down with the seed that produced it, and never changes it. Anyone with the seed and the stored inputs can replay the split and check that it was a draw, not a choice made after seeing the results.

​
Randomized order

The default, for a mission with 8 or more questions. A seeded random draw puts the questions into waves (at most 4, each at least 4 questions). The first wave is worked straight away. The later waves are the control until their turn comes, 30 days apart, so nothing is withheld for good: every question is worked, only the order is random. Because the order was drawn at random, a difference between the waves worked first and the waves still waiting is evidence the work made the difference. This is the only design whose report sets causal: true.

​
Matched, not randomized

The fallback for a small mission. Every question is worked, and for each one the mission picks up to 2 untouched questions from your workspace that look like it: the same engines, topic and stage, and a baseline green rate within 15 points. Those similar questions are the control for 30 days. A matched comparison is honest about what it is. Its sentence opens with “Compared with” and ends by saying the comparison does not show cause, because the controls were chosen, not drawn:
Compared with 20 similar questions we left alone, questions we worked on went green 18 points more at the +30 check (worked +18, similar +0; p<0.001, matched, not randomized, n=3 samples a day). The controls were matched, not drawn at random, so this comparison does not show cause.
The scorecard row reads Holdout: +18 points vs similar questions, not proof of cause.

​
No control available

If fewer than 4 usable control questions exist, the mission says so once and shows before and after only. It is a label, not a step for you to take:
No control available: fewer than 4 questions like these were left untouched, so this mission is shown before and after only, not against a control.
A control question whose answers start citing a page one of the mission’s changes touched is dropped from the comparison, because the work has reached it. If that leaves fewer than 4, the comparison has no control.

​
The power floor

Before it estimates anything, the comparison checks whether it could have seen a difference at all. Each mission is sized to see a 10-point difference with an 80% chance. The size depends on how many questions are in each group, how many engines read them, and n. A comparison too small to see 10 points does not report “no difference”. Silence from a test that could not have heard anything is not evidence, so it says so and names the smallest difference it could have seen:
Not enough measurement to compare: with 3 worked and 4 control questions (n=1 sample a day), the smallest difference this mission could see is 23 points and it needs to see 10, so a quiet result would not mean nothing happened.
More questions, more engines or a plan with more reads a day lower that floor.

​
The controls must have tracked before the work

The comparison also checks that the control was a fair comparison to begin with. It looks at the 28 days before the work: if the control questions were already drifting apart from the worked ones (p below 0.05), no estimate is given, whatever happened afterwards:
No estimate: in the 28 days before the work, the 20 control questions already drifted 20 points apart from the 20 worked ones, so they are not a fair comparison (n=3 samples a day).

​
The two looks

A mission is looked at twice, and only twice, at fixed times set when it starts: The +14 check only speaks for an overwhelming difference. If it does, the mission’s answer is in and the +30 check is not needed to decide it. Otherwise the report waits for the +30 check, which keeps nearly all of a 5% error budget. Splitting the budget this way means looking early does not make a false “moved” more likely: together the two looks are wrong at most about 5% of the time. Each look compares the 14 days before the work with the 14 days up to the look. The p-value comes from relabeling questions between the two groups many times and asking how often a split that size appears by chance, not from a formula. Each question counts once however many engines read it. Before the +14 check the report says:
Too early: the holdout comparison is first read at the +14 check on 2026-10-15.

​
Every verdict

No verdict, in either design, says the work “caused” an outcome or that anything happened “because of” it, and none puts a money figure on it. We measure answers, not revenue.

​
Reading it over MCP and the API

Once it is live, the report is the same object on every surface: over MCP as analyze with view: "mission_holdout", and over REST as GET /v1/missions/{id}/causal.
design is randomized_staggered, matched_observational or none. looks holds each look read so far, interim first, with its estimate, p-value, the smallest detectable difference (mde), the pre-work drift and the group sizes, all in points. next_look_at is the next look still to come, or null once the answer is in. A mission with no holdout returns a 404 with the standard error envelope. In the dashboard and in the docs a “question” is what the API calls a prompt.