> ## Documentation Index > Fetch the complete documentation index at: https://docs.attensira.com/llms.txt > Use this file to discover all available pages before exploring further. # Causal proof: comparing with questions we left alone > How a mission compares the questions it worked on with a control set of questions it left alone: the randomized and matched designs, the power floor, the two looks at +14 and +30 days, and the exact wording of each verdict. > [!WARNING] > > **Not live yet.** The holdout comparison is built but not switched on. No mission builds a control split, no +14 or +30 check runs, and no holdout sentence appears on a receipt, on the scorecard or in a mission's thread. Neither read is served either: `analyze` refuses `view: "mission_holdout"` as an unknown view, and `GET /v1/missions/{id}/causal` answers `404` on `webapp.attensira.com` and on `api.attensira.com`. This page describes how the comparison works once it is live, and the example under [Reading it over MCP and the API](#reading-it-over-mcp-and-the-api) is the shape both reads will return. A [receipt](https://docs.attensira.com/measure/receipts.md) tells you whether the answers for one change moved. It cannot tell you whether they would have moved anyway: engines update, rivals publish, a whole topic can drift up in a month. A mission answers that second question by keeping a **control**: questions the mission leaves alone while it works on the others. At the end it reports whether the questions it worked on went green more than the control did. This page explains how the control is built, when the comparison is allowed to speak, and what each verdict says. ## The report in one line A mission's holdout report is one sentence. It appears in three places with the same words: on the mission's receipt, as a row on the scorecard, and as a message in the mission's thread at each look. > Questions we worked on went green 18 points more than 20 questions we held back at random at the +30 check (worked +18, held back +0; p\<0.001, randomized order, n=3 samples a day). Every number in it is in **points of green rate**: the share of answers in which you are named and linked, on a 0 to 100 scale. `worked` and `held back` are each group's own change, so a difference is never mistaken for a direction. Worked questions that fell less than their control did not rise, and the sentence shows both numbers so you can see that. `n` is how many times a day each question is read on each engine. It depends on your plan: 1 on Starter, 3 on Growth and Business. A comparison read three times a day can see a smaller difference than one read once a day. ## The two designs When a mission starts, it decides which of its questions are worked and which are the control, writes that split down with the seed that produced it, and never changes it. Anyone with the seed and the stored inputs can replay the split and check that it was a draw, not a choice made after seeing the results. ### Randomized order The default, for a mission with 8 or more questions. A seeded random draw puts the questions into waves (at most 4, each at least 4 questions). The first wave is worked straight away. The later waves are the control until their turn comes, 30 days apart, so **nothing is withheld for good**: every question is worked, only the order is random. Because the order was drawn at random, a difference between the waves worked first and the waves still waiting is evidence the work made the difference. This is the only design whose report sets `causal: true`. ### Matched, not randomized The fallback for a small mission. Every question is worked, and for each one the mission picks up to 2 untouched questions from your workspace that look like it: the same engines, topic and stage, and a baseline green rate within 15 points. Those similar questions are the control for 30 days. A matched comparison is honest about what it is. Its sentence opens with "Compared with" and ends by saying the comparison does not show cause, because the controls were chosen, not drawn: > Compared with 20 similar questions we left alone, questions we worked on went green 18 points more at the +30 check (worked +18, similar +0; p\<0.001, matched, not randomized, n=3 samples a day). The controls were matched, not drawn at random, so this comparison does not show cause. The scorecard row reads `Holdout: +18 points vs similar questions, not proof of cause`. ### No control available If fewer than 4 usable control questions exist, the mission says so once and shows before and after only. It is a label, not a step for you to take: > No control available: fewer than 4 questions like these were left untouched, so this mission is shown before and after only, not against a control. A control question whose answers start citing a page one of the mission's changes touched is dropped from the comparison, because the work has reached it. If that leaves fewer than 4, the comparison has no control. ## The power floor Before it estimates anything, the comparison checks whether it could have seen a difference at all. Each mission is sized to see a **10-point** difference with an **80%** chance. The size depends on how many questions are in each group, how many engines read them, and `n`. A comparison too small to see 10 points does not report "no difference". Silence from a test that could not have heard anything is not evidence, so it says so and names the smallest difference it could have seen: > Not enough measurement to compare: with 3 worked and 4 control questions (n=1 sample a day), the smallest difference this mission could see is 23 points and it needs to see 10, so a quiet result would not mean nothing happened. More questions, more engines or a plan with more reads a day lower that floor. ## The controls must have tracked before the work The comparison also checks that the control was a fair comparison to begin with. It looks at the 28 days before the work: if the control questions were already drifting apart from the worked ones (p below 0.05), no estimate is given, whatever happened afterwards: > No estimate: in the 28 days before the work, the 20 control questions already drifted 20 points apart from the 20 worked ones, so they are not a fair comparison (n=3 samples a day). ## The two looks A mission is looked at twice, and only twice, at fixed times set when it starts: | Look | When | Speaks when the difference is unlikely by chance at | | - | - | - | | +14 check | 14 days after the work started | p below 0.005 | | +30 check | 30 days after the work started | p below 0.048 | The +14 check only speaks for an overwhelming difference. If it does, the mission's answer is in and the +30 check is not needed to decide it. Otherwise the report waits for the +30 check, which keeps nearly all of a 5% error budget. Splitting the budget this way means looking early does not make a false "moved" more likely: together the two looks are wrong at most about 5% of the time. Each look compares the 14 days before the work with the 14 days up to the look. The p-value comes from relabeling questions between the two groups many times and asking how often a split that size appears by chance, not from a formula. Each question counts once however many engines read it. Before the +14 check the report says: > Too early: the holdout comparison is first read at the +14 check on 2026-10-15. ## Every verdict | `status` | What it means | Scorecard row | | - | - | - | | `pending` | The +14 check is not due yet. | Too early: holdout comparison pending. | | `moved_vs_control` | The worked questions changed more (or less) than the control by more than the noise allows at this look. | Holdout: +18 points vs questions held back at random (…) | | `no_difference` | A comparison able to see 10 points saw nothing it can prove. The sentence names the smallest difference it could see. | Holdout: no difference we can prove (…) | | `too_few_questions` | The comparison could not see 10 points, so its silence means nothing either way. | Holdout: not enough measurement to compare (…) | | `pre_trend_failed` | The control did not track the worked questions before the work. | Holdout: no estimate, controls did not track before the work (…) | | `no_control` | Fewer than 4 usable controls. | No control available: before and after only. | No verdict, in either design, says the work "caused" an outcome or that anything happened "because of" it, and none puts a money figure on it. We measure answers, not revenue. ## Reading it over MCP and the API Once it is live, the report is the same object on every surface: over MCP as `analyze` with `view: "mission_holdout"`, and over REST as `GET /v1/missions/{id}/causal`. ```json theme={null} { "mission_id": "m_123", "design": "randomized_staggered", "causal": true, "status": "moved_vs_control", "look": "final", "method": "randomized order", "samples_per_day": 3, "headline": "Questions we worked on went green 18 points more than …", "receipt": "…", "scorecard": "Holdout: +18 points vs questions held back at random (p<0.001, n=3 samples a day).", "thread": "Holdout comparison, +30 check\n…", "control": "Randomized: 40 questions drawn into 2 waves. …", "looks": [ { "look": "interim", "verdict": "no_difference", "estimate": 0.4, "p_value": 0.81, "mde": 6.1, "…": "…" } ], "next_look_at": null } ``` `design` is `randomized_staggered`, `matched_observational` or `none`. `looks` holds each look read so far, interim first, with its estimate, p-value, the smallest detectable difference (`mde`), the pre-work drift and the group sizes, all in points. `next_look_at` is the next look still to come, or `null` once the answer is in. A mission with no holdout returns a `404` with the standard error envelope. In the dashboard and in the docs a "question" is what the API calls a `prompt`. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.