# How to Design Experiments: 80% Power to Prevent False Kills in Multi-Pilot Portfolios

Ivy Nakamura · October 26, 2026

> Prevent false kills in multi-pilot portfolios by pre-registering each pilot's minimum detectable effect and sizing tests for about 80% power.

| Takeaway | Detail |
| --- | --- |
| False kills are a portfolio-design problem, not a statistics problem | Prevent them by pre-registering a minimum detectable effect for each pilot before it enters the portfolio. |
| Size each pilot's test window for roughly 80% power | The primary metric must have about 80% power against its pre-registered MDE. |
| Use a one-line power contract before launch | This pilot scales only if the primary metric beats the break-even threshold with at least 80% power against an MDE of X over the sized window. |
| The break-even threshold sets the scaling gate | The contract's threshold is derived from break-even rather than from an arbitrary target. |

This guide shows how to size multi-pilot tests around a pre-registered minimum detectable effect.

It gives a one-line power contract for deciding when a pilot scales.

![How to Design Experiments](https://static.mm-ais.com/article-images-ai/how-to-design-experiments-80-power-to-pr-ai-e35dba44.jpg)

## Why false kills happen in portfolios

A false kill in a multi-pilot portfolio is not proof that an idea failed; it is often proof that one small test was asked to carry a decision it could not support. The portfolio-level mechanism is simple: an underpowered single pilot can stop an idea that would have cleared the bar once its evidence is combined with the rest of the portfolio. The pilot is judged as if it were the whole experiment, even though the portfolio is the actual unit of risk.

Check the mechanism by re-auditing last year's stopped pilots against their original minimum detectable effects. A pilot stopped at low power has a high chance that the true effect exists but the test was too small to see it. If the original MDE is missing, treat that stop as unverifiable rather than as evidence of failure. The audit is not about rescuing every stopped idea; it is about separating "no effect" from "not enough evidence."

Then check the portfolio effect. Count how many pilots ran at the same time, and tally kills against scale decisions from your last two quarters. If many pilots were underpowered, the portfolio systematically over-produces kills: each test has a high chance of missing a real effect, and those misses accumulate into a portfolio that looks disciplined but is actually discarding viable options. The pattern matters more than any single stop.

For each stopped pilot, ask one question: was the MDE pre-registered, or was it chosen after the result? If the threshold moved after the data arrived, the kill may reflect a changed standard rather than a failed idea. Also ask whether the primary metric, sample, and test window matched the design that was approved before launch. A stop is credible only when the rule existed before the evidence did.

Not every stopped pilot is a false kill, and this section is not a license to keep weak ideas alive. The point is that pilot-level power and portfolio-level decisions are different objects. When a portfolio kills ideas one underpowered pilot at a time, it can mistake noise for discipline. The correction belongs in the design phase: define the effect that would make scaling worthwhile, size the window to see it, and record the rule before launch.

Make the check routine. Keep a stop log with the pilot, primary metric, pre-registered MDE, intended power, actual sample, and final decision. Review it quarterly and look for clusters of stops with small samples or missing MDEs. That log turns false kills from an invisible portfolio habit into a visible design defect.

![Why false kills happen in portfolios — How to Design Experiments](https://static.mm-ais.com/article-images-ai/how-to-design-experiments-80-power-to-pr-ai-a10d42ac.jpg)

## Derive the MDE from break-even

The break-even-derived MDE is the minimum effect that makes a pilot worth scaling, not the effect you hope to see. Compute the break-even effect first: the smallest improvement in the primary metric that covers the pilot's incremental cost. Then set the MDE at or slightly above that break-even level, never at an aspiration. This keeps the test anchored to economics rather than to enthusiasm.

When computing break-even, include only incremental cost attributable to the pilot, and value the primary metric in the same units the scaling decision will use. A retention pilot, for example, should value retained users or revenue in the same accounting frame that would justify rollout; mixing a cost in dollars with a metric in percentage points without a conversion leaves the MDE undefined.

To convert that MDE into a required sample, use a standard power calculation with alpha set at 0.05, power at 0.80, and sidedness declared in advance. The inputs are the MDE, the baseline rate, and the variance; a free calculator or stats library can produce the required sample or test window. The result is a data requirement tied to the economic threshold, not a guess about likely performance.

If the required sample exceeds what the pilot can deliver, do not shrink the MDE to fit. Redesign the pilot—its metric, population, channel, or cost structure—or drop it. Shrinking the threshold to match available data would make the pilot look testable while leaving it unable to support the scaling decision.

Declare one-sided or two-sided testing before the pilot starts, because that choice changes the required sample. Base the baseline rate and variance on the population and period the pilot will actually use, not on a best-case segment. A mismatched baseline can make a break-even MDE appear either trivial or impossible to detect.

Treat the MDE as a gate rather than a forecast: it defines the smallest result that justifies scaling, and it should be written before the pilot enters the portfolio. If the pilot cannot be sized to detect that effect with roughly 80% power, the problem is in the pilot design, not in the statistics.

![Derive the MDE from break-even — How to Design Experiments](https://static.mm-ais.com/article-images-pixabay/how-to-design-experiments-80-power-to-pr-b91629ae.jpg)

## Options compared: sizing approaches

Sizing is a portfolio-cadence decision before it is a statistical one. The default should be the approach that keeps every pilot on the same review calendar, because comparability across pilots matters more than squeezing out marginal precision from any single test.

Fixed-window sizing wins for portfolios: run the pilot for twelve weeks and make the decision at week twelve regardless of interim noise. The calendar is deterministic, every pilot is judged on the same horizon, and cross-pilot ranking stays meaningful. One four-week extension is allowed, but only under a pre-set rule written before launch, not as a reaction to a disappointing week-eleven read.

| Approach | Calendar behavior | Portfolio fit | Required guardrail |
| --- | --- | --- | --- |
| Fixed-window (12 weeks) | Decision at week 12 for every pilot | Best: shared review date | One 4-week extension only by pre-set rule |
| Fixed-sample (until N) | Ends when observations arrive | Poor: slow pilots drift past review | Use outside portfolio cadence |
| Sequential / peeking | Weekly checks and possible early stop | Risky: highest false-kill risk | Pre-committed spending function (e.g., O'Brien-Fleming); weekly decision only |

Fixed-sample sizing is statistically clean but operationally awkward in a portfolio. A pilot that needs more time to reach its observation target can drift past the shared review date, which breaks cadence and makes one pilot's result hard to compare with another's. Reserve it for pilots that are not competing for the same decision slot.

Sequential or peeking designs carry the highest false-kill risk unless you pre-commit to a spending function such as O'Brien-Fleming. They are justified only when a weekly decision is genuinely required; otherwise the temptation to stop early turns noise into a kill. If you use one, the spending rule must be fixed before the first observation, not chosen after a promising week.

Default to fixed-window, allow one extension only by its pre-set rule, and treat sequential designs as an exception with a pre-committed spending function. That keeps the portfolio's calendar intact while leaving room for a pilot that needs slightly more time.

![Options compared: sizing approaches — How to Design Experiments](https://static.mm-ais.com/article-images-pixabay/how-to-design-experiments-80-power-to-pr-c06e0035.jpg)

## The power contract template

Before a pilot enters the portfolio, copy this contract into the pre-registration log and complete every blank. A pilot with any blank empty is not a pilot; it is an uncontrolled spend. The contract is the only document that can authorize a scale decision.

For this illustration, the winner is "scale" only when two conditions hold together: the primary metric beats the break-even threshold, and the test has at least 80% power against the MDE over the sized window. The break-even trigger is therefore the threshold derived from break-even, not a hoped-for lift. If the metric clears the threshold but the power check fails, the result is not a clean win; it is an underpowered read that should not authorize scaling.

Keep the MDE, threshold, power level, and window together in one line so the decision can be audited later. The arithmetic is stable; the inputs are not, so replace the illustration with the pilot's own cost, variance, and window before anyone acts on it.

![The power contract template — How to Design Experiments](https://static.mm-ais.com/article-images-pixabay/how-to-design-experiments-80-power-to-pr-c2ebd53a.jpg)

## What 80% power does not buy you

In every contract, state plainly that 80% power is a design target derived from your own MDE math, not a published statistic about innovation portfolios. The honesty limits are three: 80% is a design target, the power math assumes the registered metric and no interference between units, and it does not cover wrong-metric risk. That sentence keeps the number in its proper role as a planning assumption rather than a certificate of validity.

The power statement is tied to the metric named in the contract. If the pilot later reports a different primary metric, changes the measurement window, or redefines the denominator, the 80% claim no longer describes the test that was run; it describes an earlier design. Before reading any result, check that the reported metric, window, and population match the registered ones exactly.

Power also does not remove wrong-metric risk. A pilot can be well powered against an MDE for a metric that turns out not to predict scaling value, so a statistically clear result on the registered metric is not automatically a clear result on the business question. The contract protects detectability, not metric choice.

The rule breaks when pilots share networks or resources, because one unit's behavior changes another's and violates independence. In that case, unit-level power math is not merely optimistic; it is the wrong model. Use cluster-level randomization or mark the power math invalid, and do not carry the 80% claim into the scale decision.

Even with metric-drift risk, a pre-registered MDE beats a post-hoc threshold because the decision rule is fixed before results arrive. Drift then appears as a deviation from the contract, which can be audited, rather than as a reason to move the goalposts after seeing the data.

| Assumption in the 80% claim | Failure signal | Action |
| --- | --- | --- |
| Registered primary metric unchanged | Reported metric differs from contract | Suspend the power claim and re-register before deciding |
| No interference between units | Shared network or resource changes another unit's behavior | Use cluster-level randomization or mark the power math invalid |
| Metric choice remains valid | Registered metric no longer predicts scaling value | Treat the result as detectability evidence only and revisit metric choice |

Write the contract as a conditional: if the registered metric, independence assumption, and measurement plan hold, the pilot has roughly 80% power against the stated MDE; if any of them fail, suspend the power claim until the design is repaired. That keeps a failed assumption from being mistaken for a failed idea.

## What to do next

| Step | Action | Why it matters |
| --- | --- | --- |
| 1 | Pre-register a minimum detectable effect for each pilot before it enters the portfolio. | False kills are a portfolio-design problem, so the MDE must be fixed before portfolio entry rather than chosen after results. |
| 2 | Write the one-line power contract before launch: "This pilot scales only if [primary metric] beats [threshold derived from break-even] with ≥80% power against an MDE of [X] over [N weeks / N units]." | The contract is the canonical scaling gate and must state all four blanks. |
| 3 | Derive the contract's threshold from break-even rather than from an arbitrary target. | The break-even threshold sets the scaling gate. |
| 4 | Size each pilot's test window around the prominent 12-week figure so the primary metric has about the prominent 80% figure power against its pre-registered MDE. | The primary metric must have about 80% power against its pre-registered MDE. |
| 5 | Apply the contract at the scaling decision: the pilot scales only if the primary metric beats the break-even threshold with at least 80% power over the sized window. | This prevents false kills in multi-pilot portfolios. |
| 6 | Reject any pilot whose contract cannot state all four blanks. | Only pilots with a complete power contract should enter the portfolio. |

## Frequently Asked Questions

**What is a false kill in a multi-pilot portfolio?**

It is not proof that an idea failed; it is often proof that one small test was asked to carry a decision it could not support.

**Why can an underpowered single pilot stop an idea?**

It can stop an idea that would have cleared the bar once its evidence is combined with the rest of the portfolio.

**What is the actual unit of risk when judging a pilot?**

The portfolio is the actual unit of risk, even though the pilot is judged as if it were the whole experiment.

**What must be pre-registered before a pilot enters the portfolio?**

A minimum detectable effect for each pilot.

**What sets the scaling gate in the power contract?**

The break-even threshold.

**How is the contract's threshold derived?**

From break-even rather than from an arbitrary target.

## Quick answers

| What kind of problem are false kills in a multi-pilot portfolio? | False kills are a portfolio-design problem, not a statistics problem. |
| --- | --- |
| How can false kills be prevented before a pilot enters the portfolio? | Prevent them by pre-registering a minimum detectable effect for each pilot before it enters the portfolio. |
| What sets the scaling gate in the contract? | The break-even threshold sets the scaling gate. |
| What is the contract's threshold derived from? | The contract's threshold is derived from break-even rather than from an arbitrary target. |
| What does this guide show? | This guide shows how to size multi-pilot tests around a pre-registered minimum detectable effect. |

Also worth reading: **3 Pre-Launch Pricing Methods: Evidence and Anchor Selection**: [3 Pre-Launch Pricing Methods: Evidence](https://tlab.fun/blog/3-pre-launch-pricing-methods-evidence-and-anchor-selection.php) · **Disney Is Buying Back $8 Billion in Stock as Employees Face Layoffs and Stricter Office Rules**: [Disney Is Buying Back $8](https://tlab.fun/blog/disney-is-buying-back-8-billion-in-stock-as-employees-face-layoffs-and-stricter-office-rules.php)

### Related reading

- [Kill thresholds in 2026: pre-register week-6 gates for your 4 pilots](https://tlab.fun/blog/kill-thresholds-in-2026-pre-register-week-6-gates-for-your-4-pilots.php)
- [How to scale innovation pilots: 12-week test, pre-set scale-or-stop criteria](https://tlab.fun/blog/how-to-scale-innovation-pilots-12-week-test-pre-set-scale-or-stop-criteria.php)
- [How to Test Targeting in 2026: Location vs Cohort by 90-Day Retention Scale or Kill](https://tlab.fun/blog/how-to-test-targeting-in-2026-location-vs-cohort-by-90-day-retention-scale-or-kill.php)
- [Hardware + Subscription Models: A Verify-Before-You-Commit Guide](https://tlab.fun/blog/hardware-subscription-models-a-verify-before-you-commit-guide.php)
- [How to pick co-brand partners: 40 pilots show 12-month revenue lift criteria](https://tlab.fun/blog/how-to-pick-co-brand-partners-40-pilots-show-12-month-revenue-lift-criteria.php)
- [Vrbo Gen-Z Booking Data: 18% Premium for Fast Host Replies](https://tlab.fun/blog/vrbo-gen-z-booking-data-18-premium-for-fast-host-replies.php)

### Latest

- [Kill thresholds in 2026: pre-register week-6 gates for your 4 pilots](https://tlab.fun/blog/kill-thresholds-in-2026-pre-register-week-6-gates-for-your-4-pilots.php)
- [How to scale innovation pilots: 12-week test, pre-set scale-or-stop criteria](https://tlab.fun/blog/how-to-scale-innovation-pilots-12-week-test-pre-set-scale-or-stop-criteria.php)
- [How to Test Targeting in 2026: Location vs Cohort by 90-Day Retention Scale or...](https://tlab.fun/blog/how-to-test-targeting-in-2026-location-vs-cohort-by-90-day-retention-scale-or-kill.php)

Canonical: https://tlab.fun/blog/how-to-design-experiments-80-power-to-prevent-false-kills-in-multi-pilot-portfolios.php
Markdown: https://tlab.fun/blog/how-to-design-experiments-80-power-to-prevent-false-kills-in-multi-pilot-portfolios.php/index.md