Eight checks, run in a fixed order every time, each one aimed at a specific way a backtest lies. The bar each check applies is locked in before the run starts and cannot be moved afterwards.
Vetoes come first
The first two checks are vetoes. If either one fails, the run stops and the rest of the scorecard is meaningless — a result built on information the strategy would not have had live is not worth grading. The remaining six are the graded profile: each is scored against its own floor, and a strategy has to clear all of them to be Cleared. Most of the graded checks need a minimum number of trades to say anything; a run that produces almost none is reported as Not enough trades to grade rather than forced into a pass or a fail.
The order is “cheapest kill first”
The stack runs the cheapest test that could invalidate a claim before the expensive one. The leak audit and the null-information check need no model and little compute, so they run early; the 200-draw placebo test runs last. If an early check is decisive, the later ones still run so you see the full picture, but the attribution is keyed to the earliest failure that mattered.
Leak audit
VetoWhat it checksRecomputes every indicator and the entry decision from scratch, using only data that existed at each decision's own timestamp, then checks it against what the full-series run produced. Catches lookahead: an indicator reading a future bar, a cross-timeframe filter seeing a still-forming higher-timeframe candle, an off-by-one in a previous-period join.
Why it mattersThis is the single most consequential check in the stack. On one of our own research strategies, fixing two leaks exactly like this took its profit factor from 3.4 down to 1.17 — the leaky and clean versions agreed on which setups to take only 16-18% of the time. The leak was the apparent edge.
Twin control
VetoWhat it checksVerifies the leak audit is actually working. Before your real strategy runs, a deliberately broken twin of it is generated — the same strategy with an obvious lookahead bug planted in — and run through the same leak audit.
Why it mattersAn unverified audit is a marketing claim, not a mechanism. If the leak audit can't catch its own known-bad copy, it's not trustworthy on your real strategy either, so the run halts rather than showing you a scorecard from a checker that might be silently broken.
Null info
What it checksCompares your strategy's actual hit rate against the pure-chance baseline implied by its own stop/target ratio — the win rate a coin flip would produce at that same risk:reward.
Why it mattersIf entries don't beat that baseline, no amount of parameter tuning or model sophistication downstream can create an edge that isn't there — the weakness is upstream of anything a model can fix. It's the cheapest, sharpest kill available: no model required, just your entries and your geometry.
Pre-registration
What it checksChecks your raw result against floors — minimum trade count and significance — that were locked in before the run: either the platform's defaults, or ones you explicitly committed to.
Why it mattersThis is what stops "test until it passes." Without it, you could retune a strategy across many attempts and only report the one that happened to clear the bar. Every run is recorded permanently against your account and your significance is adjusted for how many trials you've already spent; floors can't be changed after the fact.
Cost realism
What it checksRe-runs your result net of realistic transaction costs — spread on entry and exit, commission, and overnight swap or financing accrued per bar held — and shows the gross and net numbers side by side.
Why it mattersA strategy can look profitable gross and be negative net once real costs land. One of our own research strategies lost 47R of its 61R total drawdown to swap financing alone; ignoring it would have made a losing strategy look roughly break-even.
Cluster inference
Pro & ResearchWhat it checksRecomputes significance treating trades that share a day (or the same market event) as correlated rather than independent, reported alongside the naive per-trade t-stat.
Why it mattersA naive per-trade t-stat overstates confidence when a handful of cascading, correlated days are doing most of the work — news-day reactions and regime transitions are the common cause. The cluster-adjusted number is the one that actually counts.
Temporal halves
What it checksSplits your sample at the median trade and requires the edge to hold up independently in both halves, not just in the full-sample average.
Why it mattersThis is the single most valuable check for a retail trader. Full-sample statistics sell the past — a strategy that worked for the first half of your data and has been flat or losing since is not tradeable today, even if the blended number still looks good.
Placebo
Pro & ResearchWhat it checksRuns 200 random permutations of your own signal timing over the same data and checks whether your actual result sits meaningfully outside that random distribution.
Why it mattersCatches a strategy that only looks good because it happens to line up with the market's own structural drift — a persistent trend, for example — rather than because its own logic contributes anything. The question this asks: would random signal on this same data look about this good too?
Optional: the economic-significance floor
Every one of the eight checks is a test of statistical significance, not of magnitude. A t-statistic is the effect size times the square root of the sample, so a genuinely tiny edge, given enough trades, clears every significance bar in the stack — and still might not be worth trading, or might lose to simply holding the asset. The fixed stack has no answer to “is it big enough,” by design: folding sample size and effect size into one number is what lets it reject noise without a magnitude floor.
If you want one, commit a minimum net-of-cost profit factor in your validation criteria, the same place you set the trade-count and significance floors. When a run carries one, a ninth economic significance row appears in the graded profile, checking the run’s net profit factor against the number you chose before it started. Leave it unset — the default — and the scorecard keeps its exact eight-check shape. The equity panel on every result already shows the annualized return and a buy-and-hold comparison whether or not you enforce a floor.
Checks that combine
Some failure shapes are more informative together than apart:
- Leak audit clean, temporal halves fail. The edge was real and has decayed — a regime that existed and ended, not a backtest artifact.
- Null-information fail and cost realism fail. There is no entry signal and what little there is gets eaten by costs. No model or cost reduction recovers this — a worked example shows a strategy that clears seven checks at the midpoint and hits exactly this shape once the fill is priced honestly.
- Everything passes, but one symbol or one day is most of the profit. The apparent edge is a concentration, not something distributed across the population.
How these become the text on your scorecard is covered in the scorecard & verdicts. The null-information check is also available on its own, with no strategy required, as the Null-info diagnostic.