Signal vs. Noise: Why a Beautiful Backtest Can Be Statistically Meaningless
Financial data is generous to anyone who searches it long enough. Take a few decades of daily prices for a few thousand securities, form a hypothesis, test it, and the odds are good that nothing turns up. Form a hundred hypotheses and something will. Form ten thousand, or let a machine form them, and the best of the results will look like the discovery of a career: a smooth equity curve, a comfortable Sharpe ratio, a t-statistic well past the conventional threshold of two. None of that is evidence that the pattern is real. It is what the best of ten thousand random results looks like, and the difference between a genuine discovery and the luckiest of many false ones is not visible in the backtest itself.
This piece is about that difference: where it comes from, how large it is, and what a research process has to do to keep it in view. The statistics are not new. They belong to a literature on multiple testing that predates quantitative finance by decades and has been applied to it with some force over the past ten years.[1] What is specific to markets is the scale of the search and the weakness of the signals being searched for. Both make the problem worse, and both are growing.
The Best of Many Is Not the Best
Start with the simplest version of the problem. Suppose a researcher generates N candidate strategies that have, in truth, no predictive power at all: their expected excess return is zero, and their backtest Sharpe ratios are pure noise. Each estimated Sharpe ratio is then a draw from a distribution centered on zero, with a standard error that shrinks with the length of the history, roughly one divided by the square root of the number of years.[2] With ten years of data that standard error is on the order of 0.3, and any single one of these strategies will, more often than not, look unremarkable.
The researcher, though, does not look at any single one. The researcher looks at the best. The expected value of the largest of N draws grows with N, quickly at first and then, on a logarithmic scale, almost linearly. Figure 1 plots the expected maximum backtest Sharpe ratio across N random strategies for three lengths of history, using a standard approximation to the expected maximum of independent normal variables. The curves say something uncomfortable. With five years of data and a thousand candidates, the best strategy in a set that contains no skill whatsoever is expected to show a Sharpe ratio in the neighborhood of 1.5; with ten thousand candidates, closer to 1.7. In this model those figures are produced by nothing at all.
Note: Each candidate has a true Sharpe ratio of zero and an estimated annualized Sharpe ratio with standard error 1/√T for T years of history. The expected maximum of N independent standard normal draws is approximated as (1 − γ)·Φ⁻¹(1 − 1/N) + γ·Φ⁻¹(1 − 1/(N·e)), with γ ≈ 0.5772 the Euler-Mascheroni constant, and scaled by the standard error. Parameters are chosen for exposition.
Sources: Oak St. research. Illustrative, stylized simulation prepared for exposition; not derived from any Oak St. portfolio, strategy, or live data.
Two features of the picture matter. The first is that more history helps, but only as the square root: doubling the data shrinks the expected spurious maximum by about thirty percent, whereas multiplying the number of trials by ten adds a roughly constant increment. A research process that can run ten times as many backtests each year outpaces the accumulation of data with ease, and computing power has been growing faster than the historical record for as long as either has been measured. The second is that N is rarely known. A researcher who reports one backtest has usually run many more: variations of the lookback window, the universe, the rebalancing frequency, the treatment of outliers. Each variation is a trial, and the one that is reported is the maximum over all of them, whether or not anyone kept count.
Raising the Bar
The classical response is to raise the threshold a result has to clear before it counts as a discovery. If a single test is judged at a five percent significance level, a t-statistic of about 2.0 clears it. If N tests are run and any one of them is allowed to count, the threshold must rise so that the chance of at least one false positive stays at five percent. The simplest such correction, the Bonferroni correction, divides the significance level by N. Figure 2 shows the resulting t-statistic hurdle as a function of the number of tests, for two significance levels, together with a stylized version of what happens when the tests are correlated with one another.
Note: Solid curves: the Bonferroni hurdle t* = Φ⁻¹(1 − α/(2N)) for family-wise two-sided levels α of 5% and 1%. Dashed curve: the 5% hurdle when N correlated trials are treated as √N effective independent tests, a stylized stand-in for the dependence among variations of one idea. The reference line is the single-test 5% hurdle of 1.96.
Sources: Oak St. research. Illustrative, stylized simulation prepared for exposition; not derived from any Oak St. portfolio, strategy, or live data.
The hurdle rises quickly and then slowly: from about 2.0 for one test to about 2.8 for ten, 3.5 for a hundred, and just above 4.0 for a thousand. This is the arithmetic behind the argument, made forcefully in the academic literature, that a newly proposed return predictor should be held to a t-statistic of around 3.0 rather than 2.0, on the grounds that hundreds of such predictors have already been tested and published.[1] The dashed curve makes a second point. Correlated tests are not independent trials; if a thousand variations of the same idea share most of their information, they behave more like a few dozen independent tests, and the appropriate hurdle is lower. The correlation among trials is therefore not a nuisance but a parameter, and estimating it is part of the job.
Two refinements matter in practice. The first is that controlling the chance of any false positive, which is what Bonferroni does, is often too strict for a research program that expects to find many small effects; a procedure that controls the expected share of false discoveries among the results declared significant is usually the better target, and the statistical literature provides one.[3] The second is that the correction can be applied to the Sharpe ratio directly, by asking what the best backtest Sharpe ratio would have been under the null hypothesis given the number of trials and the variance among them, and deflating the observed figure accordingly.[2] Under either approach the conclusion is the same: the number of trials is a required input to the evaluation of any result, and a research process that does not record it cannot evaluate its own output.
Base Rates and the Arithmetic of False Discoveries
There is a second way to look at the same problem that we find more useful in daily work, because it makes the role of prior belief explicit. Suppose that of all the hypotheses a research group tests, some fraction are true: the signal exists, at some economically meaningful size. Call that fraction the base rate. A test with a given significance level and a given power, the probability of detecting a true signal when one is there, produces a mix of true and false positives, and the share of declared discoveries that are false depends on the base rate at least as much as on the threshold.
Figure 3 works through the arithmetic for five base rates and two testing regimes: a conventional five percent test with a fifty percent chance of detecting a true signal, and a stricter hurdle near a t-statistic of three, which detects fewer true signals but admits far fewer false ones. When one candidate in a hundred is real, about nine in ten of the discoveries declared by the conventional test are false. Raising the hurdle helps, but even then roughly half of what passes is noise. Only as the base rate rises toward one in four does the conventional test become tolerable, with something like a quarter of its discoveries false, and by then the stricter test is nearly clean.
Note: Share of declared discoveries that are false = α(1 − π) / (α(1 − π) + (1 − β)π), where π is the base rate of true hypotheses, α the significance level, and 1 − β the power. Conventional test: α = 5%, power 50%. Stricter test: α = 0.27% (a two-sided t-statistic of 3.0), power 25%. The power values are assumptions for exposition.
Sources: Oak St. research. Illustrative, stylized simulation prepared for exposition; not derived from any Oak St. portfolio, strategy, or live data.
The implication is that the quality of a research pipeline is set upstream of any statistical test, by the quality of the hypotheses fed into it. A group that generates candidates by exhaustive search over transformations of price has a base rate near zero and will drown in false discoveries however carefully it corrects for multiple testing. A group that begins from an economic reason for a pattern to exist, a risk that someone is paid to bear, a constraint that forces someone to trade, a piece of information that is genuinely costly to process, starts from a higher base rate and can afford a lower hurdle. Statistical discipline and economic reasoning are not alternatives. The second is what makes the first affordable.
The Many Ways Selection Creeps In
Multiple testing is the cleanest form of the problem but not the only one. Most of the ways a backtest comes to overstate a signal are forms of selection: the researcher, the data, or the field as a whole has already chosen, before the test begins, in a way that favors a positive result. Figure 4 lists the forms we watch for most closely, how each enters, what it tends to look like in a result, and the countermeasure. None of the countermeasures is exotic. Their difficulty is organizational rather than technical: each requires someone to have made a decision before seeing the result, and to hold to it afterward.
| Bias | How it enters | What it looks like | Countermeasure |
|---|---|---|---|
| Look-ahead | Fields stored at their current value rather than the value known at the time: restated financials, later index membership, corrected prices | Predictive power that is strongest exactly where the data was revised | Point-in-time databases; every field date-stamped by when it became known, not when it refers to |
| Survivorship | A universe assembled from the securities that exist today | Signals that correlate with having survived look predictive; drawdowns are understated | Include delisted securities; reconstruct the universe as it stood on each date |
| Multiple testing | Many variants tried, the best one reported | A single impressive result whose siblings were never mentioned | Log every trial, including failures; raise the hurdle with the count; report the count with the result |
| Parameter overfitting | Parameters tuned on the same data used to judge them | A sharp peak in parameter space; performance collapses when a parameter moves slightly | Prefer broad plateaus to peaks; hold out by time; nested validation for tuned models |
| Sample-period selection | Start and end dates chosen after the fact | Results that depend on where the window begins | Test across rolling windows; report sensitivity to the start date |
| Field-wide data snooping | The same public datasets searched by thousands of researchers over decades | A published anomaly that is one of hundreds | Treat published effects as already-tested; discount by the size of the literature that produced them |
| Hold-out contamination | The reserved period consulted more than once while a model is revised | An out-of-sample result that was, in fact, seen during development | Touch the hold-out once; write down in advance what result would count as success |
| Reporting bias | Only successes are written up; failures leave no record | A research history that reads as a string of wins | Keep the failures on file and count them; they are the denominator |
Note: Qualitative summary of the selection mechanisms discussed in the text. The list is not exhaustive, and several of the mechanisms compound one another.
Sources: Oak St. research. Illustrative, stylized simulation prepared for exposition; not derived from any Oak St. portfolio, strategy, or live data.
Several of these deserve comment. Look-ahead bias is the hardest to see because it is usually invisible in code: a database that stores the current value of a field rather than the value that was known at the time contaminates every test run against it, and the contamination looks exactly like alpha. Survivorship bias has a similar character. A universe assembled from today's constituents is a universe of winners, and any signal that correlates with having survived will appear predictive. Both are addressed by data engineering rather than statistics, which is one reason the two disciplines are inseparable in this business.
Hold-out contamination is subtler. The purpose of reserving a period of data is to obtain one clean test of a hypothesis that was formed without it. That test can be run once. The moment a researcher looks at the hold-out result, adjusts the model, and looks again, the hold-out has become in-sample data, and the second look is not a test but another trial. The discipline is to decide, in writing and in advance, what result on the reserved data would count as success, and then to accept the answer.
What This Means for How We Work
Several practices follow. We treat the count of trials as data in its own right: every backtest that is run is logged, including the ones that fail, and the number of variations tried is reported alongside any result so that the hurdle it faces can be set accordingly. We prefer results that are robust to their own parameters, a broad plateau rather than a sharp peak, since a sharp peak is what a maximum over noise looks like. We weight results by the economic reasoning behind them, and we are more skeptical of a strong result with no story than of a modest result with a good one. And we reserve data by time, consult it once, and record what we said we expected before we did.
None of this makes the search for signals less necessary. It makes the search honest about its own output. A research process that runs ten thousand trials and reports the best one, without correction, is not measuring markets. It is measuring its own capacity to generate noise, and its results will look better every year as that capacity grows. The alternative is slower, less flattering, and the only kind of research whose findings tend to survive contact with the future.
- [1]Harvey, Liu and Zhu (2016), “… and the Cross-Section of Expected Returns,” Review of Financial Studies 29(1), catalog several hundred published return predictors and argue that, given the extent of the search that produced them, a newly proposed predictor should clear a t-statistic of roughly 3.0. We cite the argument as literature; nothing in this piece reproduces their data.
- [2]The sampling distribution of an estimated Sharpe ratio is set out in Lo (2002), “The Statistics of Sharpe Ratios,” Financial Analysts Journal 58(4); the 1/√T standard error used in Figure 1 is its leading term under the null of zero skill. The approximation to the expected maximum of N normal draws, and the deflation procedure described in the text, are from Bailey and López de Prado (2014), “The Deflated Sharpe Ratio,” Journal of Portfolio Management 40(5). The related probability of backtest overfitting is developed in Bailey, Borwein, López de Prado and Zhu (2014), “Pseudo-Mathematics and Financial Charlatanism,” Notices of the American Mathematical Society 61(5).
- [3]Benjamini and Hochberg (1995), “Controlling the False Discovery Rate,” Journal of the Royal Statistical Society, Series B 57(1). The base-rate arithmetic of Figure 3 follows the framing in Ioannidis (2005), “Why Most Published Research Findings Are False,” PLoS Medicine 2(8), which was written about biomedical research and applies to backtests without modification.
Interested in related insights?
The Backtest That Never Happened: How Historical Simulations Learn What No One Could Have Known
The Half-Life of Alpha: Is Predictive Power Disappearing, or Just Becoming Shorter-Lived?
Enjoyed this piece?
This document is provided for informational purposes only and does not constitute investment advice or an offer to sell (or the solicitation of an offer to buy) any security, investment product, or service.
The views expressed are those of OAK ST LLC as of the date of the document, are subject to change without notice, and may not reflect the criteria used by OAK ST LLC to evaluate investments. Figures described as illustrative, stylized, or simulated are hypothetical constructions prepared for exposition; they do not depict the results of any OAK ST LLC strategy, portfolio, or account, and no representation is made that any account will or is likely to achieve results similar to those shown. Historical market trends are not reliable indicators of future market behavior.
Information obtained from third-party sources is believed to be reliable but has not been independently verified, and OAK ST LLC does not guarantee its accuracy or completeness. Nothing in this document is a recommendation to buy, sell, or hold any instrument.
This document may not be reproduced or distributed without the prior written authorization of OAK ST LLC. The Terms of Use and the Important Legal and Regulatory Disclosures govern its use. Copyright © 2026 OAK ST LLC. All rights reserved.