The Cost of Knowing Too Much: Does a Richer Model Predict Better, or Only Fit Better?
A model that fits the past better is not, on that evidence alone, a model that will forecast the future better. In most fields the two move together closely enough that the distinction can be left to the statisticians. In financial markets they come apart early and violently, because the thing being predicted is mostly noise, the history available to learn from is short, and the relationships in it drift. Add parameters to a model of returns and the in-sample fit improves without limit. The out-of-sample forecast improves for a while, then stalls, then gets worse, and the model itself gives no sign of which of those phases it is in.
This piece is about that divergence and what it implies for how a quantitative research process should treat complexity. We start with the simplest possible arithmetic of the gap, decompose it into bias and variance, ask what changing the class of model actually buys, and then turn to the form of complexity that we think matters most and is discussed least: the complexity of the search rather than of the model. The conclusion is not that simple models are better. It is that complexity is a budget, that markets make it an expensive one, and that the discipline of spending it deliberately is most of what separates research from curve fitting.
The Gap
Consider a forecasting model with p free parameters fitted to N observations. There is some quantity of genuine, learnable structure in the data, and a model with more parameters can capture more of it, up to a ceiling set by how much structure exists. But each parameter also gets to fit some of the noise, and noise fitted in-sample is noise that will not recur. For a linear model the two effects have a simple and well-known form: the in-sample explained variance is inflated by roughly p/N, and the out-of-sample explained variance is deflated by roughly the same amount.[1]
Figure 1 draws that arithmetic for a stylized return-forecasting problem in which the attainable explained variance is small, as it is in markets, and the number of observations is a few hundred, as it often is once observations are required to be independent. The in-sample curve rises without limit. The attainable curve rises and saturates. The out-of-sample curve peaks early, at fewer than ten parameters in this construction, and crosses zero not long after: beyond that point the model forecasts worse than a constant would.
Note: Attainable R² = 0.10 · (1 − exp(−p/12)). In-sample = attainable + p/N and out-of-sample = attainable − p/N, with N = 250 independent observations, the leading-order optimism of a linear fit. The parameters are chosen to make the shape legible, not estimated from any data.
Sources: Oak St. research. Illustrative, stylized simulation prepared for exposition; not derived from any Oak St. portfolio, strategy, or live data.
Three things about the picture generalize beyond the toy. First, the peak of the out-of-sample curve arrives long before the in-sample curve shows any sign of slowing, so the fit carries no information about where the peak is. Second, the location of the peak is set by the ratio of learnable structure to sample size, and in markets that ratio is typically small: as in the construction of Figure 1, on the order of a few hundred independent cross-sections, explained variance of a percent or less, and every additional parameter charged against that sample.[2] Third, the gap between the two curves, which is what a researcher who looks only at the fit will mistake for skill, grows linearly in p. The richer the model, the larger the flattering error.
Why Markets Punish Complexity Harder
The standard way to think about the trade-off is the decomposition of prediction error into bias, variance, and noise. Bias is the error from a model too simple to represent the structure that is there; it falls as parameters are added. Variance is the error from a model that has learned the particular sample rather than the process behind it; it rises as parameters are added. Noise is the part of the outcome that no model could have predicted, and it sets a floor beneath which neither term can reach.
Figure 2 draws the two reducible terms for the same stylized problem as Figure 1. The bias term decays with the number of parameters; the variance term grows with it; their sum is the U-shaped curve whose minimum is the model size a researcher should actually want. What makes markets unusual is not the shape of the curves but their scale. The noise floor, omitted from the chart so that the reducible part is legible, sits at around ninety percent of the variance of returns in this construction. The entire contest between bias and variance is played out in the remaining tenth.
Note: Bias² = 0.10 · exp(−p/12), the attainable structure not yet captured; variance = p/250, the share of noise absorbed by p parameters fitted to 250 observations. The irreducible noise, 0.90 of return variance in this construction, is omitted so the reducible part is legible. Same parameters as Figure 1; the minimum coincides with that figure's out-of-sample peak.
Sources: Oak St. research. Illustrative, stylized simulation prepared for exposition; not derived from any Oak St. portfolio, strategy, or live data.
That scale is why techniques that work in image recognition or language modeling do not transfer to return prediction unmodified. In those domains the signal is most of the variance and the sample is effectively unlimited, so the variance term can be pushed to the right for a long time before it bites. In markets the signal is a sliver, the effective sample is small, and it is smaller than it looks, because return observations that are adjacent in time are not independent and the relationships that generate them shift. A model complex enough to fit a decade of history has almost always fitted several regimes that will not repeat, and the variance term charges for each of them.
What a Model Class Buys
A natural response is to say that the trouble lies with the linear model of Figure 1, and that a more expressive class, boosted trees or a neural network, captures nonlinearities and interactions a linear model cannot. Sometimes it does. But the question is not whether the richer class fits better in-sample; it always will. The question is whether the improvement survives out of sample by a margin larger than the uncertainty in the measurement, and that uncertainty is large.
Figure 3 makes the comparison for six stylized model classes fitted to the same hypothetical panel and scored by information coefficient, the correlation between the model's ranking of securities and their subsequent returns. The in-sample IC climbs steeply with expressiveness. The out-of-sample IC is nearly flat, and the dashed lines, which bound a confidence band of the width one would expect from roughly a decade of monthly cross-sections, overlap for every pair of models. On this evidence no ranking among the six can be established, and a second decade would produce a different one.
Note: Out-of-sample ICs are set by construction to 0.028–0.036, with differences deliberately smaller than the band. In-sample ICs add 0.0007 per effective parameter, with effective counts of 10, 20, 40, 60, 90 and 160 for the six classes in order. The band is ±1.96 × 0.10/√120: the sampling error of a mean IC over 120 monthly cross-sections when the monthly IC has a standard deviation of 0.10.
Sources: Oak St. research. Illustrative, stylized simulation prepared for exposition; not derived from any Oak St. portfolio, strategy, or live data.
Two lessons follow. The first is that the honest unit of comparison between model classes is the out-of-sample estimate together with its uncertainty, and that in return forecasting the uncertainty is usually wide enough to swallow the differences. The second is that where a richer class does earn its place, it tends to do so for a reason that could have been stated in advance: an interaction the economics predicts, a nonlinearity that a known constraint produces, a regime dependence with a mechanism behind it. Expressiveness bought to capture structure whose existence was hypothesized beforehand is a different purchase from expressiveness bought in the hope that the data contains something. The first is a model. The second is a search.
The Complexity You Do Not See
Which brings us to the form of complexity we think matters most. The parameter count of the final model is the visible complexity. The invisible complexity is the number of models, features, transformations, lookback windows, and universe definitions that were tried before that one was chosen. Every one of those choices is a degree of freedom, and degrees of freedom exercised in the search overfit exactly as parameters exercised in the fit do. A three-parameter model selected from ten thousand candidates is not a three-parameter model.
The arithmetic is unforgiving. If a candidate signal's measured IC on a given sample has a standard error of some size even when the signal is pure noise, then the best of N such candidates has an expected IC that grows with the square root of the logarithm of N.[3] Figure 4 draws that expectation for standard errors consistent with ten and with five years of monthly observations. With ten candidates the best noise signal already looks respectable. With a thousand, on either sample, it looks better than a genuinely useful signal in the same stylized model. Nothing about the winning candidate reveals which kind it is; that information lives only in the count of things tried, which is the one number a backtest never reports.[4]
Note: Each curve is σ · √(2 ln N), the leading term of the expected maximum of N independent noise draws, with σ = 0.10/√120 and 0.10/√60 for ten and five years of monthly ICs, evaluated at 41 log-spaced values of N from 1 to 10,000. The reference line is the out-of-sample level used in Figure 3.
Sources: Oak St. research. Illustrative, stylized simulation prepared for exposition; not derived from any Oak St. portfolio, strategy, or live data.
This is the sense in which the title of this piece is meant. A research process that knows too much about its own history, that has examined it from too many angles, has consumed the sample. The remedy is not ignorance. It is bookkeeping: recording the number of trials, raising the threshold a candidate must clear accordingly, and reserving data the search has never touched for the one evaluation that counts.
Spending Complexity Deliberately
None of this argues for simple models as a matter of principle. It argues for treating complexity as a budget spent against a specific expected return, with controls in place that make the spending visible. Figure 5 lists the controls we consider basic, what each constrains, what each costs, and the failure it guards against. They are not alternatives. A process that relies on one of them is exposed on the others.
| Control | What it constrains | What it costs | Failure it guards against |
|---|---|---|---|
| Regularization (ridge, lasso, dropout) | Parameter magnitudes; shrinks estimates toward zero or toward a shared structure | Bias: some genuine structure is shrunk along with the noise | Variance from parameters that fit sample-specific noise |
| Early stopping | How far an iterative fit is allowed to proceed | A validation set withheld from training; a slightly under-converged fit | A fit that keeps absorbing noise after it has finished learning structure |
| Parameter budget | The ratio of effective parameters to independent observations | Expressiveness the sample could not have supported anyway | The linear penalty of Figure 1 |
| Nested cross-validation, purged and embargoed | The separation of model selection from model evaluation | Compute, several fits per candidate, and observations lost to the purge and embargo | Leakage across fold boundaries; selecting on the score that is later reported |
| Multiple-testing correction | The threshold a candidate must clear, as a function of the number of trials | Some real signals rejected as the count of trials grows | Selecting the best of many noise candidates (Figure 4) |
| Economic priors | The set of hypotheses admitted to the search | Structure that has no story yet goes unexamined | A search wide enough to find something whatever the data contains |
| Held-out data, examined once | The number of looks at the final evaluation sample | Data unavailable to research; one attempt at the answer | The sample being consumed by the search |
Note: Qualitative summary of the mechanisms described in the text. The controls are complementary; each closes a channel the others leave open.
Sources: Oak St. research. Illustrative, stylized simulation prepared for exposition; not derived from any Oak St. portfolio, strategy, or live data.
The controls fall into two families. The first constrains the model: regularization shrinks parameters toward zero or toward each other; early stopping halts a fit before it has finished absorbing the noise; a parameter budget caps the ratio of parameters to independent observations. These are cheap, mechanical, and easy to audit. The second family constrains the researcher: nested cross-validation, with purging and embargo periods so that adjacent observations do not leak across the fold boundary, keeps the selection of a model separate from its evaluation;[5] multiple-testing corrections raise the bar as the number of trials rises; economic priors restrict the search to relationships with a mechanism, which is the only control that reduces the number of trials rather than merely accounting for them.
That last control is the one we weight most heavily, because it changes the odds rather than only measuring them. A hypothesis with an economic reason behind it, a risk that must be compensated, a constraint on some class of investors, a friction in the way information travels, has a prior probability of being real that a pattern found by search does not. Fitting a rich model to test a hypothesis that could be stated before the data was examined is research. Fitting the same model to see what it finds is the other thing, and the two are indistinguishable from their results alone.
Knowing Less, On Purpose
The practical posture that follows is one of deliberate restraint. Prefer the model whose out-of-sample estimate is within uncertainty of the best and whose complexity is lower. Treat the parameter count as one line in a budget that also includes the number of specifications tried. Keep a portion of history that no one has looked at, and look at it once. Record every trial, including the ones that failed, because the count is the most important number in the process and the easiest to lose.
Markets reward knowing something others do not. They do not reward knowing the past in more detail than the past can support. The cost of knowing too much is paid out of sample, quietly, and by the time it is visible the model that incurred it has usually already been replaced by a richer one.
- [1]The p/N approximation follows from the expected optimism of the training error in a linear model with Gaussian noise; see Hastie, Tibshirani, and Friedman, The Elements of Statistical Learning (2nd ed., 2009), chapter 7, and Akaike's information criterion, which is the same quantity expressed as a penalty. For nonlinear models an effective number of parameters replaces p, and it is often much larger than the count of weights would suggest.
- [2]The orders of magnitude are those of the published return-predictability literature rather than measurements of our own. Welch and Goyal, "A Comprehensive Look at the Empirical Performance of Equity Premium Prediction," Review of Financial Studies (2008), find monthly out-of-sample R² for aggregate returns that is near zero or negative for most predictors, and Gu, Kelly, and Xiu, "Empirical Asset Pricing via Machine Learning," Review of Financial Studies (2020), report monthly out-of-sample R² for individual stocks below one percent even for their best models. A few hundred independent observations is what a few decades of monthly data amount to once overlap and regime drift are accounted for.
- [3]The expression σ · √(2 ln N) is the leading term of the expected maximum of N independent normal draws with standard deviation σ. It overstates the expectation somewhat for small N and ignores correlation among candidates, which reduces the effective N in practice, though rarely by enough to change the conclusion.
- [4]Harvey, Liu, and Zhu, "... and the Cross-Section of Expected Returns," Review of Financial Studies (2016), and Bailey, Borwein, López de Prado, and Zhu, "Pseudo-Mathematics and Financial Charlatanism," Notices of the American Mathematical Society (2014), both treat the count of trials as the missing variable in the evaluation of a backtested signal, and propose thresholds that rise with it.
- [5]López de Prado, Advances in Financial Machine Learning (2018), sets out purged and embargoed cross-validation for overlapping financial observations. The purge removes training observations whose labels overlap the test period; the embargo adds a buffer after it.
Interested in related insights?
The Backtest That Never Happened: How Historical Simulations Learn What No One Could Have Known
Signal vs. Noise: Why a Beautiful Backtest Can Be Statistically Meaningless
Enjoyed this piece?
This document is provided for informational purposes only and does not constitute investment advice or an offer to sell (or the solicitation of an offer to buy) any security, investment product, or service.
The views expressed are those of OAK ST LLC as of the date of the document, are subject to change without notice, and may not reflect the criteria used by OAK ST LLC to evaluate investments. Figures described as illustrative, stylized, or simulated are hypothetical constructions prepared for exposition; they do not depict the results of any OAK ST LLC strategy, portfolio, or account, and no representation is made that any account will or is likely to achieve results similar to those shown. Historical market trends are not reliable indicators of future market behavior.
Information obtained from third-party sources is believed to be reliable but has not been independently verified, and OAK ST LLC does not guarantee its accuracy or completeness. Nothing in this document is a recommendation to buy, sell, or hold any instrument.
This document may not be reproduced or distributed without the prior written authorization of OAK ST LLC. The Terms of Use and the Important Legal and Regulatory Disclosures govern its use. Copyright © 2026 OAK ST LLC. All rights reserved.