The baseline you skipped is the result you needed

There is a genre of confession that appears regularly in the quantitative trading corners of the internet. Someone describes years of work: transformers attending over features then time, then time then features, regime-identification layers, cross-attention, latent representations, variational autoencoders and custom variants of them, memory augmentation, reinforcement learning agents. Hundreds of architectures, dozens of normalisation and transformation schemes, drawn from a pile of papers.

And then the conclusion, always the same: simpler and more transparent systems worked best. Minor variations in model config, training window, asset choice or feature set would cause the complex ones to fail terribly.

One version of it I find hard to shake: I've spent over a decade wading through petabytes of data, trying every combination of hyperparameters with every known model architecture. And this whole time the solution was a basket of EMAs and a random forest classifier?

I have collected a lot of these. The observation is robust and I do not doubt it. What I want to argue is that the standard explanation is wrong in a way that matters.

The standard explanation, and why it is not enough

The usual account is a signal-to-noise argument. Market data is overwhelmingly noise. Complex models have the capacity to fit noise. Therefore complex models overfit and simple models, lacking that capacity, generalise better. Occam's razor with a finance accent.

This is not false. But it does not explain the specific pattern people report, which is not "the complex model generalised slightly worse." It is "the complex model failed terribly under minor perturbations to configuration, asset, or data window."

That is not the signature of mild overfitting. Mild overfitting degrades gracefully. Catastrophic sensitivity to a changed random seed or a shifted training window is the signature of a model that never had signal in the first place, and whose apparent validation performance was an artefact of the validation procedure.

The distinction matters because the two diagnoses prescribe opposite actions. If it is capacity, the fix is regularisation and simpler models. If it is validation, then simplifying is treating the symptom — you will get a simple model whose apparent performance is also an artefact, just a less volatile one, and you will trust it more because it is simple.

The failure is upstream

Consider the sequence of decisions before a model sees data. Someone picks a labelling scheme, a train/test split, a cross-validation strategy, a feature set, a target horizon. In market data, every one of these has a leakage-shaped failure mode.

The split is the obvious one — a random split on time series data lets the model interpolate between points it has effectively seen, and the resulting performance is meaningless. But the subtler ones are worse because they survive scrutiny. Features computed over a window that includes the label period. Normalisation statistics computed over the full dataset before splitting. A universe selected using knowledge of which instruments were liquid across the whole period. Hyperparameters tuned against the test set across dozens of experiments, which is leakage conducted by the researcher rather than the code.

None of these produce an error. All of them produce optimistic validation numbers. And here is the part I think is underappreciated: a complex model exploits this leakage more efficiently than a simple one. More capacity means more ability to find and use whatever shortcut the leak provides. So under a leaky protocol, the complex model looks better in validation and collapses in deployment, while the simple model looks worse in validation and collapses less. The observed ranking is real, and it reverses the true one.

Which is to say: the ranking people report may be an artefact of the same broken validation that produced the numbers being ranked.

The clearest statement of the trap

The most useful thing I have read on this is a piece of ragebait, which I am fairly sure was written as a joke:

You train a small model on scikit-learn. It gets 80% accuracy. You fine-tune a transformer. It gets 90%. You build a custom data pipeline. 93%. You ensemble three models. 95%. Feature engineering and SHAP. 96%. Gradient boosting. 97%. Custom loss function. 98%. You realise the labels were wrong. Fix them, retrain. 99%.

The joke lands because everybody recognises it. Every step in that escalation is a modelling improvement, and every one of them was applied on top of a target definition that was broken. The labels being wrong did not make the model worse — it went from 80% to 98% while the labels were wrong, and the metric rose smoothly the whole way. Nothing in the sequence surfaces the error, because accuracy against a wrong label is a perfectly well-behaved quantity.

There is a related failure with better provenance. A team replicated a forecasting approach from a paper published by a large ride-hailing company. The problem: they were not that company, and their data had nothing in common with millions of ride requests — it was small and fragmented. When the copied approach failed they reached for clustering, because dynamic time warping sounded sophisticated. In production, washing machines in Germany got clustered with coffee makers in Ecuador, because the distance metric said they were similar. The forecasts were accordingly ridiculous, clients complained, and much of the team left.

The moral offered was: ask why you would cluster at all. What is the logic, what is the value? Do not chase technique for technique's sake. A fancy black box that does not work is still a black box.

What I actually do about it

Three things, in order of how much they have saved me.

The simple baseline is a test, not a starting point. If a moving-average crossover beats the transformer out of sample, the transformer pipeline is decoration — but the useful reading is not "use the crossover." It is that something in the pipeline is broken, and the crossover is the instrument that detected it. Baselines are diagnostic equipment.

Treat the backtest as a hostile witness. The questions are specific: did I leak future data, did I tune hyperparameters against the test set, did I select the universe with hindsight, is the P&L carried by a handful of days or one regime. Each of these needs a test that could come back positive, not a reassurance.

Prefer transparency for a reason that is not aesthetic. The argument I find compelling is not that simple models perform better. It is that when a transparent system breaks, you know where to look. A researcher I read put it as not liking surprises, including the big wins — if you do not know why the model made money and cannot reproduce it, you have not learned anything you can rely on next month.

The opinion

"Simple models beat complex ones on market data" is true as an observation and misleading as an explanation, because it lets you attribute a validation failure to model capacity — and the fix it implies, using simpler models, produces a system that is wrong less dramatically rather than one that is right.

The uncomfortable corollary is that the confession genre I opened with may be evidence of something other than what its authors think. Years of architectures failing under minor perturbation is consistent with markets being unpredictable. It is equally consistent with a validation protocol that never had the power to distinguish a working model from a broken one — in which case the years were spent measuring the protocol, not the market.

I cannot tell from the outside which of those it was in any given case. I can tell that almost none of these accounts describe the validation setup in enough detail for a reader to judge, and that the ones which do describe it usually reveal a random split.