The test that ignores the logic of your experiment

Here is the shape of the data. I had a manipulation with a continuous parameter — modulation depth, swept across thirty-three values from untouched speech down to nearly flat. At each value I had a metric computed under two conditions, ANC on and ANC off. So: two curves, thirty-three points each, and the question of whether the condition made a difference.

The reflexive move in the psychophysics and psychology literature is a repeated-measures two-way ANOVA. Condition is one factor. The swept parameter is the other, entered as thirty-three levels. Look for a main effect of condition, look for an interaction, report the F and the p, move on. Everyone does this. Reviewers expect it.

I ran it. Then I stopped, because something about it was wrong, and it took me a while to name what.

What the factorial model throws away

Entering a continuous swept parameter as a categorical factor tells the model that your thirty-three levels are thirty-three unordered categories. Level 12 and level 13 are, as far as the test is concerned, no more related to each other than level 12 and level 31. The model does not know that the parameter has an order. It does not know the spacing between levels. It certainly does not know that the underlying function is expected to be smooth and monotone.

But the order and the smoothness are the whole design. I did not pick thirty-three arbitrary conditions. I picked a range and sampled it, because the shape of the curve across the range is the object of study. Whether the two curves diverge at the shallow end, at the deep end, or uniformly is a different scientific claim in each case, and an interaction term collapses all three into one significant p-value that cannot distinguish them.

The clearest statement of this I found was in, of all places, a statistics FAQ page for a commercial plotting package, arguing that you should instead quantify each subject's curve in some meaningful way — area under it, its peak, its time to peak, a fitted rate constant — and compare those values across groups. Then came the line that reorganised my thinking:

This kind of analysis follows the scientific logic of the experiment, unlike two-way ANOVA which essentially ignores the entire logic of the experiment.

That is a strong sentence and I think it is correct. The factorial test is not wrong in the sense of being invalid. It is wrong in the sense of being uninformative about the thing you set out to learn, while looking exactly like the thing that answers it.

The p-value problem underneath

There is a second issue that the ANOVA route obscures, and it is worse than the first.

Active noise control is a closed-loop system reacting to its acoustic environment in real time. Run the identical input signal through the identical headphone twice and the recordings are not identical. There is genuine trial-to-trial variability in the device itself, on top of the recording chain. So statistical treatment is appropriate — this is not a deterministic measurement.

But my thirty-three-point curves came from a recording protocol running overnight, and with enough repetitions almost any systematic difference becomes significant. Thirty sentences, thirty-three depths, four offset values, four headphones, two conditions: the effective sample size is large enough that significance stops being informative. The question was never "is the difference distinguishable from zero." It was "is the difference big enough to care about."

Which is the standard argument for reporting effect sizes, and I want to put the sharpest version of it on the record because it changed how I wrote the entire thesis:

It is possible, and unfortunately quite common, for a result to be statistically significant and trivial. It is also possible for a result to be statistically nonsignificant and important.

For my chapter, effect size was not a supplementary courtesy. It was the primary result, because the metrics returned values on eight mutually incompatible scales and a unitless standardised effect was the only representation in which "did ANC matter more for this metric than that one" could even be asked.

What I did instead

Three changes, in order of how much they mattered.

I reported effect sizes as the headline and significance as context. Eta-squared per comparison, computed alongside the existing tables. Unitless, comparable across metrics on different scales, and honest about magnitude in a way an F-ratio at large n is not.

I moved to nonparametric tests where the distributional assumptions were doing real work. Rank-based comparisons — Mann-Whitney and its relatives — preserve more of the raw variability in the data than fitting a parametric model to curves whose functional form I could not defend. I had considered fitting a regression with the swept parameter as a continuous predictor, which at least respects the ordering. I decided against it, on the grounds that I had no principled model for the shape and any parametric fit would be smuggling in an assumption I could not justify. Where a psychometric function was involved, model-free estimation was available and was the better answer for the same reason.

I stopped treating the swept axis as a factor and started treating the curve as the unit of analysis. This is the change that follows the logic of the experiment rather than fighting it.

The opinion

Choosing a statistical test is a modelling decision about what your experiment means, not a lookup from a decision tree keyed on data type — and the standard factorial test is the single most common way that a well-designed experiment gets analysed as though it were a badly-designed one.

The reason this happens is that the ANOVA route is enormously convenient. Everything is set up for it. The software wants a long-format table with factor columns, the reporting conventions are fixed, the reviewers recognise the output. Meanwhile the analysis that follows the logic of your experiment usually has to be assembled by hand, has no canonical way to report, and invites questions.

I do not think there is a clean way around that cost. I do think the trade is worth naming when you make it: I chose the harder analysis because the easy one would have answered a question I was not asking, with a number that would have looked exactly as authoritative.