Ten instrumental metrics, two corpora, and a jury that cannot agree

The scaling problem this attacks

The listener experiment took 80 participants, 74 minutes each, ethics approval and a noise-panelled room, to produce one result. That does not scale to a product-development cycle where a new device or firmware revision needs evaluating every few weeks.

The field's response is instrumental metrics: signal-processing algorithms that consume audio and emit a number intended to predict what listeners would have said. Full-reference metrics need the original and the degraded signal; no-reference metrics need only the degraded one. Most of the good ones compute on the amplitude envelope internally, which makes them a natural fit for a thesis about envelope fidelity.

The framing matters here, and the chapter is explicit about it: the goal is not to evaluate the metrics' performance, and not to declare one headphone better than another. It is to establish whether a curated panel of these models is sensitive enough to register the ANC difference on shallow-envelope speech — and to see what happens when you ask ten of them the same question.

The jury

Ten metrics, each with a different theoretical commitment, deliberately spanning envelope-explicit and envelope-agnostic designs:

• FES (Fidelity of Envelope Shape) — degree to which target envelope shape is preserved per frequency channel, averaged over sub-bands. Built for hearing-aid compression; the direct ancestor of chapter 2's EDI.
• GPSMq (Generalized Power Spectrum Model for quality) — modulation-filterbank based; computes envelope power signal-to-noise ratios on internal representations. Validated as more accurate than HASQI, PESQ and PEMO-Q on loudspeaker distortions specifically. A distortion metric, so higher is worse, range 4.0–17.6.
• HASPI (Hearing Aid Speech Perception Index, v2) — combined coherence and envelope-fidelity intelligibility measure, with audiogram support; v2 uses a ten-neural-network ensemble auditory model.
• HASQI (Hearing Aid Speech Quality Index) — product of a non-linear term sensitive to noise and distortion and a linear term sensitive to long-term spectral change.
• PESQ (ITU-T P.862) — the industry workhorse. No overt dependence on amplitude modulation; uses a log-energy envelope only for alignment. Included partly as a control for what happens when the metric is not envelope-based.
• PEMO-Q — modulation-filterbank based, correlation of internal representations; its Perceptual Similarity Measure correlates r = 0.97 with listener mean opinion scores.
• STOI (Short-Time Objective Intelligibility) — correlation between log-power envelopes over ~384 ms segments; only two free parameters.
• ViSQOL — built on a neurogram similarity index, i.e. auditory-nerve responses as the internal representation. Not explicitly envelope-based.
• wSTMI (Weighted Spectro-Temporal Modulation Index) — applies a band-importance weighting to the modulation spectrum, fixing STMI's assumption that all modulation frequencies matter equally.
• SRMR — the only no-reference metric here; a speech-to-reverberation modulation energy ratio built for de-reverberation assessment.

Every implementation was used as published, unmodified.

The two-corpus design

This is the methodological decision I would defend hardest. Chimeras were generated from two speech corpora:

• CRM — 10 sentences at each of 26 modulation depths, 260 signals, dc bias 0.008 (matched to the pilot values from the human experiment).
• NOIZEUS — 30 samples of ~3 s at each of 33 depths, 990 signals, dc bias 0.0058.

Depths ran 0 to −32 dB, enough range to span easy through difficult. The two dc biases were deliberately set slightly apart to produce different overall degradation, so that trends could be compared rather than a single operating point measured.

The reason for two corpora is a specific, unglamorous worry: instrumental metrics are routinely developed and tuned on the same handful of public datasets — NOIZEUS above all. If a metric were quietly overfit to its development corpus, a single-corpus study would report that overfitting as a finding about ANC. Running both is the cheapest available check, and it earned its keep.

What the jury said

On the headline question, most of the panel agrees: the ANC-condition effect is significant for 8 of 10 metrics on CRM and 10 of 10 on NOIZEUS, mostly at p < 0.001, and the direction matches the physical chapters — envelope fidelity is better with ANC on. Some effect sizes are enormous by the standards of this thesis: GPSMq at η²ₚ = 0.90 (CRM) and 0.89 (NOIZEUS), HASQI at 0.73 and 0.67, PEMO-Q at 0.63 and 0.18.

Then the disagreements start, and they are the actual content.

Two metrics report no ANC effect on CRM. STOI (p = 0.137) and SRMR (p = 0.154) find nothing — while GPSMq on the same signals finds an effect explaining 90% of the variance. On NOIZEUS, STOI is still null (p = 0.0597) but SRMR becomes significant (p < 0.001). A metric whose verdict flips with the corpus is not measuring what its name claims to measure, at least not here.

SRMR reports the effect backwards. Its means run 2.26 (ANC on) against 2.99 (off) on CRM and 2.92 against 3.70 on NOIZEUS — consistently worse with ANC enabled, for a metric where higher is better. Every other metric points the other way. SRMR is the one no-reference metric in the panel and was designed for reverberation, so the most likely reading is that it is out of domain rather than that it has found something the others missed. But "the metric is probably wrong" is a hypothesis, and it is stated as one.

Effect sizes are wildly inconsistent across metrics. For the same recordings and the same question, ANC-condition effect size ranges from 0.001 (SRMR, CRM) to 0.90 (GPSMq, CRM) — three orders of magnitude. PESQ, the least envelope-dependent full-reference metric, lands at 0.42 on CRM but 0.04 on NOIZEUS: a tenfold swing between corpora.

Interaction terms are large and pervasive. GPSMq shows headphone × ANC interaction at 0.75 (CRM) and 0.84 (NOIZEUS); HASQI at 0.56 and 0.48; ViSQOL at 0.10 and 0.40. This reinforces the interaction finding from chapter 2 in a much stronger form: what ANC does to the playback signal is a property of the particular device, and any single-number "ANC quality" figure would be averaging over the thing that matters most.

HASPI is saturated and should be discarded here. Its means sit at 0.99998 with standard deviations around 1e-06. The ANOVA reports significance, but a metric pinned against its ceiling with variance in the sixth decimal place is not measuring this degradation — it is reporting that the speech is intelligible, which nobody disputed.

The inconvenient conclusion

The chapter's own summary: the metrics do not agree much, even at the level of broad trends across ANC conditions or headphones, so no one of them can be relied on to characterise ANC headphones. This is a known failure mode — perceptual instrumental models perform well on the degradation types they were validated against and degrade rapidly on novel signal types, and shallow-envelope chimeras are novel for all ten.

That is a less satisfying result than "the metrics agree with the listeners, so use the metrics." It is also more useful, because it is actionable. If you wanted to build the "ANC index" the chapter proposes as future work, this data says which jurors to seat: GPSMq, HASQI and PEMO-Q are sensitive and consistent across both corpora; STOI and SRMR are unusable on this degradation; HASPI saturates; PESQ is corpus-dependent. Nobody could have known that in advance, and the only way to find out was to run all ten on two corpora and look at the disagreements instead of averaging them away.

Cross-checking against the human data adds one more constraint. Chapter 4's listeners showed an intelligibility effect size of 0.0008 and no effort effect. GPSMq reports 0.90. Whatever GPSMq is exquisitely sensitive to, it is not something the listeners consciously experienced — which is a caution about reading any large instrumental effect size as a perceptual promise, and an argument that these metrics measure signal fidelity well and human benefit only loosely.

Honest limits

Metrics were run as-is, without tuning to this degradation type — deliberately, since tuning them would make the sensitivity finding circular, but it does mean each is being used slightly out of its designed domain. Only two corpora, both English, so accent and language effects are unexamined; the same chimera parameters across corpora with different accents would be a clean follow-up. The no-reference category is represented by exactly one metric, and it is the one that misbehaved, so no general conclusion about no-reference metrics is available. And the fourth question the chapter set out to answer — whether there are meaningful patterns in the scores across modulation depths — resisted statistical treatment entirely: the modulation-depth functions are erratic but not random, hinting at structure in the interaction between headphone, ANC behaviour, chimera and metric that this design cannot resolve.

Where it sits

This is the machine half of the modulation-depth axis; the 80-listener experiment is the human half, on chimeras from the same algorithm. The modulation-frequency axis is covered by the electroacoustic metrics and the simulated auditory nerve. All rest on the problem framing.

Full thesis

The full PhD thesis, with all figures, statistical tables, and references, is available as a PDF: PhDThesis_Sohhom_15510011_final.pdf.