--- name: interspeech-experiments description: Use when designing or auditing the experimental evidence for an INTERSPEECH paper — task-correct metrics (WER/CER, MOS/CMOS, EER/minDCF, PESQ/STOI), baselines reviewers accept, significance testing over utterances and seeds, condition coverage across speakers/noise/languages, and data hygiene for speech corpora. --- # INTERSPEECH Experiments "Not convincing" is the standard Interspeech rejection, and it almost always means the experimental design — not the idea — failed. Speech evaluation has decades of conventions per task; an experiment section that ignores them is illegible to the reviewer pool regardless of how good the numbers are. ## Metric-task law | Task family | Primary metrics | Convention notes | |---|---|---| | ASR | WER / CER | normalization + scorer disclosed; CER for unsegmented scripts | | TTS / VC | MOS, CMOS (+ objective proxies) | panel protocol reported; CMOS for close systems | | Speaker verification | EER, minDCF | official trial lists; DCF prior/costs stated | | Diarization | DER / JER | collar and overlap handling stated | | Enhancement / separation | PESQ, STOI/ESTOI, SI-SDR (+ DNSMOS-style proxies) | wideband vs narrowband named | | SLU / speech translation | intent acc / F1, BLEU/COMET on ASR output | cascaded vs end-to-end made explicit | | Paralinguistics / health | UAR, F1 | speaker-disjoint splits are mandatory | Using a proxy where the community expects the primary (e.g., only neural MOS predictors for a TTS claim) needs an explicit defense sentence. ## Baselines that count - The **stock recipe** of a public toolkit on the same corpus (an ESPnet/ SpeechBrain recipe number is a shared, checkable baseline). - The **latest challenge baseline** if your task has a running challenge — reviewers know those numbers by heart. - Your method's **ablated self** — at Interspeech, one clean ablation of the single proposed component often persuades more than two extra datasets. - Reimplemented prior work must be validated: show your reimplementation matches its published number before showing you beat it. ## Significance: over what randomness? State which variation your statistics cover — the two are routinely conflated: - **Test-set variation**: bootstrap over utterances (or speakers, if claims are speaker-level) → CI on the metric difference between systems. - **Training variation**: multiple seeds → mean ± sd; a 0.2 WER gain with sd 0.3 across seeds is not a result. - Matched-pairs tests (paired bootstrap; the classic MAPSSWE-style segment test for ASR) for A-vs-B claims on the same test set. - Subjective scores: CIs over raters *and* stimuli; ±0.1 MOS is panel noise under most protocols (see `interspeech-reproducibility`). ## Condition coverage — the speech-specific axis A speech claim is implicitly quantified over speakers, acoustic conditions, and often languages. Reviewers probe the quantifier: - **Speaker-disjointness**: train/test speaker overlap invalidates verification, paralinguistic, and health claims outright. - **Condition breakdown**: report clean vs noisy, near- vs far-field, read vs spontaneous where the corpus offers them — an average hides the regression your method causes in one condition. - **Language scope**: "multilingual" needs a per-language table; English-only results support English-only claims. - **Domain leakage**: SSL pretraining data overlapping the test corpus (the LibriSpeech-descendant problem) must be checked and stated for foundation-model work. ## Data hygiene - Official partitions only, or published manifests for custom splits. - No tuning on test: LM weights, thresholds, and checkpoint selection all happen on dev — say so in one line. - License and consent status of every corpus stated (see `interspeech-artifact-evaluation`); leaked or scraped audio can sink an otherwise strong paper on ethics review. ## Designing inside 4 pages Budget roughly one column for the decisive comparison, half for ablation, half for analysis. The analysis half is what separates accepted Interspeech papers: one error-pattern finding (where the gains live — short utterances, overlapping speech, a phone class) converts a benchmark delta into a scientific statement. ## Worked micro-example: is 4.9 vs 5.6 WER real? ```text Claim: proposed 4.9% vs baseline 5.6% WER on test-other (2939 utts). 1. Paired per-utterance errors → paired bootstrap, 1000 resamples. 2. Δ WER 95% CI: [-0.9, -0.5] — excludes 0 → test-set variation covered. 3. Across 3 seeds: 4.9, 5.0, 4.8 (sd 0.1) vs 5.6, 5.7, 5.6 (sd 0.06) → training variation does not swallow the gap. 4. Report: "−0.7 abs. WER (95% CI [−0.9, −0.5], paired bootstrap; consistent across 3 seeds)." ``` Two randomness sources, two checks, one sentence in the paper. If step 2's CI had straddled zero, the honest paper reports the trend and softens the verb — and usually survives review better than the inflated version. ## Negative results and regressions Interspeech's mixed jury respects a disclosed regression far more than a suspicious clean sweep. If the method loses on clean speech while winning on noisy, print both numbers and make the trade-off the story — condition-dependent behavior is a *finding* in a field about acoustic variability, and hiding it is the reviewer-trust equivalent of a failed significance test. ## Pre-submission experiment audit ```text [ ] Primary metric matches task convention; ruler disclosed [ ] Baseline set includes a public-recipe or challenge anchor [ ] Each A>B claim carries a CI or matched-pairs test [ ] Seeds: n stated; variance reported or single-run admitted [ ] Speaker-disjoint splits verified where required [ ] Condition/language breakdown present; regressions named [ ] Dev-only tuning stated; test touched once [ ] One analysis finding, not just deltas ``` ## Output format ```text [Claim inventory] each claim → metric → evidence status [Metric-law check] conventions met / violations [Baseline verdict] anchored / self-referential / stale [Statistics] randomness covered (test-set / seeds / raters) per claim [Coverage gaps] speaker / condition / language / leakage [Cheapest decisive fix] ``` Metric conventions are community law rather than CFP text and move slowly, but challenge editions and recipe baselines roll every year — re-anchor at design time (sources logged in `resources/official-source-map.md`, 2026-07-08).