X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

2) Transcription-consistency filtering

The training pool combines a heterogeneous corpus exceeding 280k hours (AISHELL, CommonVoice, Emilia, GigaSpeech, LibriSpeech, WenetSpeech, in-vehicle speech, etc.) (source: §4.2). For each utterance, two strong ASR systems (Qwen3-ASR-1.7B, Qwen3.5-Omni) produce offline transcriptions (hypotheses), and the pairwise edit rate (CER/WER) between the reference and the two hypotheses is computed (source: §3.2). Utterances are ranked into 9 consistency tiers (class 1–9) by the maximum discrepancy $e_{\mathrm{max}}$, and the experiments use only class 1 data, the most consistent tier (source: §3.2, Tab. data tiers).

3) Behavioral progressive pruning

Pruning proceeds in two hops (18→16→14). The first hop removes the original layers $\{1,18\}$. In the second hop, all candidates start from the identically recovered 16-layer checkpoint; after the same 0.3-epoch LoRA warm-up, each single-layer removal and each layer-pair removal is evaluated by the total TER across 5 development benchmarks (source: §3.3). This is more expensive than static scores, but it directly measures post-removal behavior under the available budget (source: §3.3).

Why is a pair probe needed? Because recovery after removing multiple layers cannot be reliably predicted from single-layer scores (source: §3.3, §5.4).

4) 3-stage recovery

Each hop uses the same 3-stage recipe. The student is the pruned Qwen3-ASR-0.6B; the teacher is Qwen3-ASR-1.7B with a 24-layer audio tower, and its parameters are frozen and discarded at inference time (source: §3.4).

• Stage 0 (representation alignment, 5%): combines intermediate-layer and bridge representation losses (MSE + cosine distance) with logit KD and transcription CE (source: §3.4):

Because the teacher and student have different hidden widths (2048 vs 1024), a 2-layer MLP with a 256-dimensional bottleneck projects teacher hidden and bridge features into the student space, and the teacher layers are uniformly partitioned into $M$ groups so that student layer $m$ aligns to the last teacher layer of group $m$ (source: §3.4).

• Stage 1 (scheduled student-policy distillation, 95%): intermediate-layer losses are turned off; training proceeds with bridge alignment + teacher-forced logit KD + gold CE (source: §3.4). After 20% of training, every 5th optimizer step the student greedily generates a prefix, and the student and teacher are evaluated on the same generation context to compare distributions over the union of top-$k$ (512) support sets. To prevent premature EOS, a min_new_tokens=3 gate, a duration-aware maximum of 256 tokens, and a degenerate-rollout rejection filter are used; if more than half the batch is rejected, training falls back to teacher forcing (source: §3.4).
• Stage 2 (LoRA fine-tuning, 1 epoch): starting from the best Stage 1 checkpoint, only gold CE is optimized and the tied head is frozen. Class 1 data is kept, but source weights are rebalanced toward the target domains (in-vehicle, AISHELL-4/5) (source: §3.4).

How it works: a concrete example

The full recovery flow in one line (source: Fig. 2):

  flowchart LR
  A["Pretrained 0.6B speech LLM"] --> B["Behavioral probe: layer-combination selection"]
  B --> C["Progressive pruning 18→16→14"]
  C --> D["Stage 0: representation alignment"]
  D --> E["Stage 1: cross-scale distillation"]
  E --> F["Stage 2: LoRA fine-tuning"]
  F --> G["Compressed model for deployment"]

Why “layer-pair interaction” is key, in numbers. In the 16→14 hop, the single-removal sweep shows that L6 (6.29%) is the strongest single removal, followed by L5 (6.42%) and L8 (6.52%) (source: §5.4). One might expect $\{6,8\}$, the union of the two strongest single removals, to be best — but it is not:

Despite combining the two strongest single removals, $\{6,8\}$ is 0.85pp worse than $\{5,6\}$ (source: §5.4, Tab. pair probe). The authors define an “interaction penalty (pair TER − mean TER of the constituent single removals)”: $\{6,8\}$ has 1.38pp, far larger than $\{5,6\}$’s 0.58pp (source: §5.4). One tentative explanation is that removing a contiguous sub-block creates only one discontinuity in the residual stream, whereas a scattered removal creates two — though the authors explicitly label this an “unverified explanation” (source: Appx).

Premature EOS is a genuinely serious problem. In the 2×2 ablation, configuration (A), with no safeguards, produced empty rollouts (empty transcriptions) in 5 of a 100-step window; configuration (D), which makes the tied head trainable and enables the min_new_tokens=3 gate, reduced empty events to 0 and improved the best TER from 6.86% → 6.75% (source: Appx, Tab. EOS ablation). The gate instead increases degenerate rollouts, triggering the rejection filter 2113–2162 times, but this is absorbed by the teacher-forced fallback (source: Appx).

Performance validation: key results

Evaluation is carried out on 10 public benchmarks — AISHELL-1, Fleurs zh/en, LibriSpeech test-clean/other, THCHS-30, Tedlium, CommonVoice v15 zh/en, WenetSpeech-meeting — in CER/WER, and the macro average weights benchmarks equally rather than by utterance count (source: §4.3).

16 layers: beats the baseline. Macro error rate drops from 5.61% → 5.27%, a 6.1% relative reduction, improving on AISHELL-1, CommonVoice zh/en, and WenetSpeech-meeting (4 benchmarks). The largest gains are CommonVoice en (−1.85pp), CommonVoice zh (−1.83pp), and WenetSpeech-meeting (−1.30pp) (source: §5.1, Tab. main_full).

14 layers: 20.7% parameter savings. The audio tower shrinks from 186.376M → 147.794M parameters (removing 4 blocks × 9.645M each), and the macro error rate stays at 5.75%, only +0.14pp over the baseline (source: §5.1, Tab. main_full_14). CommonVoice zh actually improves by +1.59pp, while the largest loss occurs on Fleurs-en at −0.93pp (source: §5.1).

Teacher scale is decisive. In a controlled experiment varying only the teacher under the same recipe, cross-scale distillation with a 1.7B teacher reaches 5.55%, while self-distillation (0.6B teacher) reaches 8.45% — a gap of 2.9pp. The self-teacher is worse than the baseline on all 10 benchmarks (+50.6%), whereas cross-scale beats the baseline on CommonVoice zh/en and AISHELL-1 (source: §5.3, Tab. teacher_scale).

Progressive pruning beats direct pruning. Removing the same layers $\{1,18,5,6\}$ all at once (18→14 in a single step) yields 6.73%; splitting it into two hops yields 5.75% (source: §5.3, Tab. ablation_direct).

Training trajectory. TER collapses in Stage 0 from 10.88% (step 500) to 7.80% (step 2500), then falls to 5.76% (step 47000) in Stage 1, and Stage 2 shaves off another 0.40pp to reach 5.36% (step 28000) (source: §5.2).

Inference efficiency. The 14-layer model cuts encoder time by −21.4% on an in-vehicle PPU (14→11ms) and −11.4% on H800 (88→78ms). However, end-to-end gains are only −4.7% and −2.6%, respectively, because autoregressive decoding dominates total time (source: §5.6, Tab. efficiency).

Our take: strengths, limitations, and why this matters

Strengths. The most compelling part is the empirical case for “don’t separate selection from recovery.” The non-additive interaction between layer pairs ($\{6,8\}$ < $\{5,6\}$), invisible to static importance scores, and the result that progressive pruning beats direct pruning, directly support the claim that recoverability is a function of combination and schedule (source: §5.3, §5.4). Likewise, the contrast of 8.45%→5.55% from merely switching to a cross-scale teacher shows that “transfer of acoustic knowledge from a stronger teacher” goes beyond simple restoration (source: §5.3). The exact reporting of parameters (186.376M, 9.645M/block) is also transparent (source: §4.1).

Limitations. As the authors honestly state, the main results come from a single run with seed 42, with no repeated seeds, confidence intervals, or significance tests (source: §6). So the +0.14pp difference at 14 layers is a “descriptive observation,” not statistically significant. Checkpoint selection also relies on a fixed development subset carved out at 25 utterances per benchmark, so selection noise exists (source: §6). The experimental scope is limited to a single model family (Qwen3-ASR) and a restricted set of pruning candidates, and only class 1 of the 9 data tiers is used (source: §6). Moreover, since only the audio tower is compressed, the end-to-end latency improvement is only −2.6~−4.7%, so the practical gain is large only in “deployments where the encoder and decoder are separated into pipelines” (source: §6, §5.6).

Why it matters. Being able to reduce encoder depth — which determines first-token latency in on-device and in-vehicle speech LLMs — without discarding the pretrained model has substantial practical value. More broadly, X-AuT’s methodology of “directly measuring post-pruning recoverability at the level of combinations” is a reusable principle for any pipeline that compresses pretrained encoder-decoder models, not just speech LLMs. The 3-stage design, which separates representation mismatch (Stage 0) from the errors of self-decoding history (Stage 1), is also a clean engineering decomposition.

What’s next?: the road ahead

The direct follow-ups the authors leave open are clear. First, validating the recipe’s generality across multiple model families (Whisper, Qwen2-Audio, etc.) and wider layer combinations and data mixtures (source: §6). Second, resolving the noise of a single run with repeated seeds and bootstrap confidence intervals — essential for interpreting fine differences such as the +0.14pp at 14 layers (source: §6).

Reasonable next steps include (1) causal activation analysis of layer-pair interactions and a larger factorial candidate set to mechanistically pin down “why scattered removal is more harmful” (source: Appx), (2) deployment research that couples the encoder and decoder with pipeline parallelism so that end-to-end latency fully captures the encoder savings (source: §5.6), and (3) exploring curriculum/re-weighting strategies that go beyond the class 1 fixation and exploit the full 9-tier consistency hierarchy. Ultimately, quantifying “the point where deeper pruning cannot be offset by representation alignment because capacity loss is irreplaceable” is the open question this work leaves behind (source: §6).

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. Nine-tier transcript-consistency hierarchy. Threshold bands use $e_{\mathrm{max}}$, the maximum pairwise edit rate. The tiers measure transcript agreement rather than ground-truth quality.

Table 2. Full-suite error rates for the 16-layer model. S1 is the Stage 1 best checkpoint and S2 is the finetuned checkpoint. Mean is the unweighted macro average. Relative mean-error change is $(\bar e/\bar e_{\rm base}-1)\times100\%$; negative is better. Bold marks the best observed value, not statistical significance.

Table 3. Full-suite error rates for the 14-layer model. The 147.8M audio tower has 20.7% fewer parameters than the 186.4M baseline. Formatting and reporting conventions follow Table .

Table 4. Main 16-layer hyperparameters. The 14-layer run uses the same recipe and initializes from the recovered 16-layer checkpoint.

Table 5. Premature-EOS safeguards on prune-16 in a 2$\times$2 ablation. All configurations share the same Stage 0 checkpoint and Stage 1 recipe. Empty events count 100-step windows with at least one empty rollout; rejections sum budget, length, and repetition filters. TER is the best development-suite observation.

Table 6. Teacher-scale comparison for the 16-layer Stage 1 student. Schedules, data, and optimization are matched; cross-scale training additionally uses the required hidden-width projections. Values are single-run best-checkpoint results.

Table 7. Pair probes for 16$\rightarrow$14 pruning after the same 0.3-epoch warm-up. TER is the macro mean over five development subsets, including proprietary SC. $\Delta$ is relative to $\{5,6\}$. Bold marks the best observed candidate.

Table 8. Direct 18$\rightarrow$14 pruning versus progressive 18$\rightarrow$16$\rightarrow$14 pruning. Both remove $\{1,18,5,6\}$ and use the same recovery/data budget. Single-run best-checkpoint results; bold marks the best observed value.

Figures in this post are taken from the original arXiv:2609.11412 (CC BY 4.0). Only size and format were changed.