SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions

Sun, 13 Sep 2026 00:00:00 +0900

Paper

SQS: Fusing Pruning and Quantization into One Bayesian Learning — Where Spike-and-Slab Meets GMM

TL;DR — Doing pruning and low-bit quantization separately leaves redundancy on the table. SQS couples a spike-and-slab prior with a Gaussian mixture model (GMM) into a single variational learning scheme, achieving higher compression at the same bit-width and lower accuracy loss at the same compression ratio.

X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

Sat, 12 Sep 2026 00:00:00 +0900

Paper

X-AuT: Pruning Audio Encoders in Speech LLMs with Behavioral Probes and Cross-Scale Distillation

TL;DR

X-AuT is a framework that progressively prunes the audio encoder (audio Transformer) of a speech LLM from 18 → 16 → 14 layers, selecting recoverable layer combinations with short behavioral probes, and recovering the loss via cross-scale distillation (1.7B teacher → 0.6B student) followed by 3-stage recovery. The 14-layer model cuts audio-tower parameters by 20.7% while keeping the macro error rate nearly intact at 5.75% (baseline 5.61%); the 16-layer model actually beats the baseline at 5.27% (source: Abstract, §1).

Model Compression on Jaehun's Blog

SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions

SQS: Fusing Pruning and Quantization into One Bayesian Learning — Where Spike-and-Slab Meets GMM

X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

X-AuT: Pruning Audio Encoders in Speech LLMs with Behavioral Probes and Cross-Scale Distillation

TL;DR