Transformer

10 pages

StepAudio 3 Gen Technical Report

Paper StepAudio 3 Gen: Unifying Speech, Song, Music, and Sound Effects with a Single Discrete Autoregressive (RVQ) ModelTL;DRStepAudio 3 …

September 17, 2026

5 min

Multimodal Learning Large Language Models Transformer Audio Generation Text-to-Speech Reinforcement Learning

SAS: Simple Attention Sparsification via End-to-End Optimization of Context Ranking

Paper SAS: A Simple Attention Sparsification That Learns Context Ranking Directly Without DistillationTL;DR — In post-training attention …

September 15, 2026

20 min

KV Cache Long Context Efficient Inference Attention Optimization Large Language Models Transformer

Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation

Paper Marigold V2: Reviving an Image-Editing Diffusion Transformer (DiT) as a Monocular Depth EstimatorTL;DRThis study fine-tunes an …

September 13, 2026

18 min

Computer Vision Image Generation Transformer Machine Learning Artificial Intelligence Monocular Depth Estimation

SQS: Bayesian DNN Compression through Sparse Quantized Sub-distributions

Paper SQS: Fusing Pruning and Quantization into One Bayesian Learning — Where Spike-and-Slab Meets GMMTL;DR — Doing pruning and low-bit …

September 13, 2026

13 min

Machine Learning Large Language Models Transformer Efficient Inference Model Compression Quantization

X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

Paper X-AuT: Pruning Audio Encoders in Speech LLMs with Behavioral Probes and Cross-Scale DistillationTL;DRX-AuT is a framework that …

September 12, 2026

11 min

Efficient Inference Large Language Models Transformer Machine Learning Knowledge Distillation Model Compression

Deadline-Aware Adaptive Prefill Chunking for Efficient Large Language Model Serving

Paper Deadline-Filling Prefill Chunks: SLOWeave — An Adaptive Chunking Scheduler for LLM Serving TL;DR — Instead of a fixed prefill chunk …

September 11, 2026

15 min

LLM Inference KV Cache Long Context Inference Acceleration Efficient Inference Transformer

SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers

Paper SMELT: Scaling Laws for Compute-Matched MoE Loop TransformersTL;DRLoop transformers scale depth by executing a layer block repeatedly, …

September 7, 2026

11 min

KV Cache Transformer Mixture of Experts

[Paper Review] Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Paper Link Structured State Space Duality: Unifying SSMs and Attention with Mamba-2 for 2–8× AccelerationTL;DRStructured State-Space Duality …

October 3, 2025

19 min

Mamba SSM Transformer Long Context Efficient Training FlashAttention GPU Acceleration

[Paper Review] Massive Activations in Large Language Models

Paper Link Massive Activations, Hidden Biases: A Reinterpretation of Self-Attention’s Secrets TL;DRJust 4–10 extreme scalar values (×10,000) …

July 9, 2025

21 min

Transformer Interpretability NeuralMechanisms Massive Activations Explicit Attention Bias

Code I/O: Condensing Reasoning Patterns via Code Input-Output Prediction

Paper CODE I/O: From Code I/O + Natural-Language CoT to General-Purpose Reasoning — Lifting 7B-30B LLMs by +2 Points on Average with Data …

July 7, 2025

36 min

DeepSeek LLM Chain-of-Thought Instruction Tuning Transformer Long Context