With-Deepseek-V4-Pro on Jaehun's Blog

With-Deepseek-V4-Pro on Jaehun's Bloghttps://jaehun.me/en/categories/with-deepseek-v4-pro/Recent content in With-Deepseek-V4-Pro on Jaehun's BlogHugoen-USTue, 08 Sep 2026 10:56:16 +0000ShallowStream: Index Shallow then Answer Deep for Streaming Video Understandinghttps://jaehun.me/en/posts/shallowstream-index-shallow-then-answer-deep-for-streaming-video-understanding/Tue, 08 Sep 2026 00:00:00 +0900https://jaehun.me/en/posts/shallowstream-index-shallow-then-answer-deep-for-streaming-video-understanding/<p><a href="https://arxiv.org/abs/2609.02780v1"target="_blank" class="inline-flex items-center gap-1" >Paper<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="shallowstream-index-shallowly-answer-deeply--unlocking-the-computational-bottleneck-of-streaming-video-understanding">ShallowStream: Index Shallowly, Answer Deeply — Unlocking the Computational Bottleneck of Streaming Video Understanding<a href="#shallowstream-index-shallowly-answer-deeply--unlocking-the-computational-bottleneck-of-streaming-video-understanding" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2><p><strong>TL;DR</strong> — In streaming video understanding, the most expensive cost is prefilling every frame through the MLLM’s <strong>full depth (28–32 layers)</strong>. ShallowStream builds frame encoding and a retrieval index with <strong>only 4–5 shallow layers</strong>, and runs full-depth computation on the retrieved evidence <strong>only at the moment a question arrives</strong>. As a result, it keeps SOTA-level performance (OVO-Bench 69.5 / StreamingBench 78.2) while cutting per-frame prefill by <strong>up to 52.1×</strong> and 10-second end-to-end latency by <strong>up to 11.9×</strong> (source: Abstract).</p>Unlocking Lossless Speedups in LLMs via Discrete Diffusionhttps://jaehun.me/en/posts/unlocking-lossless-speedups-in-llms-via-discrete-diffusion/Tue, 08 Sep 2026 00:00:00 +0900https://jaehun.me/en/posts/unlocking-lossless-speedups-in-llms-via-discrete-diffusion/<p><a href="https://arxiv.org/abs/2609.04010v1"target="_blank" class="inline-flex items-center gap-1" >Paper<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="uno-ar-and-diffusion-in-one-model--lossless-parallel-generation-for-llm-acceleration">Uno: AR and Diffusion in One Model — Lossless Parallel Generation for LLM Acceleration<a href="#uno-ar-and-diffusion-in-one-model--lossless-parallel-generation-for-llm-acceleration" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2><p><strong>TL;DR</strong> — LLMs are slow because next-token prediction (NTP) can emit only one token at a time by its very sequential structure. Uno plants <strong>AR weights that own quality</strong> and <strong>lightweight LoRA diffusion weights that own speed</strong> into a single architecture; the diffusion path drafts token blocks in parallel, and the AR path verifies them via rejection sampling. With neither a separate draft model nor a lossy AR→diffusion conversion, it achieves <strong>lossless</strong> acceleration that exactly preserves the AR distribution. In practice it delivers <strong>up to 3×</strong> speedup over the base AR model and still <strong>up to 2×</strong> at the largest batch sizes, while beating open d-LLMs (26B DiffusionGemma) and proprietary models (Mercury 2) on agentic, coding, and long-form reasoning benchmarks (source: Abstract).</p>BeaconKV: Key-Value Cache Compression Guided by Beacon Queries for Efficient Large Reasoning Model Inferencehttps://jaehun.me/en/posts/beaconkv-key-value-cache-compression-guided-by-beacon-queries-for-efficient-large-reasoning-model-inference/Mon, 07 Sep 2026 00:00:00 +0900https://jaehun.me/en/posts/beaconkv-key-value-cache-compression-guided-by-beacon-queries-for-efficient-large-reasoning-model-inference/<p><a href="https://arxiv.org/abs/2609.04971v1"target="_blank" class="inline-flex items-center gap-1" >Paper<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="catch-the-moment-a-model-revisits-its-thoughts-how-beaconkv-compresses-reasoning-kv-caches-58-with-beacon-queries">Catch the Moment a Model “Revisits Its Thoughts”: How BeaconKV Compresses Reasoning KV Caches 5.8× with Beacon Queries<a href="#catch-the-moment-a-model-revisits-its-thoughts-how-beaconkv-compresses-reasoning-kv-caches-58-with-beacon-queries" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2> <blockquote class="border-primary bg-muted/30 hover:bg-muted/40 my-6 rounded-r-lg border-l-4 py-4 pr-4 pl-6 italic transition-all duration-200 ease-out hover:translate-x-1" > <div class="prose prose-sm max-w-none"> <p><strong>TL;DR</strong> — The KV cache of a Large Reasoning Model (LRM) grows linearly as it produces longer Chains-of-Thought (CoT). When Qwen3-4B generates 32K tokens at batch size 16, the KV cache alone exceeds <strong>77 GB</strong>, approaching the 80 GB GPU limit (source: §1). Existing compression methods predict future important tokens from <strong>recent queries</strong>, but this assumption breaks down during reasoning because of a phenomenon called Thought Revisiting Token (TRT), in which the model <strong>re-reads distant past context (problem definition, solution plan)</strong> (source: §3). Building on the geometric insight that the <strong>global queries</strong> behind TRT form a few clusters in embedding space, BeaconKV selects a <strong>beacon query</strong> to represent each cluster online via Continual FPS and compresses the KV cache accordingly. This achieves <strong>up to 5.8× memory savings (77.0 → 13.3 GB)</strong> and <strong>4.3× throughput gains (82.3 → 356.4 tokens/s)</strong>, while improving accuracy by up to <strong>31.7 pp</strong> over RPC and R-KV (source: §5.4, Fig. 8).</p>Hardware-Aware FP4 FlashAttention-4https://jaehun.me/en/posts/hardware-aware-fp4-flashattention-4/Mon, 07 Sep 2026 00:00:00 +0900https://jaehun.me/en/posts/hardware-aware-fp4-flashattention-4/<p><a href="https://arxiv.org/abs/2609.04105v1"target="_blank" class="inline-flex items-center gap-1" >Paper<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="why-dont-fp4-tensor-cores-make-attention-faster-blackwell-flashattention-4-solved-with-direct-p-and-quantized-backprop">Why Don’t FP4 Tensor Cores Make Attention Faster? Blackwell FlashAttention-4, Solved with Direct-P and Quantized Backprop<a href="#why-dont-fp4-tensor-cores-make-attention-faster-blackwell-flashattention-4-solved-with-direct-p-and-quantized-backprop" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2><p><strong>TL;DR</strong> — Blackwell’s FP4 tensor cores handle matrix multiplication far faster than BF16, but attention does not automatically inherit that benefit, because a “middle operation” — <strong>softmax</strong> — sits between the <em>two matrix multiplications</em>. This paper shortens the critical path with <strong>Direct-P</strong>, which reframes softmax probability generation not as a sequential “accurate exponential → round” path but as a problem of <strong>directly classifying scores into E2M1 codes</strong>, achieving up to <strong>2.13×</strong> forward throughput over BF16 on an NVIDIA GB200 (source: §Abstract). For training, backward reuses the quantization state that forward produced to speed a single step of an 8B model by up to <strong>1.14×</strong>, but lowering P/V to MXFP4 makes <strong>every trajectory diverge</strong>, so P/V must be kept in FP8 (source: §7.6).</p>Language Models Can Control Their Own Attentionhttps://jaehun.me/en/posts/language-models-can-control-their-own-attention/Mon, 07 Sep 2026 00:00:00 +0900https://jaehun.me/en/posts/language-models-can-control-their-own-attention/<p><a href="https://arxiv.org/abs/2609.02737v1"target="_blank" class="inline-flex items-center gap-1" >Paper<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="language-models-control-their-own-attention-declarative-attention">Language Models Control Their Own Attention: Declarative Attention<a href="#language-models-control-their-own-attention-declarative-attention" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2><p><strong>TL;DR</strong> — Existing sparse attention methods still paid $O(N)$ per decoding step by <em>approximating externally</em> which tokens matter. This paper flips the direction: it lets the model <strong>“declare”</strong> directly, inside Chain-of-Thought (CoT), <strong>where it will look</strong>. The inference engine parses this declaration like a tool call and skips most KV cache reads. Without any training (zero-shot), Gemma-4-31B cuts decoding attention cost by <strong>52.0%</strong> with only a <strong>1.27pp</strong> accuracy drop (source: §1, §5.1).</p>Same Request, Different Answer: Quantization Amplifies Cache-Induced Divergence in LLM Servinghttps://jaehun.me/en/posts/same-request-different-answer-quantization-amplifies-cache-induced-divergence-in-llm-serving/Mon, 07 Sep 2026 00:00:00 +0900https://jaehun.me/en/posts/same-request-different-answer-quantization-amplifies-cache-induced-divergence-in-llm-serving/<p><a href="https://arxiv.org/abs/2609.04748v1"target="_blank" class="inline-flex items-center gap-1" >Paper<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="same-request-different-answer-prefix-caching-makes-serving-non-reproducible-and-quantization-amplifies-it">Same Request, Different Answer: Prefix Caching Makes Serving Non-Reproducible, and Quantization Amplifies It<a href="#same-request-different-answer-prefix-caching-makes-serving-non-reproducible-and-quantization-amplifies-it" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2> <blockquote class="border-primary bg-muted/30 hover:bg-muted/40 my-6 rounded-r-lg border-l-4 py-4 pr-4 pl-6 italic transition-all duration-200 ease-out hover:translate-x-1" > <div class="prose prose-sm max-w-none"> <p><strong>TL;DR</strong> — Prefix caching is billed as a “transparent optimization,” but it is not transparent. With caching off, repeated runs of the same workload are <strong>bit-identical across all 10 configurations × 80 episodes = 800 runs</strong> (source: Tab. 2); with caching on, agent trajectories change in <strong>36.2% of episodes at 16 bits and 75.0% under 4-bit quantization</strong> (source: §IV-D). Yet average accuracy does not move — this is not “degradation” but “instability,” and the two call for different responses (source: §IV-F, §V).</p>SGD-KV: Summarization Guided KV Cache Compressionhttps://jaehun.me/en/posts/sgd-kv-summarization-guided-kv-cache-compression/Mon, 07 Sep 2026 00:00:00 +0900https://jaehun.me/en/posts/sgd-kv-summarization-guided-kv-cache-compression/<p><a href="https://arxiv.org/abs/2609.03235v1"target="_blank" class="inline-flex items-center gap-1" >Paper<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="sgd-kv-finding-heads-that-are-good-at-summarizing-cuts-the-1m-token-kv-cache-by-up-to-75">SGD-KV: Finding ‘heads that are good at summarizing’ cuts the 1M-token KV cache by up to 75%<a href="#sgd-kv-finding-heads-that-are-good-at-summarizing-cuts-the-1m-token-kv-cache-by-up-to-75" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2> <blockquote class="border-primary bg-muted/30 hover:bg-muted/40 my-6 rounded-r-lg border-l-4 py-4 pr-4 pl-6 italic transition-all duration-200 ease-out hover:translate-x-1" > <div class="prose prose-sm max-w-none"> <p><strong>TL;DR</strong> — Attention heads do not all do the same job. <strong>SGD-KV</strong> pinpoints the <strong>summarization heads</strong> — the ones responsible for the high-level information compression called ‘summarization’ — with a diagnostic task, then hands out the KV cache budget in proportion to their scores. At contexts of up to 1 million (1M) tokens it cuts KV cache memory by up to <strong>75%</strong> while beating existing head-level compression methods.</p>SMELT: Scaling Laws for Compute-Matched MoE Looped Transformershttps://jaehun.me/en/posts/smelt-scaling-laws-for-compute-matched-moe-looped-transformers/Mon, 07 Sep 2026 00:00:00 +0900https://jaehun.me/en/posts/smelt-scaling-laws-for-compute-matched-moe-looped-transformers/<p><a href="https://arxiv.org/abs/2609.01343"target="_blank" class="inline-flex items-center gap-1" >Paper<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="smelt-scaling-laws-for-compute-matched-moe-loop-transformers">SMELT: Scaling Laws for Compute-Matched MoE Loop Transformers<a href="#smelt-scaling-laws-for-compute-matched-moe-loop-transformers" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2><h2 id="tldr">TL;DR<a href="#tldr" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2><p>Loop transformers scale depth by executing a layer block repeatedly, but most prior work fixed <strong>only the parameter count</strong> and let FLOPs grow, racking up “free wins.” SMELT shows that looping is a pure architectural gain even while matching all three budgets — <strong>per-token FLOPs, total parameters, and KV cache</strong> — simultaneously (source: §1). On the compute-optimal frontier it saves <strong>6.8–18.0% of training FLOPs</strong>, and these gains come out larger downstream than validation loss predicts (source: §4.3, §5.1).</p>Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLMhttps://jaehun.me/en/posts/why-gated-deltanet-survives-4-bit-quantization-nvfp4-w4a4-for-the-recurrent-half-of-a-hybrid-27b-llm/Mon, 07 Sep 2026 00:00:00 +0900https://jaehun.me/en/posts/why-gated-deltanet-survives-4-bit-quantization-nvfp4-w4a4-for-the-recurrent-half-of-a-hybrid-27b-llm/<p><a href="https://arxiv.org/abs/2609.04098"target="_blank" class="inline-flex items-center gap-1" >Paper<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="the-recurrent-half-is-the-easy-to-quantize-half-why-gated-deltanet-survives-at-4-bits">The Recurrent Half Is the Easy-to-Quantize Half: Why Gated DeltaNet Survives at 4 Bits<a href="#the-recurrent-half-is-the-easy-to-quantize-half-why-gated-deltanet-survives-at-4-bits" class="heading-anchor" aria-label="Link to this section"><svg class="h-4 w-4" aria-hidden="true" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><g fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2"><path d="M10 13a5 5 0 0 0 7.54.54l3-3a5 5 0 0 0-7.07-7.07l-1.72 1.71"/><path d="M14 11a5 5 0 0 0-7.54-.54l-3 3a5 5 0 0 0 7.07 7.07l1.71-1.71"/></g></svg></a></h2><p><strong>TL;DR</strong> — The 48 recurrent <strong>Gated DeltaNet (GDN)</strong> layers of a hybrid 27B LLM (Qwen3.8-27B) have until now been protected at 8–16 bits, on the intuition that “error accumulates across the context.” This paper shows that intuition is <strong>exactly backwards</strong>. <code>Minima</code>, which quantizes <strong>all 496 layers — GDN gates included — to NVFP4 W4A4</strong>, matches BF16 within seed noise on 6 benchmarks (5-task average −0.52) while also being the smallest (17.5 GiB) and the fastest at prefill (+14–19%). On top of that, a four-stage mechanistic study delivers an architecture-level account of <strong>why</strong> — block scaling localizes outliers, gate nonlinearity compresses noise, and the delta rule actively erases state error.</p>