Attention-Kernels on Jaehun's Bloghttps://jaehun.me/categories/attention-kernels/Recent content in Attention-Kernels on Jaehun's BlogHugoko-krSun, 20 Sep 2026 09:55:15 +0900VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attentionhttps://jaehun.me/posts/paper-2609-15810/Sun, 20 Sep 2026 00:00:00 +0900https://jaehun.me/posts/paper-2609-15810/<p><a href="https://arxiv.org/abs/2609.15810"target="_blank" class="inline-flex items-center gap-1" >논문 링크<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="vc-attention-저비트-어텐션의-남은-두-병목을-푼-value-smoothing과-softmax-casting">VC-Attention: 저비트 어텐션의 남은 두 병목을 푼 Value Smoothing과 Softmax Casting<a href="#vc-attention-%ec%a0%80%eb%b9%84%ed%8a%b8-%ec%96%b4%ed%85%90%ec%85%98%ec%9d%98-%eb%82%a8%ec%9d%80-%eb%91%90-%eb%b3%91%eb%aa%a9%ec%9d%84-%ed%91%bc-value-smoothing%ea%b3%bc-softmax-casting" class="heading-anchor" aria-label="이 섹션에 대한 링크"></a></h2><p>한 줄 요약 (TL;DR): 비디오 DiT의 저비트 어텐션에서 값(value) 양자화 오차와 FP32 softmax를 동시에 제거해 B200에서 어텐션 1.59× , 클립 생성 1.19× 를 달성한 학습-프리 커널이다 (근거: §4.3, Fig. 6).</p>