Massive Activations

'Massive Activations' 태그의 모든 글

총 2개의 글

시간순 정렬

[논문 리뷰] Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin

[논문 리뷰] Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin

[논문 리뷰] Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coin

논문 링크 Massive Activations가 만든 하나의 서사: Attention Sink 와 Compression Valley 를 잇다 TL;DR 잔차 스트림의 massive activations(특히 BOS) …

2025년 10월 23일

2510.06477v1 Massive Activations Attention Sink Compression Valley Residual Stream Representation Geometry Anisotropy Information Bottleneck Mix–Compress–Refine MLP Ablation Layerwise Analysis LLaMA3 Qwen2 Pythia LogitLens TunedLens LLM Internals Activation Dynamics Paper Review

[논문리뷰] Massive Activations in Large Language Models

논문 링크 Massive Activations, Hidden Biases: 재해석된 Self-Attention의 비밀 한 줄 요약 (TL;DR) 수천만 개 중 4–10 개의 초거대 스칼라(×10 000) 가 LLM · ViT의 성능을 사실상 ‘혼자서’ …

2025년 07월 09일

2402.17762v2 Transformer SelfAttention BiasMechanism RepresentationLearning Interpretability NeuralMechanisms Massive Activations Explicit Attention Bias