https://jaehun.me/tags/none/Recent content in None on Jaehun's BlogHugoko-krFri, 18 Sep 2026 23:50:33 +0900https://jaehun.me/posts/deepseekmoe-towards-ultimate-expert-specialization-in-mixture-of-experts-language-models/Sun, 29 Jun 2025 00:00:00 +0900https://jaehun.me/posts/deepseekmoe-towards-ultimate-expert-specialization-in-mixture-of-experts-language-models/<p><a href="https://arxiv.org/abs/2401.06066v1"target="_blank" class="inline-flex items-center gap-1" >논문 링크<svg class="h-3 w-3 flex-shrink-0" id="external-link" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24"><path fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M15 3h6v6m-11 5L21 3m-3 10v6a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2V8a2 2 0 0 1 2-2h6"/></svg> </a></p> <h2 id="deepseekmoe-정리--dense-성능을-40--flops로-끌어낸-세분화-전문가-트릭">DeepSeekMoE 정리 – Dense 성능을 <strong>40 % FLOPs</strong>로 끌어낸 ‘세분화-전문가’ 트릭<a href="#deepseekmoe-%ec%a0%95%eb%a6%ac--dense-%ec%84%b1%eb%8a%a5%ec%9d%84-40--flops%eb%a1%9c-%eb%81%8c%ec%96%b4%eb%82%b8-%ec%84%b8%eb%b6%84%ed%99%94-%ec%a0%84%eb%ac%b8%ea%b0%80-%ed%8a%b8%eb%a6%ad" class="heading-anchor" aria-label="이 섹션에 대한 링크"></a></h2><h3 id="한-줄-요약-tldr">한 줄 요약 (TL;DR)<a href="#%ed%95%9c-%ec%a4%84-%ec%9a%94%ec%95%bd-tldr" class="heading-anchor" aria-label="이 섹션에 대한 링크"></a></h3><p><strong>Fine-Grained Expert Segmentation (FGES) + Shared Experts (SEI)</strong> 로 FFN-MoE를 재설계한 <strong>DeepSeekMoE</strong>는 동일 FLOPs에서 <strong>Dense 상한선의 95 %↑</strong> 성능을 달성하고, 16 B 모델 기준 <strong>LLaMA-2 7 B와 동급 품질을 연산량 0.4×</strong> 로 구현한다.</p>