DeepSeek-V3 Technical Report

Prompt 1.1.1 (research gap)

"Analyze the 'Introduction' and 'Related Work' sections of the paper and explain the core research gap, the decisive limitations of prior work, or the open questions this study explicitly sets out to address. Summarize the 'state of the art' the authors describe at the time of publication."

Core summary (3 lines)

1. The research gap is the ‘performance gap’ in which open-source LLMs still fall short of GPT-4o and Claude-3.5, together with the cost and efficiency limits of training and running megamodels.
2. Existing MoEs use aux-loss to balance load, sacrificing performance, while high-efficiency techniques such as FP8 and multi-token prediction had not been validated on very large models.
3. DeepSeek-V3 combines ①an aux-loss-free (load-balancing-bias) MoE, ②the world’s first 671B-param FP8 training run, and ③a 128K context + multi-token prediction objective to approach GPT-4o/Claude-3.5 in performance while cutting training cost to $5.6 M (−56%).

1. Research gap and open questions

2. Decisive limitations of prior work

DeepSeek-V3 systematically resolves these limitations with an MLA-compressed KV cache, a DualPipe 2-way pipeline, and node-limited EP routing.

3. Snapshot of the state of the art (SOTA) at the time of the paper

*Source: figures published in each model’s report/blog, converted assuming the same hardware.

4. Conclusion & implications

• Academic contribution: aux-loss-free MoE and large-scale FP8 present a new Pareto frontier of high performance at low cost.
• Engineering contribution: DualPipe PP and EP64 cross-node communication kernels maximize efficiency on H800 clusters.
• Open questions: (1) the root cause of the reasoning gap that still remains vs. closed-source models? (2) is further precision reduction to FP4/INT4 feasible? (3) the stability of dynamic expert routing at smaller batch sizes?

Prompt 1.1.2 (central hypothesis)

"What is the central hypothesis or core claim of this paper? State it in one clear, concise sentence in the form: 'The authors hypothesize that by using [proposed technique] they can achieve [specific result], overcoming [existing limitation].'"

Central hypothesis in one sentence The authors hypothesize that, by using a 671B-parameter MoE (37B active) that combines aux-loss-free load-balancing bias, FP8 mixed-precision training, and DualPipe pipelining, they can overcome the cost–performance bottleneck of large LLMs (400B dense ≈ 6.3 M GPU·h, degraded quality) and reach quality close to GPT-4o/Claude-3.5 (MMLU −0.3 pt) at 2.8 M H800 GPU·h (−56%).

Prompt 1.2.1 (identifying novelty)

"Based on the full paper, list the 1–3 most important and original contributions as distinct items. For each, clearly classify it as a new architectural component, a new training technique, a new theoretical insight, a new dataset, or a novel application of an existing method."

TL;DR (3-line core)

1. Aux-loss-free MoE routing achieves load balancing without performance loss, reaching GPT-4o-class quality.
2. The world’s first FP8 mixed-precision training of a 671B-param model drastically cuts memory and communication (2.664 M H800 GPU·h).
3. A Multi-Token Prediction (MTP) objective lifts general benchmark performance and accelerates speculative decoding ×1.8 in TPS.

The 3 most important and original contributions

Summary

On three axes — (1) MoE routing that resolves the load-balancing–performance dilemma, (2) an ultra-large-scale FP8 training scheme, and (3) an MTP objective that captures both performance and speed — DeepSeek-V3 redefines the performance–cost Pareto frontier of open-source LLMs.

Prompt 1.2.2 (strengths from the authors’ perspective)

"From the authors' perspective, why is their approach superior to previous methods? Quote or clearly explain the key arguments they use to support the novelty and strengths of their work."

Two-line summary

• The authors claim that the “Aux-loss-free MoE + FP8 + DualPipe + MTP” combination reaches GPT-4o/Claude-3.5-class performance (e.g., MMLU-Pro 75.9) at −56% training cost vs. a dense 405B (2.788 M H800 GPU·h).
• In other words, the crux of its superiority is being “the first 671B open-source LLM to cut cost, resources, and latency at once without sacrificing performance.”

Conclusion The authors’ central argument is a “system-level innovation that catches all three hares of performance, cost, and scaling.” They stress that DeepSeek-V3’s superiority lies in four designs — the bias-based MoE, FP8 training, DualPipe communication overlap, and the MTP objective — coming together to narrow the performance gap with prior open-source LLMs while drastically cutting cost and resource demands.

Prompt 1.3.1 (step-by-step algorithm explanation)

"Explain the core algorithm, model architecture, or main methodology step by step. Assume the reader is an AI graduate student. In particular, build a very simple, concrete toy example with sample input — such as a simple sentence, a 3x3 pixel image, or a small state space — and walk through each step to show how the input is transformed into the final output. Define every key term and variable the moment it appears."

Key points at a glance (3 lines)

1. Aux-loss-free Load-Balancing Bias — dynamically adjusts each expert’s bias bᵢ to flatten token routing, removing the load-balancing ↔ performance trade-off.
2. In a toy example, only 2 of 3 experts are active (Kʀ = 2), and the whole pipeline for a single token is shown numerically.
3. This strategy trains a 671B-param MoE with 0% token drops and MMLU +0.8 pt, cutting the balance deviation by 38%.

Terminology

Step-by-step algorithm (with a toy example)

Setup hidden state of token x₁: u₁ = [0.6, −0.2], 3 experts (E₁…E₃), Kʀ = 2, initial bias b=[0,0,0].

Repeating this for every token and step levels out expert load without any aux-loss and keeps efficiency intact with no token drops.

Multi-Token Prediction (MTP) summary

At depth 1 (D = 1), predicting “current + 1 token” simultaneously raises data-signal density:

• Flow: main representation h⁰ᵢ → MTP module → h¹ᵢ → cross-entropy on both tokens.
• Evaluation: acceptance rate 85–90%, TPS ×1.8 speedup.

Why does it matter?

➡️ Net result: GPT-4o-level quality at 2.788 M H800 GPU·h (−56%).

Note: actual hyperparameters (summary)

• 256 experts in total, Kʀ = 8, node limit ≤ 4.
• γ = 0.05, sequence-wise balance loss α ≪ 1.
• FP8 + DualPipe pipelining fully overlaps communication and computation.

These toy numbers show how an Aux-loss-free MoE flows from input → expert selection → output and dynamically self-adjusts, lifting both the efficiency and the performance of an ultra-large-scale model at once.

Prompt 1.3.2 (identifying the ‘secret weapon’)

"Identify the single most decisive mathematical formula, algorithm step, or architectural component that enables the paper's core contribution. Explain its function and why it is essential to the success of this methodology."

One-line conclusion

“Aux-loss-free Load-Balancing Bias” — adding a bias $b_i$ to token–expert affinities to steer Top-K routing, and updating it dynamically each step with $b_i \leftarrow b_i \pm \gamma$, so that load balance and performance are achieved simultaneously with no extra aux-loss.

1. The ‘secret weapon’ formula and how it works

• $s_{i,t}=\sigma(u_t^\top e_i)$ : affinity between token $u_t$ and expert $e_i$

• $b_i$ : the routing bias of expert i (updated in real time during training)

• Top-K : selects the number $K_r$ of experts to activate per token

• Update rule : at the end of each step

  $$ b_i \leftarrow \begin{cases} b_i-\gamma, & \text{expert overloaded} \\ b_i+\gamma, & \text{expert underloaded} \end{cases} $$

  ($\gamma$: hyperparameter controlling the bias update speed)

Role: experts with a high $b_i$ become less likely to be selected and those with a low $b_i$ more likely, automatically easing batch-level load deviation. Because gating still runs on the affinity $s_{i,t}$ itself, there is no performance degradation.

2. Why is it essential to DeepSeek-V3’s success?

Key arguments

1. Removes the performance–balance trade-off: improves balance metrics while gaining +0.5 ~ +2 pt across several benchmarks.
2. Ensures stability at scale: bias updates alone complete the entire 14.8T-token run without expert collapse.
3. System efficiency: routing is batch-local, so it minimizes EP64 cross-node communication and overlaps fully with DualPipe.

3. Summary

DeepSeek-V3’s Aux-loss-free Load-Balancing Bias solves the challenge of a “balanced MoE without sacrificing performance” and makes all of the following possible at once:

• GPT-4o/Claude-3.5-class quality,
• training cost below half,
• ultra-large-scale FP8 and 128K context

— the decisive single move (the killer technique).

Prompt 1.3.3 (‘secret weapon’ analysis)

"I understand that Aux-loss-free Load-Balancing Bias is a technique that keeps the experts of an MoE trained in a balanced way. If that's right, could you explain in detail, with an example, how it works? And I'd also like to know what advantages it has over existing methods."

Summary at a glance

• Aux-loss-free Load-Balancing Bias (LBB) adds a real-time bias $b_i$ to each expert’s routing score and adjusts it every step as $b_i \leftarrow b_i \pm \gamma$, leveling batch-wise load without any additional auxiliary loss.
• Thanks to it, DeepSeek-V3 achieves 0% token drops, greater expert specialization, validation loss 2.258 → 2.253, and consistent gains over aux-loss methods across several benchmarks, e.g., BBH +2.0 pt and GSM8K +2.5 pt.

1. How does it work? — a 3-expert toy example

Repeating this moves the bias $b_i$ automatically toward ‘overloaded expert ↓ / underloaded expert ↑’, preventing routing collapse.

2. Key differences from existing auxiliary-loss methods

3. Experimental evidence (summary)

Interpretation: LBB avoids the performance–balance trade-off entirely and even nudges performance up slightly.

4. Why is it better?

1. 0% compute and memory overhead — removing the extra loss backprop saves speed and memory.
2. Minimal hyperparameters — only γ needs tuning, an advantage for large-scale experiments.
3. Enables expert specialization — since only batch-level balance is required, domain-specific experts form naturally.
4. Stable training/inference — no token drops or routing collapse; converges normally even with 256 experts and EP64.
5. Higher performance — consistent gains or parity across several benchmarks and on validation loss.

ⓘ Summary

• That’s right. LBB is a technique that keeps MoE expert load balanced during training.
• The method is a simple loop: “affinity + bias → Top-K routing → bias update”.
• The advantage is that it captures balance, specialization, and performance at once, without the performance sacrifice and tuning hassle of aux-loss methods.

Prompt 1.4.1 (core results analysis)

"Analyze the key results, including the tables and figures in the 'Experiments' or 'Results' section. What are the main performance metrics used? On which benchmark datasets are the results reported? Summarize the headline results the authors emphasize most as evidence of their method's success."

TL;DR — the ’trump card’ the authors play DeepSeek-V3 (671B MoE, 37B active) hits MMLU 88.5 (+0 pt vs. GPT-4o), MMLU-Pro 75.9 (+2.6 pt vs. GPT-4o), Codeforces 51.6 (+26 pt vs. LLaMA-3.1), MATH-500 90.2 (+16.4 pt vs. LLaMA-3.1), and finishes the entire training run in 2.788 M H800 GPU·h — less than half the cost of an existing 400B dense model.

1. Evaluation metrics & setup

All models were evaluated with a max output of 8K tokens under the same prompt formats (simple-eval, Zero-Eval).

2. Representative benchmarks used

• Knowledge & reasoning: MMLU, MMLU-Pro, GPQA-Diamond, BBH, AGIEval
• Long-document understanding: DROP, FRAMES (~100K context), LongBench v2
• Code & engineering: HumanEval-Mul, LiveCodeBench, Codeforces, SWE-Bench-Verified
• Math: MATH-500, AIME-2024, CNMO-2024
• Multilingual: MMMLU, C-Eval, CLUEWSC, etc.

3. Key result highlights

Observations

• Among open-source models, V3 ranks first in every area, with particularly large gaps in code and math.
• vs. closed-source models it is close on knowledge/reasoning metrics (±1 pt) and ahead on some code and math metrics.
• Long-document understanding (91.6 F1) demonstrates 100K-context capability.

4. Efficiency and stability metrics

*Estimated for a 405B dense model, converted to the same hardware generation.

5. ‘Evidence of success’ the authors emphasize

1. Best open-source performance and a smaller gap to closed models: the first to approach or exceed the GPT-4 family on educational knowledge (MMLU-Pro 75.9) and math (MATH-500 90.2), among others.
2. Broad leads in long-document, code, and engineering: wide margins on DROP, LongBench, Codeforces, and SWE-Bench.
3. Redefining the cost-efficiency curve: 2.8 M GPU·h for the full 14.8T-token run — 56% less than an equivalent dense model.
4. Training stability: FP8 + aux-loss-free MoE, zero rollbacks over 14.8T tokens.

Conclusion

DeepSeek-V3’s experimental results show it solved the “triple challenge of performance, cost, and scaling” at once. It makes large gains especially in competitive coding, hard math, and very long-document understanding while cutting GPU usage in half, offering a practical alternative for both academic and industrial research.

Prompt 1.4.2 (critical comparison)

"How does the proposed method perform against the main baselines and state-of-the-art (SOTA) models mentioned in the paper? Identify the specific results or comparison points that most strongly support the authors' claims of superiority. Conversely, look for results where the proposed method failed to beat competing models or where the improvement was marginal. If any exist, what reasons do the authors give for these cases?"

TL;DR — the ’trump card’ the authors play DeepSeek-V3 (671B MoE, 37B active) hits MMLU 88.5 (+0 pt vs. GPT-4o), MMLU-Pro 75.9 (+2.6 pt vs. GPT-4o), Codeforces 51.6 (+26 pt vs. LLaMA-3.1), MATH-500 90.2 (+16.4 pt vs. LLaMA-3.1), and finishes the entire training run in 2.788 M H800 GPU·h — less than half the cost of an existing 400B dense model.

1. Evaluation metrics & setup

All models were evaluated with a max output of 8K tokens under the same prompt formats (simple-eval, Zero-Eval).

2. Representative benchmarks used

• Knowledge & reasoning: MMLU, MMLU-Pro, GPQA-Diamond, BBH, AGIEval
• Long-document understanding: DROP, FRAMES (~100K context), LongBench v2
• Code & engineering: HumanEval-Mul, LiveCodeBench, Codeforces, SWE-Bench-Verified
• Math: MATH-500, AIME-2024, CNMO-2024
• Multilingual: MMMLU, C-Eval, CLUEWSC, etc.

3. Key result highlights

Observations

• Among open-source models, V3 ranks first in every area, with particularly large gaps in code and math.
• vs. closed-source models it is close on knowledge/reasoning metrics (±1 pt) and ahead on some code and math metrics.
• Long-document understanding (91.6 F1) demonstrates 100K-context capability.

4. Efficiency and stability metrics

*Estimated for a 405B dense model, converted to the same hardware generation.

5. ‘Evidence of success’ the authors emphasize

1. Best open-source performance and a smaller gap to closed models: the first to approach or exceed the GPT-4 family on educational knowledge (MMLU-Pro 75.9) and math (MATH-500 90.2), among others.
2. Broad leads in long-document, code, and engineering: wide margins on DROP, LongBench, Codeforces, and SWE-Bench.
3. Redefining the cost-efficiency curve: 2.8 M GPU·h for the full 14.8T-token run — 56% less than an equivalent dense model.
4. Training stability: FP8 + aux-loss-free MoE, zero rollbacks over 14.8T tokens.

Conclusion

DeepSeek-V3’s experimental results show it solved the “triple challenge of performance, cost, and scaling” at once. It makes large gains especially in competitive coding, hard math, and very long-document understanding while cutting GPU usage in half, offering a practical alternative for both academic and industrial research.

Prompt 1.5.1 (stated and potential limitations)

"What limitations, weaknesses, or failure modes do the authors explicitly acknowledge in the paper? Next, based on your analysis of the methodology and results, what potential limitations or weaknesses do you think the authors may not have mentioned? (e.g., reliance on strong assumptions, scalability issues, high computational cost, limits of generalization, potential negative social impact, etc.)"

One-line summary

DeepSeek-V3 excels on both performance and cost as a 671B MoE model, but the authors also acknowledge deployment-side weaknesses — dependence on large-scale hardware, communication/quantization limits, and inference latency — with additional risks lurking in data, safety, and generalization.

1. Limitations explicitly stated by the authors (Section 6 & related text)

2. Potential and informal limitations (additional analysis)

Key takeaway

DeepSeek-V3 pushes the “performance–efficiency curve” far outward, but the large-GPU-stack prerequisite, communication/quantization technical debt, and safety and data governance are the keys to the next stage of research and productization.

Prompt 1.5.2 (future research trajectory)

"What specific future research directions do the authors propose? Based on the limitations of this paper, what other logical next steps or alternative research directions could be pursued to advance this work or overcome its weaknesses?"

Core conclusion (3 lines)

1. The authors set out ① maximizing architectural/system efficiency, ② diversifying data and training signals, ③ expanding ‘deep reasoning’ capability, and ④ diversifying evaluation metrics as future research tasks.
2. This review goes further and proposes six practical follow-ups for low-cost inference, safety, and generality.
3. These research trajectories extend the performance–cost–responsibility triangle at once, amplifying the impact of the DeepSeek-V3 line.

1. What the authors propose (future directions)

2. Additional logical next steps we propose

3. Summary

• The authors’ roadmap aims for “bigger and smarter models” through technical scale-up and broader evaluation.
• The additional proposals target “cheaper, safer, and more widely usable models.”
• If both tracks progress complementarily, the DeepSeek ecosystem stands a strong chance of taking the next step toward a next-generation open-source AGI candidate.

Prompt: data and preprocessing

"What tokenization strategy was used (e.g., BPE, WordPiece, SentencePiece)? How are out-of-vocabulary words handled? Describe the text cleaning and normalization steps."

One-line summary DeepSeek-V3 trains its own 128K-vocab Byte-level BPE tokenizer to improve multilingual compression, and because it operates at the byte level, out-of-vocabulary words effectively do not exist. On the preprocessing side, it uses document packing + Fill-in-Middle (10%), deduplication, and randomly splits special tokens (’newline + punctuation’ bundles) to ease token-boundary bias.

1. Tokenization strategy

“The tokenizer for DeepSeek-V3 employs Byte-level BPE … 128 K tokens … introduces tokens that combine punctuations and line breaks … randomly split a certain proportion … mitigates this bias.”

2. Handling out-of-vocabulary (OOV) words

• Because Byte-level BPE uses UTF-8 bytes (0–255) as its smallest sub-token unit, an unseen word is decomposed automatically: new_word → UTF-8 byte sequence → BPE merges.
• A dedicated <unk> token is therefore almost never used, and every input character is represented losslessly.

3. Text cleaning and normalization pipeline

4. Key advantages summarized

• Language generality — byte-based, so every script that needs more than one byte (including Korean and emoji) is handled losslessly.
• Compression efficiency — the 128K vocab plus punctuation–newline bundles lowers the token/character ratio, losing little context length.
• Training stability — the boundary-bias trick reduces token-split mismatches in few-shot and multi-line inputs.
• Preprocessing consistency — document packing, FIM, and deduplication together secure 14.8T high-quality tokens that underpin overall performance.

Thanks to this tokenizer and preprocessing design, DeepSeek-V3 achieves high compression and strong generalization on long documents, multilingual text, and code even at 671B parameters.

Prompt: model architecture

"If a transformer was used, describe the attention mechanism configuration (e.g., number of heads, number of layers) in detail. How are positional encodings handled? If it is a Seq2Seq model, explain the encoder-decoder interaction."

One-line core summary DeepSeek-V3 combines a 61-layer Transformer decoder with 128-head MLA (Multi-head Latent Attention) + RoPE positional encodings + a 256-expert MoE FFN, activating only 37B parameters per token while handling a 128K context.

1. The full stack at a glance

2. Multi-head Latent Attention (MLA) — a KV-cache-compressing attention

1. Down-projection $c_{KV,t}=W^D_{KV} h_t$ (7168 → 512)
2. Up-projection $[k^C_{t,1};…;k^C_{t,n_h}] = W^U_K c_{KV,t}$ $[v^C_{t,1};…;v^C_{t,n_h}] = W^U_V c_{KV,t}$
3. Apply RoPE $k^R_t = \text{RoPE}(W_{KR} h_t)$ (per-head 64 dim)
4. Per-head KV $k_{t,i}=[k^C_{t,i};k^R_t]$ (128 + 64 = 192 dim/head)

Thanks to the 512-dim compressed latent $c_{KV}$, the KV cache uses only about ×0.27 of the original memory, and RoPE is applied only to the decoupled key, preserving accuracy at long contexts.

3. Positional encodings — Decoupled RoPE

• Method : Rotary Positional Embedding (RoPE) mixed into each head separately at 64 dims.

• Advantages :

  • stable up to 128K tokens together with linear-scale remapping (YaRN);
  • sine/cosine rotation is applied only to decoupled keys/queries, so it does not clash with MLA’s compressed space.

4. Feed-Forward Network — DeepSeekMoE

FFN_i(x) = W₂ σ(W₁ x)  (i = expert id)
h′ = x + Σ_{i∈Top-K} g_i · FFN_i(x)

• routes Top-K = 8 experts per token.
• achieves load balance via the aux-loss-free bias, with 0% token drops.
• each expert uses a 2 048 intermediate dim and FP8 mixed-precision multiplies to minimize compute and communication cost.

5. Architecture highlights

1. MLA cuts the KV cache ≈62% → memory headroom even at a 128K context.
2. MoE + FP8 → 37B active parameters per token; training cost −56% vs. a dense 405B at equal performance.
3. RMSNorm & RoPE → training stability and long-context generalization at the same time.

Not an encoder-decoder: DeepSeek-V3 is a GPT-style decoder-only model, so no separate encoder-decoder interaction exists.

Prompt: training and optimization

"What is the specific language modeling objective (e.g., Causal LM, Masked LM, Prefix LM)? What corpus was the model pre-trained on? Describe the fine-tuning strategy for downstream tasks."

Conclusion first (3-line summary)

1. Objective: DeepSeek-V3 augments a causal LM with a Multi-Token Prediction (MTP) auxiliary loss (predicting the next 2 tokens at once) to densify the learning signal.
2. Pre-training: trained on 14.8T tokens (with a larger share of math and code, plus more languages) from a self-built RefinedWeb++ corpus, after document packing, Fill-in-Middle (10%), and deduplication.
3. Fine-tuning: staged SFT → RL (Constitutional AI) + distillation of DeepSeek-R1 long-chain-of-thought to strengthen both human preference and reasoning ability.

1. Language modeling objective

Takeaway: while keeping the traditional next-token LM, MTP raises both information density and reasoning readiness.

2. Pre-training corpus

3. Downstream fine-tuning strategy

Total fine-tuning cost: 0.124 M GPU·h (SFT + RL + YaRN) for a large jump in reasoning and safety over the base model.

Key takeaways

• The MTP-augmented causal LM doubles the density of the learning signal without any extra loss, buying data efficiency and inference speed at once.
• The 14.8T multi-domain corpus and careful preprocessing underpin generalization across long contexts, multilingual text, and code.
• The three-stage SFT → distillation → RL fine-tuning balances performance, safety, and dialogue quality at an added cost of under 4% of the entire training run.

Prompt: implementation and resources

"What are the key software dependencies (e.g., CUDA, MPI, specific libraries)? What memory footprint is expected during training and inference (VRAM for GPUs, RAM for CPUs)? What throughput is achieved on the stated hardware (e.g., images/sec, tokens/sec)? Provide a detailed breakdown of the total compute cost (e.g., total FLOPs or petaflop-days)."

TL;DR — resource summary

*Approximations assuming FP8 low-precision storage and communication plus ZeRO-1 sharding.

1. Software and library dependencies

Thanks to the ‘HAI-LLM + DualPipe + FP8’ triangle, communication–compute overlap is maximized, achieving a near-“zero-bubble” pipeline even at EP64.

2. Memory usage (estimated)

During training

• Model and optimizer states

  • params in FP8 (1 B) + master FP32 (4 B) ⇒ 5 B/param
  • 37B active → ~185 GB, but ZeRO-1 shards this (64-way) → ~2.9 GB/GPU.

• Optimizer moments BF16 (2 B) → ~74 GB globally → ~1.2 GB/GPU.

• Activation checkpoints FP8 cache + RMSNorm recomputation etc. add ≲ 5 GB/GPU.

Total ≈ 9 GB/GPU (model + optimizer + activations) plus kernel/communication overhead ≲ 80 GB (within the H800 HBM limit).

During inference

• Prefill: TP4 + EP32, FP8 KV cache → about 74 GB/GPU (37B active × 2 B FP16 + marginal cache).

• Decoding: routed across EP320 → ~1 expert per parameter shard, dominated by the KV cache → 60–65 GB/GPU.

• Minimum batch unit

  • Prefill: 32 GPUs (4 nodes × 8)
  • Decode: 320 GPUs (40 nodes)

3. Throughput and efficiency

Cluster-wide: ≈ 3.16 M tok/s (2048 GPUs) → processing 1T tokens takes 180 K GPU·h (2.2× faster than a dense 72B).

4. Total compute (approximate PF-days)

Assuming the H800’s FP8 peak of 989 TFLOPS (on an H100 basis):

2.788 M GPU·h × 0.989 PFLOPS ÷ 24 h ≈ 115 PF-days

≈ 1.3 × 10² PF-days — more than 40% lower than the existing dense 400B class (200+ PF-days).

5. What the key optimizations buy

1. FP8 + fine-grained quantization → 2–4× memory savings on model, activations, and optimizer, plus a 2× speedup.
2. DualPipe scheduler → halves the PP bubble and hides communication latency → higher per-token GPU-FLOP productivity.
3. Custom all-to-all kernels (20 SMs, PTX) → overlapping IB+NVLink = “effectively zero communication cost” for EP64 training.

The result: training the same trillion tokens costs 55% of a dense 72B model (180 K ↔ ~330 K GPU·h), and at inference only 37B active parameters are used to reach LLaMA-3 405B-class quality.

Notes / remarks

• Actual VRAM can vary with batch size, prompt length, and KV-cache settings.
• FLOP figures approximate the H800, whose public specs are unavailable, by assuming parity with H100 FP8 performance.
• All numbers are recomputed from or conservatively estimated from values stated in the DeepSeek-V3 paper.

Prompt: model evaluation and metrics

"What are the main metrics for evaluation: latency, throughput, power consumption, or performance per cost? How does the system scale with more data, users, or compute nodes?"

Conclusion summary (3 lines)

1. DeepSeek-V3 delivers a training efficiency of 180 K GPU·h per 1T token (≈ 3.7 days @2 048 H800), a ×1.8 inference TPS speedup, and GPT-4o-adjacent quality at a total cost of $5.6 M.
2. The key evaluation metrics are training-stage cost-per-performance (GPU·h, $/pt), inference throughput (TPS) and latency, memory footprint, and node-scaling efficiency.
3. Thanks to DualPipe + Node-Limited MoE, even as the number of GPUs grows the compute:communication ratio stays 1:1, guaranteeing an almost linear scale-up.

1. Evaluation metric definitions

2. Key figures (DeepSeek-V3-Base)

3. Scaling characteristics

4. Reading the metrics

1. Training stage — FP8 + MoE lowers FLOPs/token, cutting GPU·h/T by 45%.
2. Inference stage — MTP + speculative decoding reaches an 85–90% acceptance rate for ×1.8 TPS, at ≈ 0.55× average response latency on the same hardware.
3. Performance per cost — delivers MMLU 88.5 and Codeforces 51.6 for $5.6 M, updating the open-source Pareto frontier.
4. Scalability — separating IB ↔ NVLink traffic with a node limit of 4 and EP64 gives N ↑ → throughput ≈ O(N).

5. Summary and implications

• DeepSeek-V3 dominates existing dense LLMs on all three metrics of performance, cost, and scale.
• The key metrics used are GPU·h/Ttok, TPS, VRAM, η_scalability, and $/score, which are easy to reproduce and optimize from a compiler/systems perspective.
• Future optimization tasks: (1) FP4/INT4 low-precision inference to get VRAM below 50 GB, (2) WAN-grade all-to-all kernels, and (3) measuring and reporting TPS per watt.

If you are planning model deployment or research expansion, the priority is to understand DualPipe scheduling and the node-limited MoE router to secure cluster-scale scaling efficiency.