Chain-of-Thought on Jaehun's Blog

Language Models Can Control Their Own Attention

Mon, 07 Sep 2026 00:00:00 +0900

Paper

Language Models Control Their Own Attention: Declarative Attention

TL;DR — Existing sparse attention methods still paid $O(N)$ per decoding step by approximating externally which tokens matter. This paper flips the direction: it lets the model “declare” directly, inside Chain-of-Thought (CoT), where it will look. The inference engine parses this declaration like a tool call and skips most KV cache reads. Without any training (zero-shot), Gemma-4-31B cuts decoding attention cost by 52.0% with only a 1.27pp accuracy drop (source: §1, §5.1).

[Paper Review] Qwen 3 Technical Report

Sat, 26 Jul 2025 00:00:00 +0900

Paper Link

Qwen 3: The Evolution of a Giant MoE Language Model with Adjustable Reasoning Depth

TL;DR (in one line)

Qwen 3 couples a user-controllable Thinking Budget with a 128-expert MoE backbone, achieving closed-source–level performance and multilingual breadth while activating only 22 B parameters out of a 235 B total.

DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition

Tue, 08 Jul 2025 00:00:00 +0900

Paper

Subgoal Curriculum + CoT Consistency: DeepSeek-Prover-V2 Reshapes Automated Theorem Proving

TL;DR

DeepSeek-Prover-V2, built on the principle of “breaking a problem into small pieces and then matching them all the way through,” achieves a new SOTA on MiniF2F Pass@32 of 82.4% (671B) and 75.6% (7B) even in a small 7B model. The key is the combination of Subgoal-guided Curriculum and Chain-of-Thought-Lean consistency reward (GRPO).

Code I/O: Condensing Reasoning Patterns via Code Input-Output Prediction

Mon, 07 Jul 2025 00:00:00 +0900

Paper

CODE I/O: From Code I/O + Natural-Language CoT to General-Purpose Reasoning — Lifting 7B-30B LLMs by +2 Points on Average with Data Design Alone

TL;DR

A single data pipeline of “code function → input/output prediction + structured Chain-of-Thought (CoT)” lets 3.5 M samples deliver larger and more balanced gains (+2.9 points) than 14 M-scale SOTA data. By nailing verifiability, low cost, and diversity at once, CODE I/O empirically demonstrates that “data quality > data quantity.”