[Paper Review] Towards Efficient and Practical GPU Multitasking in the Era of LLM

Thu, 09 Oct 2025 00:00:00 +0900

Paper

“Let’s put an OS on the GPU”: A proposal for a GPU multitasking OS layer for the LLM era

One-line summary (TL;DR)

In LLM serving, GPUs often stay at ≤10% utilization (field observation), and load fluctuates by 3× within minutes. The authors propose a “GPU multitasking OS layer” that combines kernel-grain time/space sharing + memory multitasking based on CUDA virtual memory (2 MB) + Guaranteed/Preemptible resource coordination, and argue that this is the only realistic solution to simultaneously achieve utilization, performance guarantees, isolation, and large-scale deployment.

Kubernetes DRA on Jaehun's Blog

[Paper Review] Towards Efficient and Practical GPU Multitasking in the Era of LLM

“Let’s put an OS on the GPU”: A proposal for a GPU multitasking OS layer for the LLM era

One-line summary (TL;DR)