Programmable World Model

Paper

Programmable World Model — Embedding “Executable State” into Generative Worlds

One-Line Summary (TL;DR)

Video world models are producing increasingly realistic interactive environments, but they only remember the pixels that flash across the screen — they have no idea “what state the world is actually in right now.” Alaya Lab’s Programmable World Model (PWM) separates a lightweight engine that manages the state of the world from a generative renderer that paints that state into images. State stays programmable and verifiable, while the fine details of appearance and motion are left to a pretrained video generation model. On its own benchmark, CombatStateBench, it achieves 94% count accuracy and 98% state accuracy, far surpassing existing interactive video world models.

Core Idea

The paper’s central hypothesis can be summarized in a single sentence.

The authors hypothesize that by separating the maintenance and evolution of world state from visual observation generation, generative worlds can overcome the existing limitation of losing state consistency during long-term interaction, while still retaining the flexibility of open-domain image generation.

Behind this sentence lies a simple yet weighty question. Generative models excel at producing “plausible next frames” but are weak at remembering “true facts.” For example, when an NPC falls in a battle scene, the model can draw that frame plausibly, but it cannot guarantee that the NPC must remain dead for the next 10 seconds. “Invisible facts” such as an enemy that has left the screen, unseen health values, inventory, and faction relations cannot be maintained by pixel prediction alone.

PWM’s answer is clear: take the responsibility of maintaining facts away from the generative model and hand it to a separate engine that manages explicit, executable state. The generative model is only responsible for “rendering” that state. It is a division of labor much like a game engine managing game logic and state while a graphics pipeline draws them on screen.

The key link in this division of labor is the state representation. How do you translate the structured state maintained by the engine into a conditioning signal that a generative model can understand? The authors introduce an intermediate representation called the “State-augmented 3D Oriented Bounding Box (OBB).” A deterministic state compiler then projects this representation, aligned to the camera trajectory, into a pixel-aligned control map. The fact that this compiler is an untrained deterministic function is the most important design choice in the paper.

Background: The Problem They Solve

Recent video world models (Genie 3, WorldPlay, AlayaWorld, Matrix-Game 3.0, etc.) are attempts to build interactive world engines on top of generative video models. By predicting the next observation from visual history and user actions, they synthesize increasingly realistic and responsive environments.

But the authors argue that “plausible observation generation” is not enough to make an “interactive world engine,” and they identify three structural limitations.

1. No entity-level control. Existing control interfaces mainly specify camera, actions, or high-level prompts. There is no way to directly point at and manipulate a specific individual entity.
2. No persistent global state. There is no explicit global state that can be accessed and updated independently of the current view. Non-visual information such as off-screen entities, inventory, task progress, and interaction history is essential to a multiplayer world engine, yet it is not maintained.
3. No programmability of world rules. Users cannot define evolution rules such as “when the door opens” or “how an interaction affects other entities.” Prompting for a desired outcome is not the same as establishing an executable rule that consistently governs subsequent interactions.

Looking at related work, there have been two strands of approach to this problem. “Explicit-state world model” approaches such as StatePlay and MASS explicitly predict state, but the state transition itself is predicted by a learned model. As a result, state prediction errors directly lead to incorrect world updates, and errors accumulate over long rollouts. PWM differs in that state transitions are performed not by a learned transition model but by a deterministic engine that executes explicit rules.

Meanwhile, the generative rendering line of work (DiffusionRenderer, AlayaRenderer, Coarse-to-Real, etc.) has shown that realistic images can be synthesized from structured conditions. They opened a spectrum in which “richer structured representation enables more precise control.” PWM picks a midpoint on this spectrum, finding a point where the representation is neither too light nor too heavy.

New Approach: Programmable World Model

PWM’s technical core is the choice of representation. The authors view the choice of representation as determining three things: (1) the extent to which things can be explicitly controlled, (2) the cost of creating and evolving state, and (3) the possibility of mismatch between the representation at training time and the representation at inference time.

The most interesting insight the paper raises here is the training–inference asymmetry. In training data, the structural representation is extracted after the dynamics have already been realized.

At inference, the direction is reversed. The system must first receive a high-level state transition and then actively generate the time-varying structural representation that matches it.

This asymmetry has significant implications. Even a representation that could be obtained at training time is not necessarily as easy to create and evolve at inference time. Specifying “the character falls down” is easy, but generating the character’s detailed body pose and limb trajectories one by one is a high-dimensional dynamics realization problem. The more precise the representation, the more low-level geometry and kinematic details the system must decide for itself.

From this perspective, choosing a representation is drawing the boundary between “explicit structural control” and “generative motion completion.”

The middle ground the authors choose is the state-augmented 3D OBB. Each OBB expresses an entity’s position, size, and orientation in a shared world coordinate system with a small number of fixed parameters, and attaches persistent identity, semantic category, dynamic state, and appearance information. Compared to text or 2D boxes/masks, it provides a skeleton that can be re-projected independently of the view in world space; compared to a G-buffer or a full 3D scene, it confines the structure that must be explicitly evolved to a compact, entity-level space. Fine details such as limb joints, local deformations, and cloth are completed by the generative renderer under this constraint.

The full framework built on this principle consists of four main components.

1. Coding agent (VLM-based). Takes a reference image and natural language description and writes an executable world program. It defines the initial states and attributes of detected entities, their relationships, supported actions, the rules by which actions and events update the world, global constraints, event triggers, and objectives.
2. Lightweight engine. Executes the program to maintain the canonical world state and performs state transitions based on player actions. The engine is the single source of truth.
3. Deterministic state compiler. Projects the updated state under the target camera, converting it into a pixel-aligned spatiotemporal control signal.
4. Generative renderer. Synthesizes the next video chunk conditioned on the control signal and visual history.

How It Works: A Concrete Walkthrough

Let’s first look at the full pipeline at a glance.

Now let’s walk through each step with a very simple example. Imagine a battle scene with two characters, A (player) and B (enemy).

State Representation: The Canonical World State

The canonical world state maintained by the engine has the following structure.

• $\mathcal{E}_t$: persistent entities and their 3D poses (each OBB’s position, size, and orientation for A and B)
• $\mathcal{A}_t$: semantic and functional attributes (A’s health 100, B’s health 20, faction labels, etc.)
• $\mathcal{Q}_t$: relations between entities (A and B are hostile to each other)
• $\mathcal{R}_t$: executable world rules (“an attack is valid only when the target is hostile, alive, and within range”, “a death event fires when health drops to 0 or below”)

Values such as health and faction membership are not directly visible on screen here, but they are latent world facts that determine subsequent state transitions. The engine explicitly stores them.

State Transition: Engine Execution

When the player selects the action $a_t$ “A attacks B”, the engine updates the state according to the rules $\mathcal{R}_t$.

It verifies whether the action is valid (hostile? alive? within range?), and if so, lowers B’s health from 20 to −10. Since health has dropped to 0 or below, a death event fires and B transitions to the dead state. From then on, this state becomes a fact guaranteed by the engine across all subsequent frames. This is the decisive difference from prompt-based approaches. A prompt-based approach only tells the model in text that “B has fallen”, without holding it responsible for continuing to remember that fact.

Control Compilation: From State to Pixel-Aligned Control

The updated state $s_{t+1}$ is still just structured data in the world coordinate system, not yet in a form the renderer can use. The deterministic compiler $P$ projects it under the target camera $C_{t+1}$.

The compiler projects each entity’s OBB under the camera and fills the projected region with three kinds of aligned control maps.

• Identity map $M_t^{\text{id}}$: Permanently assigns each persistent entity to one of $K$ learnable identity slots. Even if an entity is occluded and reappears, or leaves the screen and comes back, the same slot stays linked to an appearance reference, so identity is preserved. At training time, slot assignments are randomized per sample to prevent a particular slot from being biased toward a specific category or viewpoint.
• Semantic map $M_t^{\text{sem}}$: Carries category-level information. The semantic label $y_i$ is embedded with a pretrained text encoder to produce $q_i^{\text{sem}} = E_{\text{text}}(y_i)$, which is broadcast over the visible projected region of the corresponding OBB. Entities of the same category share the same semantic representation but are distinguished by identity slots. This is especially useful for entities absent from the initial observation or lacking an appearance reference.
• Direction map $M_t^{\text{dir}}$: Expresses the object’s own motion in a camera-relative manner. The world-coordinate velocity maintained by the engine is rotated into the target camera coordinate frame and quantized into one of seven states (forward, backward, left, right, stationary, up, down). The key point is that this direction is computed from world-coordinate velocity, not image-space displacement. Even when the camera moves and the projected position changes, a stationary object is still labeled stationary.

This last direction map is PWM’s quiet “secret weapon.” Camera motion and object motion are inextricably mixed on screen; by separating them, the renderer can resolve the ambiguity of “did the camera move, or did the object move?” The authors note that identity swaps and placement errors are observed even in instruction-based video control like LooseControlVideo, and view this disambiguation as key to placement accuracy.

Generative Rendering

The renderer attaches a learnable Structured Spatial ControlNet on top of LingBot-World-v1’s pretrained camera-conditioned video generation backbone. The control map sequence $M_{1:T}^{\text{ctrl}}$ is encoded at the video latent space resolution, and the per-layer control features $r^{(\ell)}$ are added to the hidden states of the corresponding backbone blocks.

At training time, the pretrained main branch (including the camera-conditioning path) is frozen and only the newly introduced ControlNet is optimized. That is, the camera module constrains the global viewpoint trajectory, while the ControlNet constrains where each entity appears under that viewpoint, what identity and semantics it has, and how it moves.

Fixed-window rendering can be written as follows.

For long-horizon generation, the renderer is extended in a chunk-autoregressive manner: bidirectional denoising within a chunk and causality at chunk boundaries. The completed frames of the previous chunk are carried forward through two kinds of memory. Temporal history conveys recent, medium, and long-range latents compressed at multiple scales (the latent of the initial frame $I_0$ is kept as a persistent anchor), while geometry-aligned spatial memory, borrowed from AlayaWorld, lifts the completed RGB frames into world space using estimated depth and camera parameters, restoring previously seen content across large camera rotations or revisits.

One important principle here: these memories are only the renderer’s visual context; they do not determine state transitions. The engine remains the single source of truth, and the compiled control $M_{t+1}^{\text{ctrl}}$ determines what is rendered in each chunk.

Training Data: An Automatic Data Pipeline

Learning this structured control requires annotations of camera geometry, persistent identity, semantics, and object motion — none of which exist in real-world videos. The authors build a data engine that automatically recovers them from unlabeled gameplay videos (Cyberpunk 2077, Forza Horizon 6, GTA V).

1. ViPE estimates camera intrinsics, pose, and metric depth to establish a shared world coordinate system.
2. A Qwen3-VL agent discovers the scene’s discrete, countable object categories.
3. SAM3 performs video instance segmentation and tracking to produce temporally consistent masks and persistent track IDs.
4. WildDet3D estimates per-frame 3D OBBs from RGB, depth, and intrinsics, converts them into the world coordinate system, and links them into per-track trajectories.
5. Object motion is computed as the world-coordinate box-center displacement between consecutive frames, removing camera-induced displacement, then rotated into the camera coordinate frame and quantized into seven directions.
6. Finally, the 3D OBBs are projected and rasterized to generate the identity, semantic, and direction control maps (occlusion resolved with a z-buffer).

The reason this pipeline matters is that it provides what the paper argues is a “path to scaling training data.” The bottleneck in training structured world models has always been annotation cost; by automating it, the range of worlds that can be supported can be expanded.

Performance Evaluation: Key Results

The authors build a controlled benchmark called CombatStateBench. It consists of 50 clips and measures how well generated videos match the world state maintained by the engine in battle scenarios that mix various camera and entity motions. It also includes interactions with off-screen entities that are absent from the first frame and appear later. Each sequence has synchronized 3D boxes, entity states, camera parameters, projected controls, and instance masks; an automatic verifier checks reprojection, depth, geometry, temporal continuity, and state-transition persistence, and only sequences that pass are used for evaluation.

Evaluation uses two permissive global metrics. Because existing world models do not expose instance-level control, a fair comparison via object-level correspondence or box alignment is impossible.

• Count Accuracy: Eight frames are randomly sampled from each video, and a VLM counts the number of visible surviving characters, compared against the engine’s record. $\text{CountAcc} = \frac{1}{N}\sum_{i=1}^{N}\mathbf{1}[\hat{n}_i = n_i^{\text{eng}}]$
• State Accuracy: Three frames are sampled after each death event recorded by the engine, and the VLM determines whether the dead character is visually represented.

The judge is the Qwen3.6-27B VLM, which is shown only RGB frames without privileged information such as GT boxes or identities. Baselines are two representative interactive video world models, LingBot-World-V2 and YUME, which convey state transitions through prompt switching ({environment description}. {action description}).

The results are shown in Table 1.

Three observations stand out.

First, there is an overwhelming gap in state consistency. Count Accuracy is 94.00, far ahead of LingBot-World-V2 (+53.25pp) and YUME (+62.00pp), and State Accuracy is 98.00, ahead of LingBot-World-V2 (+90.00pp) and YUME (+40.00pp). In particular, LingBot-World-V2’s State Accuracy of 8.00 shows that prompt-based approaches effectively fail to guarantee the visual realization of death events.

Second, the improvement is large even though these metrics are permissive and do not directly reward “instance-level correspondence.” As the authors emphasize, this means explicit engine state is a far more reliable control than prompt switching.

Third, image quality was not sacrificed. Ours records the best scores on all four VBench metrics. Subject Consistency is 94.74, 2.39pp higher than YUME, indicating stronger preservation of entity appearance and structure; Background Consistency (96.98), Temporal Stability (99.00), and Imaging Quality (67.62) are also the highest. The interpretation is that structured control improves entity and background consistency while maintaining frame quality and temporal stability.

Qualitative results tell the same story. The figure below compares death events under two conditions: a static camera (a) and a dynamic camera with moving entities (b).

The baselines produce videos that respond plausibly to the action prompt but do not reliably maintain the resulting state. A character that should survive disappears, a dead character keeps moving, and the number of visible characters changes without any engine event. PWM reflects the engine-maintained state more faithfully while preserving the visual realism of the underlying video generator.

In addition, the authors demonstrate (a) generalization to a new minotaur scene generated with GPT Image 2, (b) correctly bringing in three characters that were behind the first frame during a large-angle camera rotation, (c) extension to a different genre, a racing game, (d) simultaneous rendering of heterogeneous categories such as humans and vehicles, and (e) scenes where NPCs gradually appear over a 897-frame autoregressive sequence.

Our Perspective: Strengths, Limitations, and Why This Work Matters

Strengths

The biggest strength is clarity of design and separation of responsibilities. The division of labor — “the engine handles state, the generative model handles appearance” — is conceptually simple, but it breaks head-on through the problem that world models have suffered by implicitly squeezing state-maintenance responsibility into a pixel-prediction objective. By not asking the generative model to “remember true facts”, it elevates state consistency from a probabilistic guarantee to a deterministic one.

The deterministic state compiler is particularly noteworthy. Because it is a deterministic function rather than a learned mapping, there is no room for hallucination in the state-to-control transformation. When the engine says “B is dead”, the compiler always produces the corresponding control. This leads to verifiability of state — in contrast to StatePlay and MASS, which use learned transition models and are exposed to the accumulation of state prediction errors.

The third strength is that the representation choice is theoretically justified. By making explicit the directional asymmetry — “training goes from realized dynamics to structure; inference goes from state transition to structure to generation” — it convincingly explains why a finer 3D representation is not always the better answer. This is not merely an engineering choice but a reusable framework for representation research. In particular, the design in which the direction map is computed from world-coordinate velocity is a concrete and elegant solution to the ambiguity between camera motion and object motion.

Finally, the automatic data pipeline offers a practical path to scaling. The real bottleneck of structured world models is annotation, and automating it with ViPE+Qwen3-VL+SAM3+WildDet3D lays the foundation for scaling up data in the future.

Limitations and Critical View

Judging only from the evidence the paper discloses, several limitations stand out.

First, the evaluation metrics are too permissive and coarse. As the authors themselves acknowledge, Count Accuracy and State Accuracy measure only “coarse, globally observable attributes” and do not directly reward the instance-level correspondence that is the core contribution of this work. “How many are visible” and “is even one dead person visible” do not adequately demonstrate the value of the fine-grained identity and position control that OBBs provide. Moreover, the VLM judge itself (Qwen3.6-27B) can have errors, adding further noise to the reliability of the metrics.

Second, the absence of ablations is disappointing. The main text does not show an ablation table that isolates the contribution of each core component (e.g., removing the direction map, deterministic compiler vs. learned mapping, presence or absence of the identity map). The mechanism explanation of the direction map, the “secret weapon”, is convincing, but how much it actually contributes to performance is not quantitatively supported. As a reader, you want to know “how many points Count Acc drops if you remove the direction map.”

Third, the benchmark scope is narrow. It consists of 50 clips, 3 games, and battle-centric scenarios. “Death” is an extreme, discrete state transition that suits this representation well, but continuous, subtle interactions such as gestures, facial expressions, and delicate physical contact are hard to express with OBBs. The coarser the representation, the more such details fall to the generative model, and the larger that share, the narrower the scope of state-consistency guarantees. The “generalization across diverse domains” claimed in the paper’s conclusion is represented by a single racing scene, and more systematic domain-extension experiments are lacking.

Fourth, the entire system depends on off-the-shelf models. If the initial 3D layout recovery (ViPE, WildDet3D) or the world-program writing (VLM agent) goes wrong, the engine — however deterministic — merely faithfully maintains a wrong state. The claim that “the engine is the source of truth” is silent about how that truth is originally produced. If the VLM writes incorrect rules, the system consistently renders an incorrect world.

Fifth, real-time performance and compute cost are not addressed. It is unclear whether chunk-autoregression plus spatial memory plus the 3D annotation pipeline can together meet the latency required for real interactive play. The paper focuses on accuracy but does not report throughput and latency metrics, which matter for the practicality of world models.

Why This Work Matters

Even so, this paper matters because it provides a clear reference point that moves the research direction of world models from “pixel prediction” to “executable state.” One sentence in the conclusion summarizes this direction: “Generative worlds should not rely solely on visual generation to remember what is true.” State must be executable and verifiable; only appearance needs to be generative.

Beyond a mere framework proposal, it offers a reusable problem formulation: the “trade-off between representational granularity and training–inference consistency.” At the point where the two currents — generative rendering and explicit state modeling — meet, PWM shows that they can be combined complementarily.

What’s Next?: The Road Ahead

Combining the directions the paper explicitly proposes with reasonable follow-up steps implied by the limitations above, we arrive at the following.

1. Stricter instance-level evaluation. Metrics that measure “who is where and in what state” rather than “how many” — such as identity preservation rate, box-alignment accuracy, and state-transition localization — are needed. In particular, only a metric that quantifies the value of the instance correspondence provided by the deterministic compiler can fully prove the superiority of this approach.
2. Ablations and error analysis. Isolate the contributions of the direction map, identity map, semantic map, and deterministic vs. learned compiler, and systematically analyze failure cases (e.g., subtle interactions, abrupt occlusion, propagation of initial 3D recovery errors).
3. Extending the representation spectrum. An adaptive representation that selectively activates finer hierarchical representations (articulated skeletons within an entity, hand- and face-level local structures) according to the needs of the state transition is the natural next step. An OBB suffices for a transition like “falling”, while a finer representation is needed for something like a “handshake.”
4. Robustifying initial state construction. Research is needed on verifying the correctness of the world programs written by the VLM agent, robustness against off-the-shelf 3D recovery errors, and improving the quality of the loop in which users modify rules during execution.
5. Real-time performance and scale. Measure latency and throughput, extend the data pipeline to more game genres and real-world footage, and show how far the promise of an “open-ended programmable world” actually holds.
6. Extension to multiplayer. Given the paper’s point that off-screen entities, non-visual attributes, and relations are essential to a multiplayer world engine, scenarios in which multiple players jointly manipulate a shared canonical state are the most natural application of this framework.

Ultimately, the question PWM poses is simple: let the generative model draw the world, but keep what the world is in a structure we know. How far this division of labor can extend is the most intriguing open question this paper leaves behind.

Tables from the paper

Tables converted mechanically from the arXiv e-print LaTeX source. The numbers are the paper’s own and did not pass through a model.

Table 1. Video quality and world-state evaluation on 50 CombatStateBench clips. All values are reported as percentages. Count Accuracy is evaluated over 400 sampled frames (eight per clip), and State Accuracy over 50 death events, with three post-transition frames sampled per event.

Figures in this post are taken from the original arXiv:2609.10540 (CC BY 4.0). Only size and format were changed.