TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models

Paper

1. Introduction: every new view reads the entire cache

Among models that generate video from multiple photographs at a new angle, the highest-quality design stores scene information in a Transformer’s key-value cache. The price of this design is that every new view requires re-reading the entire cache, so rendering slows down as the number of input photographs grows. This paper proposes a method that bakes that information once into the weights of a small neural network, making the cost of rendering a new view independent of the number of input photographs.

2. Background: what is changed, and why

2.1 The task, and the criterion that divides the cost

Novel view synthesis is the problem of regressing RGB for an arbitrary new camera position given the camera poses and intrinsics of $V$ context images. Such models split into two stages. Reconstruction (Recon) turns the context into a scene representation $S$, and rendering (Rend) draws a new view conditioned on $S$.

The key question is how the cost is divided between these two stages. Reconstruction happens once per scene, so it can be expensive; rendering happens repeatedly, so it must be cheap. The choice of representation for $S$ determines this division directly.

2.2 The KV cache of decoder-only LVSM and DVSM

LVSM (Large View Synthesis Model) is a family of models that regress new views from a latent representation alone, without constructing 3D geometry. The decoder-only variant uses a single Transformer decoder for both reconstruction and rendering. DVSM adds a KV cache to the cross-view attention layers. In other words, the scene representation $S$ is the cache itself.

Decoder blocks stack in the order of Eq. 3: intra-view attention, FFN, cross-view attention, FFN. During reconstruction, context tokens are stored as the keys and values of each cross-view attention; during rendering, new-view tokens read those keys and values. Here $V$ is the number of context views and $L$ is the number of tokens per view.

The problem arises here. A single new-view token reads $V \times L$ keys and values at every layer. Rendering cost per token is therefore linear in $V$, and cache size also grows linearly in $V$. At 960p with 64 views, DVSM drops to 1 FPS (Table 1, PDF p.7). This cost structure is summarized in Table 8 (PDF p.20). DVSM has reconstruction $O(V^2L^2)$, rendering $O(V \cdot L)$, and the scene representation is the KV cache.

2.3 fast-weight MLP and LaCT: speed solved, image quality lost

An alternative that removes the cache entirely is the Test-Time Training (TTT) layer. A fast-weight MLP is a small MLP that normally uses fixed weights, but its weights are updated directly by a few steps of gradient descent per scene so that it memorizes the context. The update rule is Eq. 6a.

$W_v X$ is the value projection of each token. Using this value as the target makes the MLP close to a linear write that memorizes each view’s own token values. Rendering only requires a single pass through the learned weights, giving $O(1)$ independent of the number of views. LaCT is a model that replaces cross-view attention in this way, and the literature [36] interprets this design as a variant of linear attention.

The problem is image quality. At 960p with 64 views, LaCT reaches 28.30 dB and DVSM 31.11 dB, a gap of 2.81 dB (Table 1, PDF p.7). Fine-tuning from DVSM weights still leaves the gap (Sec. 3.3, PDF p.5).

3. A small example: keep weights instead of a cache

The example in this section is a reduced setting for explanation. It is not a value from the paper’s experiments. The actual model uses 768-dimensional latents, 12 blocks, and patch size 8; here we shrink to $V=4$, 4 tokens per view, and latent dimension 4 to observe only the change in state.

DVSM approach

• Reconstruction: encode the RGB and poses of 4 context views into 16 tokens. The cross-view attention of each layer stores these 16 tokens as 16 keys and 16 values. → Retained state: 16 keys and 16 values per layer.
• Rendering: 4 new-view tokens read all 16 keys at each layer and take a weighted sum of the 16 values. → $4 \times 16 = 64$ key comparisons per layer. Retained state: none. Output: RGB patch.

TTB approach

• Reconstruction: duplicate the 4 context views into two copies. 16 ground-truth tokens carrying RGB and pose, and 16 rendering tokens carrying pose only. At each layer, cross-view attention computes the output $O$ for the ground-truth tokens, and the MLP takes the rendering tokens as input and $(O - X^{(\text{ctx-rend})})$ as the residual target, training for 3 steps. → Retained state: per-layer fast weight $\theta$. Not stored: the key-value cache.
• Rendering: 4 new-view tokens pass through the same MLP. → 0 key comparisons, no cache reads, no retained state. Output: RGB patch.

[Reconstruction] once per scene
  4 context views
    ├─ 16 RGB+pose tokens ──cross-view attention──► O ──┐
    │                                                  │ match with MSE
    └─ 16 pose-only tokens ──W_k──► f_θ ─► Y' ─────────┘
                                                       │
                                      Retained state: per-layer fast weight θ
                                      Not stored: KV cache (16 keys·16 values)

[Rendering] per new view
  4 new-view tokens ──W_k──► f_θ ─► Y' ─► RGB 2×2 patch
                   (0 key comparisons, no cache reads)

[Reconstruction] once per scene
  4 context views
    ├─ 16 RGB+pose tokens ──cross-view attention──► O ──┐
    │                                                  │ match with MSE
    └─ 16 pose-only tokens ──W_k──► f_θ ─► Y' ─────────┘
                                                       │
                                      Retained state: per-layer fast weight θ
                                      Not stored: KV cache (16 keys·16 values)

[Rendering] per new view
  4 new-view tokens ──W_k──► f_θ ─► Y' ─► RGB 2×2 patch
                   (0 key comparisons, no cache reads)

The difference between the two paths comes down to what is retained. DVSM retains arrays of keys and values and re-reads them each time. TTB retains only the MLP weights, and retrieves the same information with a single pass through those weights. The size of the retained state becomes independent of the number of views.

4. Method

4.1 Bake the output of attention, not replace attention

TTB updates the weights $\theta$ of a fast-weight MLP per scene, but changes the training target.

Here $g$ is cross-view attention, $X^*$ is the context token $X^{(\text{ctx-gt})}$ that includes RGB, and the input $X$ is $X^{(\text{ctx-rend})}$, which has pose only. TTT can be seen as the special case where $X^*=X$ and $g$ is a linear projection.

The reason for changing the target is that the output of cross-view attention already mixes all context views. Using that output as the target means the input-output pairs the MLP memorizes inherit information from the entire scene rather than from a single view. Instead of removing attention, the method uses attention’s product as training material.

The paper verifies this with input-gradient sensitivity. In block 7 of TTB with $V=16$, it measures how much each context image contributes to the output of the new-view rendering token, sorts by output position, and averages (Fig. 4, PDF p.6). With the $W_vX$ target, contribution concentrates on the top-1 view and then falls off quickly; with the cross-view attention target, contribution remains on mid-ranked views as well.

Complexity splits. Rendering is a single MLP pass as in TTT, so it is $O(1)$ in $V$. Reconstruction, by contrast, contains cross-view attention inside the inner loop, making it $O(V^2L^2)$ (Table 8, PDF p.20). Context pairing increases the number of reconstruction tokens by 2x and adds an inner loop, so reconstruction costs more than DVSM (at $V=64$ in Table 1, reconstruction goes from 60 s to 65 s; Sec. 3.3.2, PDF p.6). To keep reconstruction linear, use TTB–, which changes the target to the linear projection $W_v X^{(\text{ctx-gt})}$. Reconstruction is $O(VL)$, but image quality is slightly lower.

Here $O(1)$ refers to complexity with respect to the number of input views $V$. Per-token cost is proportional to MLP width and the number of blocks. It means that rendering cost does not grow as $V$ grows, not that rendering is free.

4.2 Match the modality of reconstruction input and rendering input

TTB’s second problem is the input distribution. During reconstruction the MLP sees context tokens that carry RGB and pose together, while during rendering the new-view tokens carry pose only. Training the same MLP on inputs with photographs and querying it with pose-only inputs forces the MLP to do view interpolation and modality generalization at the same time. This is what the authors call the reconstruction–rendering modality mismatch (Sec. 3.3.2, PDF p.6).

The fix is to split each context view into two copies.

TTB takes $X^{(\text{ctx-rend})}$ as input and $\mathrm{CrossAttn}(X^{(\text{ctx-gt})})$ as target. The new-view token $X^{(\text{nv})}$ also carries pose only, so the rendering input and the reconstruction input now share the same modality.

Without this pairing, distribution alignment is delayed. Fig. 2 (PDF p.2) shows t-SNE plots of the three variants. What the source text confirms is that only (c) aligns the two token streams from the first block (the color mapping is not described in the text), while (b) aligns only in later blocks.

Original Fig. 2 (PDF p.2). (a) is TTT, which memorizes value tokens directly; (b) is TTB, which uses cross-view attention output as the target but without context pairing; (c) is the final TTB with input and target modalities matched. The source shows, in the t-SNE panels, when the two token distributions overlap.

4.3 Detailed design of the TTB layer: the inner loop

The inner loop uses an MSE loss, $T=3$ steps, the Muon optimizer, and weight normalization. The target is the residual after subtracting the input, not an absolute value.

The actual computation order in Algo. 1 is as follows. At each layer, compute $O = \mathrm{CrossAttn}(X^{(\text{ctx-gt})})$, take $Y = W_v(O - X^{(\text{ctx-rend})})$ as the target and $X = W_k X^{(\text{ctx-rend})}$ as the input, and obtain $\theta^{(T)}$. The output $Y'$ of the baked MLP then sends the two streams ($X^{(\text{ctx-rend})}$, $X^{(\text{nv})}$) to the next layer, and is added to $X^{(\text{ctx-gt})}$ as an auxiliary signal. Having all three streams pass through the same operator in the next layer is this feedback path.

Original Fig. 3 (PDF p.4). In reconstruction, each context view is duplicated into an RGB+pose token and a pose-only token, and the TTB layer updates the MLP so that the latter matches the cross-view attention output of the former. In rendering, new-view tokens pass through the same baked MLP to read scene information. Following the arrows shows which tokens carry on to the next layer and which outputs feed back as context tokens.

The contribution of the two design choices is broken down in a table. Using the residual as target reduces the magnitude of the value to regress, making it easier to fit within a few steps. Adding feedback makes the three token streams pass through the same operator in the next layer. Each alone raises PSNR by +0.44 dB and +0.43 dB respectively, and together they give +0.52 dB (Table 6, PDF p.9, 720p ablation).

The loss and step count also matter. Replacing TTT’s unnormalized inner-product loss with MSE raises 23.12 dB at $T=1$ with the inner product to 23.39 dB at $T=3$ with MSE, a gain of 0.27 dB, in the same TTT structure. In TTB– it rises from 23.78 to 24.22 dB, a further 0.44 dB. Going to $T=5$ gives 24.29 dB, only 0.07 dB above $T=3$, so the authors choose $T=3$ as the balance point between cost and quality (Table 5, PDF p.9).

The layer consists of a SwiGLU fast-weight MLP with a single head, hidden dimension 3072 (4x the model dimension), a linear projection sharing query and key, and an additional linear head that predicts a per-token learning rate (base value 0.01) (Sec. 4.1, PDF p.7; App. A.2, PDF p.15).

Training starts from a DVSM checkpoint. Patch embedding, input LayerNorm, Plücker encoding, intra-view attention, both FFNs per block, and the RGB head are carried over as is, while modules inside the TTB layer are freshly initialized. DVSM’s cross-view attention projections are not reused. The TTB output projection is initialized with a small standard deviation (residual zero-init) so that the TTB block behaves close to identity at fine-tuning step 0. This initialization and parameter inheritance contribute +1.02 dB of pretraining benefit (App. A.2, PDF p.15; Table 13, PDF p.22).

Memory is not small. Keeping the activations of the $T$-step inner loop for backpropagation is TTB’s dominant memory cost. Two levels of gradient checkpointing make 64×A100·960p·$V=32$ training feasible, with a training peak memory of 76 GB (App. A.2, PDF p.17; Table 14, PDF p.23).

4.4 local-duplicate inference to widen fast-weight capacity

The capacity of the fast-weight MLP is fixed, so quality saturates as $V$ grows. The paper works around this at inference time. It runs k-means on a 6-dimensional descriptor summarizing the camera pose (camera origin and unit optical axis), groups context views into $K=V/16$ spatially local groups, reconstructs each group independently, and obtains per-group, per-layer fast weights. At rendering, a new view is routed to the nearest centroid group and queries only that copy. Per-token rendering cost is unchanged, and switching between copies is a hard switch without blending (Sec. 3.3.4, PDF p.7; App. A.6, PDF p.18).

5. Results

5.1 Experimental setup

Training is on DL3DV-10K. At training, 16 or 32 context views are sampled at random per scene, and the same number of target views is sampled. Evaluation is on 140 held-out DL3DV test scenes. Resolution is 960p, and evaluation input views are 16/32/64. Metrics are PSNR, SSIM, LPIPS, per-scene reconstruction time, and new-view rendering FPS. Model sizes are TTB 271M, TTB– 244M, DVSM 163M, and LaCT 379M (24 blocks); TTB uses the same 12-block, 768-dimensional structure as DVSM (Sec. 4.1, PDF p.7; Table 3, PDF p.8; Table 9, PDF p.20).

5.2 DL3DV 960p: the quality-speed trade

Each cell is PSNR(dB) / scene reconstruction time(seconds) / rendering FPS (Table 1, PDF p.7). Reconstruction is once per scene, and FPS is new-view rendering. TTB(dup) uses $K=V/16$ fast-weight copies at $V=32/64$.

At $V=64$, TTB gives up 0.91 dB of PSNR versus DVSM and gains 23x in rendering. Against LaCT it is 1.90 dB higher in PSNR and 2.3x faster in rendering, but reconstruction takes 65 s, about 9x longer than LaCT’s 7.3 s. TTB– is 29.29 dB·23 FPS·7.5 s, 0.99 dB above LaCT with comparable reconstruction.

The quality gap varies with $V$. At $V=16$, TTB at 27.00 dB is slightly above DVSM at 26.86 dB; at $V=32$ it is 28.92 vs 29.13 dB, and at $V=64$ it widens to 30.20 vs 31.11 dB. “Nearly maintained” does not mean the same thing at every $V$. At $V=64$, SSIM/LPIPS are also slightly lower for TTB at 0.897/0.131 versus DVSM at 0.920/0.104. TTB(dup) reaches 31.33 dB and 0.919/0.108, comparable to DVSM, but with 4 copies and reconstruction time unchanged at 65 s.

The conditions behind the 23x figure also need to be stated. It is the ratio at $V=64$, 23 FPS against DVSM’s 1 FPS. At $V=16$ it is about 5.8x, from 4 to 23 FPS, and at $V=32$ about 11.5x, from 2 to 23 FPS. TTB’s rendering FPS stays constant at 23 as $V$ grows. However, Table 1 does not state the measurement GPU, so one cannot generalize that the same FPS would appear on other GPUs (not reported in the source).

5.3 Contribution of each design element

The 720p ablation breaks the path from TTT to TTB into three elements.

This is the $V=32$ setting of Table 4 (PDF p.9). Stacking all three elements narrows the quality gap to DVSM while keeping TTT-level rendering speed (about 50 FPS). In exchange, reconstruction time grows quadratically in $V$ because of cross-view attention in the inner loop. This table is a smaller setting with 8×A100 and patch size 16, so it should not be compared directly with 960p Table 1.

5.4 Generalization beyond training scenes

The 960p model trained on DL3DV is applied zero-shot to MipNeRF360, Free, and Hike (64 views, Table 3, PDF p.8).

Single TTB is 0.27 dB below DVSM on MipNeRF360 and higher on Free and Hike. With copies, it exceeds DVSM on all three datasets. The same trend continues at 16 views (Table 10, PDF p.21): TTB reaches 22.03 dB on MipNeRF360, above DVSM’s 21.51 dB, with FPS still 23 versus DVSM’s 4. Under the 512-pixel long-side condition, LagerNVS is 7 FPS and TTB is 67 FPS, with PSNR 23.62, 2.64 dB above 20.98 (Table 3, PDF p.8).

5.5 How many new views are rendered

Looking at FPS alone, TTB always appears to win, but adding reconstruction time changes the answer depending on the number of new views. Table 15 (PDF p.23) gives end-to-end time as reconstruction time plus $N$/FPS. At $V=64$, with 1 new view, DVSM is 61 s and TTB 65 s. At 10 views it is 70 s vs 65.4 s, and at 100 views 160 s vs 69.3 s, with TTB ahead. The paper reports that TTB overtakes after 5–7 views at all $V$. TTB– is ahead from the first view.

5.6 Scene state size and actual memory

Original Fig. 7 (PDF p.24). The horizontal axis is the number of input views $V$, and the vertical axis is scene representation size (MB). DVSM’s KV cache grows linearly in $V$, while TTB’s fast weights stay constant regardless of $V$. TTB(dup) grows with the number of copies $K=V/16$ but is 21x smaller than the KV cache at the same $V$, and the $V=256$ region where DVSM goes OOM at 80GB is extrapolated as empty dots.

TTB’s fast weights total 324 MB in FP32 across all blocks and remain constant regardless of $V$. DVSM’s KV cache grows linearly in $V$, reaching an 85x difference at $V=64$ and 340x at $V=256$. This 324 MB should not be read as an overall GPU memory saving. At $V=32$, 544×960, TTB’s inference peak memory is 18 GB, only slightly below DVSM’s 20 GB (Table 14, PDF p.23). What shrinks is the scene state, and peak memory retains a portion not explained by state size alone. In the same table, training peak is 76 GB for TTB and 65 GB for DVSM, and training time per iteration is 90 s for TTB and 133 s for DVSM.

5.7 Qualitative comparison

Original Fig. 5 (PDF p.7). Each column shows ground truth, DVSM, LaCT, LVSM, TTB–, and TTB. Zooming in on thin, repeating structures such as railings, window bars, and sign lettering, LaCT and LVSM blur the boundaries while TTB retains detail similar to DVSM. This is the material for checking high-frequency preservation differences that PSNR tables alone do not show.

5.8 Number of views and the two-view condition

Two-view Re10K is not the main target. At 512×512, TTB is 31.28 dB and DVSM 31.17 dB, similar in quality, but the cross-attention cost itself is small, so the rendering gain is limited to 64→72 FPS at 256×256 and 45→52 FPS at 512×512 (App. C.1, PDF p.21). TTB’s gain comes from conditions with many views.

6. Limitations and deployment judgment

6.1 Limitations acknowledged by the authors

The authors acknowledge four limitations (Sec. 4.3, PDF p.9). First, the capacity of the fast-weight MLP is fixed, so quality saturates as $V$ grows. In the 720p ablation, the single MLP stalls near 25.8 dB after $V=64$ and is 25.78 dB even at $V=256$. In the same table, DVSM keeps rising, from 27.20 dB at $V=128$ to 27.56 dB at $V=256$ (Table 11, PDF p.22). The local-duplicate variant exceeds DVSM at $V=256$ with 27.74 dB, but reconstruction time is 52.1 s, longer than the single MLP’s 20.5 s and DVSM’s 18.1 s. In the same table, rendering FPS stays in the 67–72 range while DVSM falls from 63.6 to 12.6.

Second, the local-duplicate inference that mitigates this is a heuristic. It splits into $K=V/16$ copies and routes to the nearest group centroid, and the routing is not learned by training (App. A.6, PDF p.18). Third, cross-view attention in the inner loop makes reconstruction quadratic in $V$, which is disadvantageous for scenarios that render a scene only once. Fourth, the model does not generate; it interpolates between context views. It does not extrapolate to unobserved views.

A table also shows widening the fast-weight MLP instead of increasing the number of copies. At $V=64$, 340p, widening from 4x to 16x raises PSNR from 25.36 to 25.61 dB, catching up to DVSM’s 25.60 dB. However, this requires retraining, so the authors keep the 4x width and widen capacity through $K$ (Table 12, PDF p.22).

6.2 The reviewer’s inference

From here on is the reviewer’s judgment. First, the phrase “nearly the same quality” should be read differently depending on $V$. The tables have no seeds or confidence intervals, and the 0.91 dB gap at $V=64$ is not small. Even where PSNR is similar, SSIM/LPIPS are lower for TTB. Second, TTB’s absolute quality leans heavily on the DVSM checkpoint. TTB trained from scratch is 23.82 dB, 1.02 dB below pretrained TTB at 24.84 dB (Table 13, PDF p.22). However, since from-scratch TTB at 23.82 dB is above pretrained TTT at 23.12 dB in the same table, the claim that it is structurally better than TTT has support. Third, Table 1 does not state the measurement hardware alongside the numbers, so one cannot generalize that the same FPS would appear on other GPUs. Fourth, the 324 MB state size is the scene state alone, and actual inference peak is 18 GB. The benefit differs depending on whether what matters for deployment is per-scene state or total GPU memory.

6.3 Deployment judgment

If the use case is an interactive viewer or trajectory rendering that draws 5–7 or more new views per scene, with 16–64 views at 960p, TTB’s trade is worthwhile. If reconstruction latency matters and some quality loss is acceptable, TTB– is the candidate. TTB–’s PSNR gap to DVSM is 0.60/1.01/1.82 dB at $V=16/32/64$ respectively, widening as $V$ grows (Table 1, PDF p.7). If $V$ exceeds 64 and the single MLP saturates, use TTB(dup).

Conversely, this method is not yet the right fit if you render only one or two new views, need to extrapolate outside the observed range, lack a training/fine-tuning budget on the scale of 64×A100, or have a contract requiring quality identical to DVSM.

Finally, the paper mentions that the baking idea could apply to long-context KV caches in LLMs, video world models, and streaming sensor input in autonomous driving (Sec. E, PDF p.24), but this is a proposal, not an experiment. The validation scope of this paper is view synthesis.

Figures in this post are taken from the original arXiv:2611.17331 (CC BY 4.0). Only size and format were changed.