Self Forcing is designed to block gradients across chunks, but its training code unintentionally allows them to leak through the KV cache. This changes the intended training gradients and increases GPU memory usage. Downstream implementations that inherit the same cache-update pattern also inherit the bug.
How a fresh cache keeps a stale graph
Self-forcing training generates a video one chunk at a time. Within each chunk, the forward pass at a selected denoising step—the exit-step forward pass—runs with gradients enabled. Its output contributes to the training loss. The model then performs a context refresh under torch.no_grad(), updating the cache with K/V computed from the generated chunk.
Both forward passes write to the same persistent cache buffers. The exit-step forward pass records an autograd connection through its in-place write, typically via CopySlices. The subsequent no-grad refresh overwrites the values without removing that connection. When the next chunk reads the cache, its gradients can flow back into the earlier exit-step forward pass.
PyTorch’s no_grad() disables recording of new operations; it does not retroactively detach an existing tensor. detach() explicitly disconnects a tensor from its autograd history. These connections keep earlier computation graphs alive across chunks, increasing GPU memory usage.
A minimal reproduction
import torch
def probe(detach_cache=False):
w = torch.ones(1, requires_grad=True)
cache = torch.zeros(4)
# Earlier chunk: a differentiable, in-place cache write.
cache[:2] = w * 2
# Refresh: replace the values without recording a new graph.
with torch.no_grad():
cache[:2] = 5.0
if detach_cache:
cache = cache.detach()
# A later chunk reads the refreshed context.
q = torch.tensor(3.0, requires_grad=True)
loss = (cache[:2] * q).sum()
grad_w, grad_q = torch.autograd.grad(loss, (w, q), allow_unused=True)
return loss.item(), None if grad_w is None else grad_w.item(), grad_q.item()
print(probe(False)) # (30.0, 12.0, 10.0)
print(probe(True)) # (30.0, None, 10.0)
Each tuple contains the loss, the gradient with respect to w, and the gradient with respect to q. None means that the loss is no longer connected to w; the gradient with respect to q is unchanged.
Download the CPU reproduction and run python kv_cache_gradient_leak.py with PyTorch installed.
The fix: detach the cache at the boundary
After refreshing the context, detach both K and V in every layer before generating the next chunk:
# Preserve the current chunk's connection to the training loss.
outputs.append(denoised_pred)
with torch.no_grad():
refresh_context(denoised_pred, kv_cache)
# Stop gradients through persistent context, not through the output above.
for layer_cache in kv_cache:
layer_cache["k"] = layer_cache["k"].detach()
layer_cache["v"] = layer_cache["v"].detach()
Apply .detach() at every chunk boundary, after the cache-update forward pass represented by refresh_context. Detachment shares the existing storage and preserves cached values. The current chunk’s output remains connected to the training loss.
Recomputing context offers another way to avoid this leak. Self Gradient Forcing (SGF) performs a no-grad rollout, then re-encodes the recorded context latents in a separate gradient-enabled forward pass. This gives the backward pass a fresh graph corresponding to the context computation used in that forward pass, avoiding the stale cache connections described above. Detachment restores Self Forcing’s zero cross-chunk gradient boundary; SGF instead enables gradients through recomputed context K/V while keeping the generated context latents detached.
Results
We compare DMD training with and without KV cache detachment using two few-step initializations: Causal Forcing’s ODE checkpoint (causal_ode.pt) and Causal Forcing++’s CD checkpoint (causal_cd.pt).
Memory and gradient norms
Training GPU memory
−19.10 GB · −22.0%Blocking unintended cross-chunk gradients also lowers the generator gradient norm during training. Across the logged steps, the mean falls from 0.1483 to 0.0921 (−37.9%) with ODE initialization and from 0.1307 to 0.0830 (−36.5%) with CD initialization.
VBench after 750 DMD iterations
Detachment improves video dynamics with both initializations, reflected in higher Dynamic Degree scores. The overall VBench score is nearly unchanged with ODE initialization and lower with CD initialization.
Causal Forcing · ODE initialization
−0.05 pointsDynamic Degree 60.00 → 71.94
Causal Forcing++ · CD initialization
−1.18 pointsDynamic Degree 80.56 → 84.72
Dynamic Degree during training
With CD initialization, detachment yields higher Dynamic Degree at every evaluated checkpoint. With ODE initialization, the initial advantage diminishes during training and reverses at 1500 iterations.
Video comparisons after DMD training
The videos use EMA checkpoints after 750 DMD iterations, with CD or ODE initialization. For each initialization, the two variants use the same training seed and initial generation noise across 21 matched prompts.
A realistic photograph in a gritty urban style of a white and orange tabby alley cat dashing across a narrow back street alley during a heavy downpour. The cat is drenched, its fur matted and slick, and it looks determinedly for shelter. Its green eyes are wide and alert, focused intently on finding a safe place. The background is blurred, revealing a dimly lit alley with wet cobblestones and a few dilapidated buildings. The photo has a sharp focus on the cat, capturing its natural movements and the dynamic environment. A medium shot from a low angle, emphasizing the cat's urgency and the wet, urban setting.
Ready to play.
832 × 480 per video · 16 fps
Where the pattern appears in official code
The table below links the papers and official code, with code evidence for each status.
Statuses apply to the linked training paths. A half-filled check marks configuration-dependent risk or incomplete training code.
| Affected? | Paper | Official code | Code evidence and scope |
|---|---|---|---|
| Likely | Self Forcing | Self-Forcing · 33593df | The refresh retains the exit graphA gradient-enabled exit-step forward pass and no-grad refresh share persistent buffers with in-place K/V writes and cache reads. The cache remains connected to the exit-step forward pass. |
| Likely | Causal Forcing | Causal-Forcing · eb8705b | Shared cache across exit and refresh |
| Likely | Causal Forcing++ | Causal-Forcing · eb8705b | The same DMD training loopThe shared repository uses the same self-forcing DMD training loop. |
| Likely | LongLive 1.0 | LongLive1.0 · 31c70d9 | Deferred writes retain autograd history Deferred K/V writes are applied within the same forward pass; refresh runs under no_grad(). Its cache-clearing helper does not detach the cache between every pair of blocks within a training chunk. |
| Likely | LongLive 2.0 | LongLive2.0 · 31c70d9 | Non-quantized cache pathThe non-quantized path reads historical cache through cat or clone, writes undetached K/V, and refreshes under no_grad(). |
| Possible | Rolling Forcing | RollingForcing · a1477d0 | Rolling-window trainingA gradient-enabled window forward pass and no-grad refresh share a cache with in-place K/V writes. The rolling-window gradient behavior remains unverified. |
| Possible | DistillAlign | DistillAlign · 00eac44 | Training rollout not releasedThe released attention implementation contains the same in-place K/V write and cache read. The training rollout is not included in this release. |
| Possible | OPSD-V | OPSD-V · 0f4b9b6 | Default settings detach; an optional mode does notThe released Self-Forcing and LongLive training configurations set opsd_backward_per_chunk=true, which detaches caches after each backward pass, as specified in Algorithm 1. Setting it to false skips detachment: persistent K/V writes retain their graph across the no-grad refresh, and later cache reads can propagate gradients into earlier chunks. |
| No | Self Gradient Forcing | Self_Gradient_Forcing · 79be9ac | Recomputed context gradientsThe entire serial rollout and refresh run without gradients; gradient-enabled replay recomputes context from detached latents. Backward differentiates the replay computation, avoiding stale rollout-cache connections. |
| No | Causal-rCM | rcm · ed3cb14 | Detached writes; read-only cache access Persistent K/V writes explicitly detach, while the gradient-enabled exit-step forward pass uses READONLY. The current chunk remains differentiable; cached context is detached. |
References
-
Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion. NeurIPS 2025. arXiv:2506.08009.
-
Hongzhou Zhu, Min Zhao, Guande He, Hang Su, Chongxuan Li, and Jun Zhu. Causal Forcing: Autoregressive Diffusion Distillation Done Right for High-Quality Real-Time Interactive Video Generation. ICML 2026. arXiv:2602.02214.
-
Min Zhao, Hongzhou Zhu, Kaiwen Zheng, et al. Causal Forcing++: Scalable Few-Step Autoregressive Diffusion Distillation for Real-Time Interactive Video Generation. arXiv preprint, 2026. arXiv:2605.15141.
-
Shuai Yang, Wei Huang, Ruihang Chu, et al. LongLive: Real-time Interactive Long Video Generation. ICLR 2026. arXiv:2509.22622.
-
Yukang Chen, Luozhou Wang, Wei Huang, et al. LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation. arXiv preprint, 2026. arXiv:2605.18739.
-
Kunhao Liu, Wenbo Hu, Jiale Xu, Ying Shan, and Shijian Lu. Rolling Forcing: Autoregressive Long Video Diffusion in Real Time. ICLR 2026. arXiv:2509.25161.
-
Jiaxing Li, Kai Zou, Cindy Zhou, et al. DistillAlign: Coordinating Mode Covering and Mode Seeking in Autoregressive Video Distillation. arXiv preprint, 2026. arXiv:2607.26811.
-
Junhao Zhuang, Shiyi Zhang, Yuxuan Bian, et al. Self Gradient Forcing: Native Long Video Extrapolation. arXiv preprint, 2026. arXiv:2607.20368.
-
Kaiwen Zheng, Guande He, Min Zhao, et al. Causal-rCM: A Unified Teacher-Forcing and Self-Forcing Open Recipe for Autoregressive Diffusion Distillation in Streaming Video Generation and Interactive World Models. arXiv preprint, 2026. arXiv:2606.25473.
-
Hongyu Liu, Chun Wang, Feng Gao, et al. OPSD-V: On-Policy Self-Distillation for Post-Training Few-Step Autoregressive Video Generators. arXiv preprint, 2026. arXiv:2607.08766.
Cite this post
@misc{yang2026kvcache,
author = {Yang, Donglin},
title = {KV Cache Gradient Leakage in Autoregressive Video Training},
year = {2026},
month = nov,
url = {https://linydthu.github.io/blog/2026/kv-cache-gradient-leakage/},
note = {Blog post}
}