Driving · 2026-10-06

LatentQuant: Preserving the Policy-Facing Latent Contract under NVFP4 VAE Quantization

Ziye Deng, Lufang Chen, Shuyu Feng, Zhenwei Duan, Zicong Ye, Yu Sun, Xiaofan Li, Ruyi Gan, Hao Wang, Hao Zhang

arXiv:2610.03959PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

LatentQuant targets NVFP4 W4A4 quantization for world action model VAEs whose encoder latents condition a frozen action policy. The authors report joint QAT can match reconstruction yet violate the “policy-facing latent contract,” collapsing control (e.g., LIBERO success 95.5%→10.5%). LatentQuant uses a two-stage NVFP4 QAT: align/fix the quantized encoder first, then freeze it while adapting only the decoder, preserving control and reconstruction while enabling NVFP4 speedups.

Why it matters

For world action models with frozen downstream policies, VAE quantization must preserve not only reconstruction but also the policy-facing latent contract. The paper shows reconstruction-centric metrics can hide latent incompatibility caused by decoder-side activation quantization redirecting gradients and inducing persistent latent drift. LatentQuant provides an optimization boundary (encoder alignment + freezing) to preserve policy compatibility under NVFP4 W4A4 quantization.

Method

  • Formulate the “policy-facing latent contract” for VAEs whose encoder latents condition a frozen downstream policy; show reconstruction-only selection can be misleading.
  • Diagnose a failure mechanism under joint NVFP4 W4A4 QAT: decoder activation quantize-dequantize changes reconstruction gradients and induces persistent learned encoder latent drift.
  • Propose LatentQuant: two-stage NVFP4 QAT that (I) aligns the NVFP4 encoder to a high-precision reference, then (II) freezes the aligned encoder while adapting only the NVFP4 decoder for reconstruction.
Abstract (from arXiv)

Recent world action models (WAMs) reuse pretrained video VAEs whose encoder latents directly condition downstream action policies. Quantization must therefore preserve not only reconstruction fidelity but also the policy-facing latent contract expected by the frozen policy. Direct NVFP4 leaves W4A4 quantization error uncompensated, whereas joint quantization-aware training (QAT) can recover reconstruction by moving this representation. On Wan2.1, joint QAT nearly matches FP32 VBench-7 (0.7403 versus 0.7409), yet LIBERO success collapses from 95.5% to 10.5%. Controlled decoder-only experiments show that activation quantize-dequantize operations alter the reconstruction signal and decoder Jacobian, redirecting the gradient returned to the encoder and inducing persistent latent drift. Based on this mechanism, we introduce LatentQuant, a two-stage NVFP4 QAT framework that first aligns the quantized encoder with its high-precision counterpart, then freezes it while adapting the decoder. Across Wan2.1 and Wan2.2, LatentQuant preserves near-baseline control and high reconstruction quality, achieving 95.75% success on LIBERO and 68.8% on RoboTwin. On NVIDIA B300 GPUs, NVFP4 execution achieves 1.17x-1.26x end-to-end VAE speedups over BF16 cuDNN.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-06 UTC

preprint

LatentQuant: Preserving the Policy-Facing Latent Contract under NVFP4 VAE Quantization

Ziye Deng, Lufang Chen, Shuyu Feng, Zhenwei Duan, Zicong Ye, Yu Sun, Xiaofan Li, Ruyi Gan, Hao Wang, Hao Zhang

Preprint posted: 2026-10-02 · Original record ↗

arxiv
2610.03959
Explicit source versions (1)
  • v1 · 2026-10-02
  1. Preprint available

    Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Recent world action models (WAMs) reuse pretrained video VAEs whose encoder latents directly condition downstream action policies. Quantization must therefore preserve not only reconstruction fidelity but also the policy-facing latent contract expected by the frozen policy. Direct NVFP4 leaves W4A4 quantization error uncompensated, whereas joint quantization-aware training (QAT) can recover reconstruction by moving this representation. On Wan2.1, joint QAT nearly matches FP32 VBench-7 (0.7403 versus 0.7409), yet LIBERO success collapses from 95.5% to 10.5%. Controlled decoder-only experiments show that activation quantize-dequantize operations alter the reconstruction signal and decoder Jacobian, redirecting the gradient returned to the encoder and inducing persistent latent drift. Based on this mechanism, we introduce LatentQuant, a two-stage NVFP4 QAT framework that first aligns the quantized encoder with its high-precision counterpart, then freezes it while adapting the decoder. Across Wan2.1 and Wan2.2, LatentQuant preserves near-baseline control and high reconstruction quality, achieving 95.75% success on LIBERO and 68.8% on RoboTwin. On NVIDIA B300 GPUs, NVFP4 execution achieves 1.17x-1.26x end-to-end VAE speedups over BF16 cuDNN.