JEPA · 2026-11-13
HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment
Robbie Hamilton, Fan Zhang, David Bull
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
HiVE-VQA applies a JEPA-based frozen video encoder (V-JEPA 2.1 ViT-B) to no-reference video quality assessment. It extracts token grids from multiple depths, stacks them into a hierarchical spatio-temporal volume, and trains only a compact 3D convolutional head (5.2M parameters, 5.7% of total). Trained on LSVQ and evaluated without fine-tuning, it outperforms 17 SOTA methods on SROCC and PLCC across intra- and cross-dataset benchmarks.
Why it matters
The authors study whether a single frozen video foundation model can serve as an effective NR-VQA backbone without task-specific encoder fine-tuning or multi-network ensembles. Their hierarchical depth-wise volumetric fusion is designed to combine low-level distortion cues with deeper semantic/temporal context, aiming for accuracy and efficiency with resolution-invariant inference.
Method
- Use a frozen V-JEPA 2.1 ViT-B video encoder as the backbone for NR-VQA (only the quality head is trained).
- Extract features from multiple uniformly spaced transformer blocks, reshape token sequences into spatio-temporal grids, and stack them into a hierarchical volume over layer, spatial, and temporal axes.
- Train a compact 3D convolutional head that performs pointwise cross-layer fusion, spatio-temporal aggregation, global pooling, and regression to a scalar quality score.
Limitation
HiVE-VQA-M’s performance gap widens for high-resolution YouTube-UGC, and the authors attribute this to keeping the encoder frozen while other networks fine-tune their features to the dataset distribution.
Abstract (from arXiv)
The proliferation of user-generated video content has made accurate and robust no-reference video quality assessment (NR-VQA) a highly important challenge. Existing research increasingly relies on architecturally complex multi-branch pipelines or computationally intensive fine-tuning of single-branch backbones. In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes, combining low-level distortion cues from shallow layers with the semantic and temporal context captured at greater depths. The encoder remains frozen throughout, so only the head is trained: 5.2M parameters (5.7\% of the total), with no auxiliary multi-modal models and an inference cost that is invariant to video resolution. Trained solely on LSVQ and evaluated without any fine-tuning, HiVE-VQA outperforms 17 state-of-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks. These results indicate that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multi-network fusion. Code and model checkpoints are available at https://github.com/RobHam99/HiVE-VQA.
Related papers
- Honeycomb: Constant-Size Scene Memory Representation for Video World Models
- CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
- HLA-WM: Hybrid Linear Attention for Long-Horizon Video World Models
- V-JEPA Policy: Building Effective World-Action Models on Predictive Visual Latents
- MoDDiT: Simple Token Routing in Diffusion Transformers
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-12 · Indexed by Pub-AI: 2026-11-13 UTC
HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment
Preprint posted: 2026-11-12 · Original record ↗
- arxiv
- 2611.17146
Explicit source versions (1)
- v1 · 2026-11-12
- Preprint available
Source date: 2026-11-12 · First detected: 2026-11-13 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.17146v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
The proliferation of user-generated video content has made accurate and robust no-reference video quality assessment (NR-VQA) a highly important challenge. Existing research increasingly relies on architecturally complex multi-branch pipelines or computationally intensive fine-tuning of single-branch backbones. In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes, combining low-level distortion cues from shallow layers with the semantic and temporal context captured at greater depths. The encoder remains frozen throughout, so only the head is trained: 5.2M parameters (5.7\% of the total), with no auxiliary multi-modal models and an inference cost that is invariant to video resolution. Trained solely on LSVQ and evaluated without any fine-tuning, HiVE-VQA outperforms 17 state-of-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks. These results indicate that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multi-network fusion. Code and model checkpoints are available at https://github.com/RobHam99/HiVE-VQA.
