JEPA · 2026-11-13

HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment

Robbie Hamilton, Fan Zhang, David Bull

arXiv:2611.17146PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

HiVE-VQA applies a JEPA-based frozen video encoder (V-JEPA 2.1 ViT-B) to no-reference video quality assessment. It extracts token grids from multiple depths, stacks them into a hierarchical spatio-temporal volume, and trains only a compact 3D convolutional head (5.2M parameters, 5.7% of total). Trained on LSVQ and evaluated without fine-tuning, it outperforms 17 SOTA methods on SROCC and PLCC across intra- and cross-dataset benchmarks.

Why it matters

The authors study whether a single frozen video foundation model can serve as an effective NR-VQA backbone without task-specific encoder fine-tuning or multi-network ensembles. Their hierarchical depth-wise volumetric fusion is designed to combine low-level distortion cues with deeper semantic/temporal context, aiming for accuracy and efficiency with resolution-invariant inference.

Method

  • Use a frozen V-JEPA 2.1 ViT-B video encoder as the backbone for NR-VQA (only the quality head is trained).
  • Extract features from multiple uniformly spaced transformer blocks, reshape token sequences into spatio-temporal grids, and stack them into a hierarchical volume over layer, spatial, and temporal axes.
  • Train a compact 3D convolutional head that performs pointwise cross-layer fusion, spatio-temporal aggregation, global pooling, and regression to a scalar quality score.

Limitation

HiVE-VQA-M’s performance gap widens for high-resolution YouTube-UGC, and the authors attribute this to keeping the encoder frozen while other networks fine-tune their features to the dataset distribution.

Abstract (from arXiv)

The proliferation of user-generated video content has made accurate and robust no-reference video quality assessment (NR-VQA) a highly important challenge. Existing research increasingly relies on architecturally complex multi-branch pipelines or computationally intensive fine-tuning of single-branch backbones. In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes, combining low-level distortion cues from shallow layers with the semantic and temporal context captured at greater depths. The encoder remains frozen throughout, so only the head is trained: 5.2M parameters (5.7\% of the total), with no auxiliary multi-modal models and an inference cost that is invariant to video resolution. Trained solely on LSVQ and evaluated without any fine-tuning, HiVE-VQA outperforms 17 state-of-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks. These results indicate that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multi-network fusion. Code and model checkpoints are available at https://github.com/RobHam99/HiVE-VQA.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-12 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

HiVE-VQA: Hierarchical Interaction via Volumetric Embedding for Video Quality Assessment

Robbie Hamilton, Fan Zhang, David Bull

Preprint posted: 2026-11-12 · Original record ↗

arxiv
2611.17146
Explicit source versions (1)
  • v1 · 2026-11-12
  1. Preprint available

    Source date: 2026-11-12 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

The proliferation of user-generated video content has made accurate and robust no-reference video quality assessment (NR-VQA) a highly important challenge. Existing research increasingly relies on architecturally complex multi-branch pipelines or computationally intensive fine-tuning of single-branch backbones. In this paper, we investigate whether a single frozen video foundation model can serve as an effective NR-VQA backbone, and propose HiVE-VQA, the first framework (to our knowledge) to apply a JEPA-based video encoder to this task. HiVE-VQA extracts token grids from multiple depths of a pre-trained V-JEPA 2.1 encoder and stacks them into a hierarchical spatio-temporal volume, which a compact 3D convolutional head fuses across its layer, spatial, and temporal axes, combining low-level distortion cues from shallow layers with the semantic and temporal context captured at greater depths. The encoder remains frozen throughout, so only the head is trained: 5.2M parameters (5.7\% of the total), with no auxiliary multi-modal models and an inference cost that is invariant to video resolution. Trained solely on LSVQ and evaluated without any fine-tuning, HiVE-VQA outperforms 17 state-of-the-art (SOTA) methods in SROCC and PLCC on both intra-dataset (LSVQ-test, LSVQ-1080p) and cross-dataset (KoNViD-1k, LIVE-VQC) benchmarks. These results indicate that hierarchical, depth-wise fusion of features from a single frozen video foundation model provides a more effective and efficient basis for NR-VQA than task-specific multi-network fusion. Code and model checkpoints are available at https://github.com/RobHam99/HiVE-VQA.