JEPA · 2026-10-28
AC-JEPA: Anchored Cross-Modal Joint Embedding Predictive Architecture for Visual-to-Symbolic Translation
Chaofei Hong, Ziyang Chen, Baochen Wang, Christina Yi Jin, Qing Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
AC-JEPA integrates cross-modal latent prediction into an SFT-trained image-to-SMILES VLM. It uses bidirectional masked-state JEPA in both visual and symbolic directions as an auxiliary objective while keeping a causal SFT “anchor” for token-level supervision. The authors report improved accuracy on WildMol Simple with clean-pair training and further gains with latent self-training using pseudo-labeled pairs.
Why it matters
The authors address cross-modal visual-to-symbolic translation when paired data can be imperfect. By making latent predictive learning an SFT-compatible plug-in inside the deployed generator, AC-JEPA aims to reuse pseudo-labeled pairs without converting pseudo-token errors into hard categorical targets, and the paper evaluates this on OCSR (OCSR as continuous-to-symbolic translation).
Method
- Adds an SFT-compatible JEPA branch at an intermediate decoder layer to predict masked image regions and masked SMILES spans using remaining multimodal context.
- Uses latent MSE for masked latent recovery with modality-wise SIGReg to prevent collapsed representations.
- During self-training, treats pseudo-SMILES as part of the masked latent prediction problem (continuous latent targets) while keeping the SFT token-level anchor active.
Limitation
The authors’ local analysis “provides a mechanism-level interpretation rather than a general guarantee for the full nonlinear training dynamics.”
Abstract (from arXiv)
Cross-modal translation requires models to capture shared semantics across modalities while preserving the fine-grained structure required by the target modality. Yet in real-world scientific domains, paired data are often imperfect, while obtaining clean pairs at scale is costly. Supervised fine-tuning (SFT) directly optimizes cross-modal translation through categorical token-level supervision, but this objective is sensitive to errors in the target sequences. JEPA offers a complementary representation-space objective, but most existing formulations focus on prediction within a modality rather than translation across modalities. We introduce Anchored Cross-Modal JEPA (AC-JEPA), an SFT-compatible plug-in that predicts masked intermediate states in both cross-modal directions using paired data. The joint-embedding objective targets cross-modal structure predictable from context, while the causal SFT path retains direct token-level supervision for the target sequence. During self-training, AC-JEPA learns from continuous latent targets derived from imperfect pseudo-labeled pairs, offering intrinsic tolerance to local data errors without promoting pseudo-tokens to hard categorical supervision. Using optical chemical structure recognition as a testbed, AC-JEPA trained on clean pairs achieves 76.07% accuracy on the Simple set of the WildMol benchmark, compared with 67.40% for SFT alone. Using the same pseudo-labeled data for both approaches, AC-JEPA-based self-training further raises accuracy to 80.23%, whereas accuracy under direct SFT drops from 67.40% to 63.47%. These results support SFT-anchored joint-embedding prediction as an effective strategy for cross-modal translation under both curated and imperfect supervision.
Related papers
- CI-JEPA: A Counterfactual Analysis of Latent Representations in Joint-Embedding Predictive Architectures for Self-Supervised Learning
- ER-JEPA: Experience Replay Improves Joint-Embedding Predictive Learning in Language Models
- Atom-JEPA: Joint-Embedding Predictive Architecture for 3D Atomistic Systems
- M-JEPA: Predictive Self-Supervised Learning for Molecular Graphs with Scaffold-Shift Evaluation on Tox21
- SaiVLA-JEPA: Evaluating Future Latent Prediction in JEPA-Conditioned VLA Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC
AC-JEPA: Anchored Cross-Modal Joint Embedding Predictive Architecture for Visual-to-Symbolic Translation
Preprint posted: 2026-10-27 · Original record ↗
- arxiv
- 2610.36493
Explicit source versions (1)
- v1 · 2026-10-27
- Preprint available
Source date: 2026-10-27 · First detected: 2026-10-28 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.36493v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Cross-modal translation requires models to capture shared semantics across modalities while preserving the fine-grained structure required by the target modality. Yet in real-world scientific domains, paired data are often imperfect, while obtaining clean pairs at scale is costly. Supervised fine-tuning (SFT) directly optimizes cross-modal translation through categorical token-level supervision, but this objective is sensitive to errors in the target sequences. JEPA offers a complementary representation-space objective, but most existing formulations focus on prediction within a modality rather than translation across modalities. We introduce Anchored Cross-Modal JEPA (AC-JEPA), an SFT-compatible plug-in that predicts masked intermediate states in both cross-modal directions using paired data. The joint-embedding objective targets cross-modal structure predictable from context, while the causal SFT path retains direct token-level supervision for the target sequence. During self-training, AC-JEPA learns from continuous latent targets derived from imperfect pseudo-labeled pairs, offering intrinsic tolerance to local data errors without promoting pseudo-tokens to hard categorical supervision. Using optical chemical structure recognition as a testbed, AC-JEPA trained on clean pairs achieves 76.07% accuracy on the Simple set of the WildMol benchmark, compared with 67.40% for SFT alone. Using the same pseudo-labeled data for both approaches, AC-JEPA-based self-training further raises accuracy to 80.23%, whereas accuracy under direct SFT drops from 67.40% to 63.47%. These results support SFT-anchored joint-embedding prediction as an effective strategy for cross-modal translation under both curated and imperfect supervision.