Unassigned · 2026-10-20
Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks
Shivaprasad B. Gowda, Sparsh Mittal
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors track representational geometry (stable and effective rank) throughout training and report three recurring rank trajectories: persistent high-rank, collapse-and-recovery, and persistent low-rank. They define Destructive Representation Collapse (DRC) as rank-one representations that fail to recover. In controlled synthetic settings, collapse-and-recovery is linked to successful optimization, while DRC is linked to failure; on real-world models the relationship is configuration dependent.
Why it matters
This paper characterizes how representation rank evolves during optimization, not just at convergence. By introducing rank-trajectory archetypes and DRC, it provides a framework to connect temporal representation dynamics with optimization success or failure across architectures and settings, and to forecast outcomes from early training milestones.
Method
- Track stable rank and effective rank across all layers at multiple logged epochs during training, using min-layer rank as the primary synthetic metric and penultimate-layer rank for endpoint compatibility.
- Partition runs into three geometric trajectory archetypes (A1 persistent high-rank, A2 collapse-and-recovery, A3 persistent low-rank) and define DRC using both stable and effective rank thresholds.
- Use large synthetic sweeps and statistical tests (Fisher’s exact test; multivariate logistic regression) to relate trajectory categories to optimization success at convergence and to test early-milestone forecasting.
Limitation
They state that archetypes are defined based on the trajectory including its final epoch, so the reported trajectory–success relationships are associated rather than prospective (they later add an early-milestone analysis for prospective support, but do not test turning it into a reliable automated early-stopping criterion).
Abstract (from arXiv)
Recent work has shown that deep neural networks exhibit an implicit low-rank simplicity bias. In this paper, we investigate the questions that naturally arise from this observation. What are the temporal consequences of low-rank simplicity bias during optimization? How does representation geometry evolve throughout optimization rather than only at convergence? Do different neural architectures exhibit distinct geometric optimization behaviors? To answer these questions, we conduct a large-scale empirical study tracking representational geometry throughout training. We find that neural networks follow three recurring geometric trajectories: persistent high-rank, collapse-and-recovery, and persistent low-rank. We further identify Destructive Representation Collapse (DRC) as a pathological regime in which representations collapse to rank-one and fail to recover, leading to a sustained loss of representational capacity. While collapse-and-recovery is consistently associated with successful optimization in controlled settings, experiments on real-world models show that the relationship between geometric trajectories and optimization outcomes is configuration dependent. Overall, our findings suggest that representation geometry evolves dynamically during training and provide a new perspective on how deep neural networks learn.
Related papers
- Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures
- Learning Intrinsic Dynamics in Latent World Models
- How Representational Priors Shape Generalization in World Action Models
- Control-Geometry Straightening for Sampling-Based Latent Planning
- Learning to Plan from Random Exploration
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-17 · Indexed by Pub-AI: 2026-10-20 UTC
Representations Have Trajectories: Characterizing Temporal Representation Dynamics in Neural Networks
Preprint posted: 2026-10-17 · Original record ↗
- arxiv
- 2610.24571
Explicit source versions (1)
- v1 · 2026-10-17
- Preprint available
Source date: 2026-10-17 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent work has shown that deep neural networks exhibit an implicit low-rank simplicity bias. In this paper, we investigate the questions that naturally arise from this observation. What are the temporal consequences of low-rank simplicity bias during optimization? How does representation geometry evolve throughout optimization rather than only at convergence? Do different neural architectures exhibit distinct geometric optimization behaviors? To answer these questions, we conduct a large-scale empirical study tracking representational geometry throughout training. We find that neural networks follow three recurring geometric trajectories: persistent high-rank, collapse-and-recovery, and persistent low-rank. We further identify Destructive Representation Collapse (DRC) as a pathological regime in which representations collapse to rank-one and fail to recover, leading to a sustained loss of representational capacity. While collapse-and-recovery is consistently associated with successful optimization in controlled settings, experiments on real-world models show that the relationship between geometric trajectories and optimization outcomes is configuration dependent. Overall, our findings suggest that representation geometry evolves dynamically during training and provide a new perspective on how deep neural networks learn.
