Latent Dynamics · 2026-10-01

Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning

Anthony Fuller, Scott C. Lowe, Daniel G. Kyrollos, Graham W. Taylor, Evan Shelhamer, James R. Green

arXiv:2609.38278PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Masked Swingers is a modified MAE that makes two independently augmented views, encodes each view separately, then exchanges the decoder’s CLS token between views before reconstructing masked patches. The authors report consistent gains over MAE on ImageNet-1K kNN and large gains on fine-grained tasks, plus a −64% relative error reduction on three state-probing datasets for preliminary world modeling.

Why it matters

The paper targets an autoencoding gap: MAE reconstructs masked patches but does not directly encourage view-agnostic representations. By independently augmenting/masking views and trading CLS tokens between views, Masked Swingers aims to force the encoder to learn summaries useful across view transformations, enabling better frozen features and preliminary improvement on state probing for world modeling.

Method

  • Create two independently augmented + masked views of the same image, patchify and encode each view separately with a shared encoder.
  • During decoding, exchange the CLS token between views (CLS from view 2 used to decode view 1, and vice versa) before reconstructing the masked patches.
  • Compute the MAE-style reconstruction loss averaged over both views; the paper also studies a Half-Swingers variant and token-exchange alternatives.

Limitation

The MotionJEPA world-modeling comparison is not directly comparable: its encoder is ViT-T/14 trained from scratch on each game, while other models are ViT-B/16 at 224×224 pre-trained on ImageNet-1K.

Abstract (from arXiv)

Self-supervised learning (SSL) removes the need for annotations and makes models that are capable across more domains than supervised learning. The autoencoder SSL framework learns by reconstructing its own input after information loss through a bottleneck or noise injection. Masked autoencoders (MAE) are the most successful instantiation of this framework: they encode a random subset of patches, then decode the masked-out patches. In this work, we introduce key modifications to improve MAEs. Our method augments an image in two different ways, then masks and encodes each view separately. It then exchanges the global representations (CLS tokens) between views before decoding the masked patches. By design, our Masked Swingers encourages learning a view-agnostic summary of the image to facilitate efficient transfer. We perform extensive experiments, and find Masked Swingers outperforms MAE by +3-5% on ImageNet-1K kNN and provides large gains on fine-grained tasks, e.g., relative gains of +45% on instance retrieval, +22% on animal re-ID, and +76% on Omniglot character recognition. To boot, Swingers reduces error -64% relative to MAE on three new state-probing datasets, opening the door to world modeling. Welcome to our Swingers party.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-09-29 · Indexed by Pub-AI: 2026-10-01 UTC

preprint

Masked Swingers: Harnessing Data Augmentation to Advance Autoencoders for Self-Supervised Learning

Anthony Fuller, Scott C. Lowe, Daniel G. Kyrollos, Graham W. Taylor, Evan Shelhamer, James R. Green

Preprint posted: 2026-09-29 · Original record ↗

arxiv
2609.38278
Explicit source versions (1)
  • v1 · 2026-09-29
  1. Preprint available

    Source date: 2026-09-29 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Self-supervised learning (SSL) removes the need for annotations and makes models that are capable across more domains than supervised learning. The autoencoder SSL framework learns by reconstructing its own input after information loss through a bottleneck or noise injection. Masked autoencoders (MAE) are the most successful instantiation of this framework: they encode a random subset of patches, then decode the masked-out patches. In this work, we introduce key modifications to improve MAEs. Our method augments an image in two different ways, then masks and encodes each view separately. It then exchanges the global representations (CLS tokens) between views before decoding the masked patches. By design, our Masked Swingers encourages learning a view-agnostic summary of the image to facilitate efficient transfer. We perform extensive experiments, and find Masked Swingers outperforms MAE by +3-5% on ImageNet-1K kNN and provides large gains on fine-grained tasks, e.g., relative gains of +45% on instance retrieval, +22% on animal re-ID, and +76% on Omniglot character recognition. To boot, Swingers reduces error -64% relative to MAE on three new state-probing datasets, opening the door to world modeling. Welcome to our Swingers party.