The Obsessed Encoder

A sneak peek to more efficient encoders.

Or: How 12 bits swallows your latent space.

Introduction

Our mission at Enigma is solving general-purpose AI for robots, by making them the most capable, as well as intuitive to use and interact with.

We want to help people solve hard tasks intuitively: from chores to complex physical work. Funnily enough, one way to do that is if a robot had an intuitive understanding of the world.

This kind of understanding does not necessarily have to originate from a billion hours of manually labeled data. Instead, one of the directions we have worked on is learning compact representations of the world, rich enough for almost any task yet small enough for fast planning.

JEPA-style self-supervision was one of the methods we have researched, following the approach of LeWM and other LeJEPA variants. Yet all our attempts fell short. We decided to look deeper.

The culprit turned out to be a long-known problem of self-supervised learning. Older systems tend to learn only the low-hanging fruit of a data distribution, latching onto slow features in video and a few easy-to-learn bits in images. Throughout our research on efficient encoders, we came up with a few interesting insights.

This old failure mode is easily reproduced in every modern JEPA-style system we tested (DINOv3, LeJEPA, and LeWM), despite each system's built-in anti-collapse defenses. In each case, a simple low-entropy feature can bypass the statistical safeguards of the model, and take over most of its latent capacity⁽¹⁾.

Existing approaches, covered in the Background section, mainly filter these features out. But sometimes this excludes important information. We thought to ourselves: "Why not fix the system itself?", rather than solving the problem with a set of "patches".

We see this problem space as a very important one — and decided to share parts of our research with the world.

Background

A joint-embedding predictive architecture (JEPA) learns from unlabeled data. It encodes two related views of the same input and predicts one encoding from the other. The prediction happens in the latent space rather than in the input itself. For images the two views are two augmented versions of the same picture, for example different crops, masked regions or color distortions. For videos, the model predicts the encoding of a future frame, optionally conditioned on the action taken in between.

The prediction task acts as an inductive bias selecting what the model learns to preserve. Features that remain stable across image augmentations or video frames are easier to predict than fleeting details. The goal is for the model to ignore irrelevant variation while retaining useful structure.
Latent prediction tasks have a trivial (collapsed) solution. If the encoder maps every input to the same vector, prediction is perfect - but the representation contains no information.

Practical systems therefore pair the prediction objective with an anti-collapse mechanism. LeJEPA and LeWM use SIGReg, whose goal is to make the embedding normally distributed. It does so by penalizing random projections without Gaussian-looking distributions⁽²⁾. DINOv3 takes a different route based on self-distillation. A student network learns to match an exponential moving average of itself.

Moreover, Sinkhorn-Knopp centering steps keep the teacher's outputs statistically balanced, and a KoLeo term pushes each embedding away from its nearest neighbor.
The failure we are chasing is not new. Chen, Luo & Li (2021) planted a few bits in ImageNet images and watched SimCLR-era encoders obsess over them, naming the effect feature suppression. Sobal et al. (2022) showed that VICReg-style world models lock onto slow features and neglect the dynamics.

Modern approaches filter those easy features out by suppressing static video backgrounds (Wang et al., 2021, Huang et al., 2021), erasing image shortcuts such as watermarks with a learned lens (Minderer et al., 2020), training a bisimulation objective on top of pretrained features to keep only control-relevant ones (Toso et al., 2026), or drawing contrastive negatives from within each trajectory (Gulati & Nemenman, June 2026).

Allocation, not selection

An encoder’s job is to select useful features and pack them into a fixed latent budget. Ideally, each feature would receive capacity in proportion to some measure of the information it carries (entropy, dimensionality, etc.).

However, JEPA-style training appears to incentivize extreme disparities in this allocation, strongly favoring features that are easy to predict. The bias toward predictability is the central premise of the JEPA framework, and has many positive aspects one would like to preserve.

Latent capacity, on the other hand, is a zero-sum game, and every feature which is disproportionately represented implies another feature that couldn't get in. Crucially, we see no evidence that the extra capacity adds meaningful detail. The inflated features could comfortably fit in a tiny fraction of the space they consume — whatever marginal refinement the remaining dimensions provide comes at the cost of every feature that was crowded out.

We demonstrate this by adding a predictable feature that carries almost no information, such as a faint watermark. In some of our experiments, a 12-bit feature came to dominate a 1024-dimensional latent space, while training looked perfectly healthy.

This is not a synthetic-only issue. In a mild variant of the classic PushT benchmark, which we dub RandGoal, we see exactly the same behavior. In this setup the agent's T always starts from the same pose, while the goal pose is randomized across episodes — the everyday situation of a robot that begins each task from its home position and must act on a target that moves. Goal-conditioned variants of PushT are well within reach of current supervised methods: Diffusion Policy reaches 68% success on the multi-goal PushT-M, so the task itself poses no fundamental difficulty. Prior solutions mainly focus on filtering out such features⁽³⁾, which fails here: the static feature is the goal itself. Moreover, this extreme response does not seem to require an extreme cause.

During experimentation, modest gaps between two otherwise comparable features were already enough to tip the allocation from sharing into winner-take-all. Gaps like that are unavoidable in real data: no two useful features are equally easy to predict.
When we understood this, it got us to believe that this misallocation exists in production systems today, quietly taxing encoders like DINOv3 and plausibly underlying practical difficulties that are otherwise hard to attribute.

Were capacity divided more evenly, these models could become much more efficient — unlocking new, easier training paradigms. How do you really make compact representations? You need a solution that directly addresses this allocation inequality (among a few other problems).

Results

We reproduce the collapse in three systems. DINOv3 is an industry standard. LeJEPA is a new promising method. LeWM is the action-conditioned world model from the robotics setting that got us down this path.

We adopt the experimental scheme of Sobal et al. (2022), until now used only in video settings, and apply it to image datasets. Following the feature suppression literature⁽⁴⁾, we plant a synthetic, predictable pattern, and compare three groups: a clean baseline, a test group whose pattern is almost constant across views of the same image, and a control group whose pattern is pixel-matched but freshly sampled every time. Because test and control corrupt the input by the same amount, any difference between them isolates predictability as the cause.

In addition, we track the loss crossover, where the test group's training loss drops below the clean baseline and control group, showing the objective actively prefers the collapse. Finally, a similarity analysis reveals that embeddings cluster by pattern instead of content.

The last two rows are a measure of what goes into the encodings. We compare pairs that share either the same image or the same injected pattern. In a healthy encoder, similarity follows the image. In a collapsed encoder, it follows the pattern. Even a faint, predictable feature can make the model focus on the pattern instead of the picture.

Two comparisons carry the entire argument:

The control tracks the baseline, so the corruption itself is not the problem. The control corrupts the input exactly as much as the test does, yet its loss curve, evaluation, and similarity analysis all track the clean run's shape, running at most a few points behind. The encoder simply learns to ignore a pattern it cannot predict.
The test collapses, so predictability is the problem. Its loss improves while its evaluation and its similarity measurements fail.

DINOv3

We train DINOv3 from scratch using the reference implementation with minor adaptations. We evaluate both global and local representations through downstream classification and depth estimation, respectively. Full implementation and evaluation details are provided in the appendix. Data is augmented with the following pattern based on a 12-bit image-dependent seed.

The control stays healthy here in exactly the template's sense. Its probe runs a few points behind clean at this horizon while matching its shape and improving throughout. Only the predictable pattern collapses, and the cases below instantiate the same template more sharply still.

The third measurement, the similarity analysis, shows the collapse at the representation level.

LeJEPA

We train a LeJEPA model from scratch on ImageNet-1k using a faithful port of the authors' minimal recipe, again adding the same watermark and control patterns. This time we use half the opacity.

The results follow our hypothesis exactly.

LeWM

We study LeWM on PushT, training and evaluating exactly as the original. Its own authors report this stack failing under mild variations of the benchmark, from agent colors to the goal pose RandGoal randomizes (Maes et al., 2026), though they offer no explanation. Video is more vulnerable to this failure than images. Consecutive frames share much of their content, so natural slow features are everywhere.

We modify the PushT data in two ways, one following the experimental template exactly and one arising from the task itself. The first adds a tiny colored square in the top left corner of the frame, 5×5 pixels, under 0.05% of the image. In the test group the square keeps one color through each episode, and in the control it is re-colored at random every frame.

The second modification, RandGoal, is a natural variant of the task. It fixes the T's start pose and randomizes the goal pose across episodes. The goal enters only through the pixels, a T outline drawn at the target pose, exactly how the original PushT renders its own goal. Goals are sampled uniformly and a scripted expert provides a successful demo for each. Randomizing the goal does not make the task hard. Supervised policies still solve it (Li et al., 2025).

Note how little the two features carry. Three numbers describe them fully, the corner square's color (𝑟, 𝑔, 𝑏) or the goal T's pose, its position (𝑥, 𝑦) and orientation 𝛼.

Both modifications produce the same failure we saw before, prediction loss below the baseline while eval success never rises above chance. RandGoal has no matched control of its own, the colored-square arms play that role, and its runs behave just like their test group.

RandGoal also undermines the existing filtering approaches. The goal pose stays constant through each episode, so to a filter it looks like exactly the slow feature it is built to remove. But the planner reads the goal through the same encoder, as an encoded goal frame, so filtering the goal out erases it from the goal frame as well. The escape is to show the planner a picture of the already solved scene, where the block itself sits in the goal pose. That is a poor fix, since in real robotics this is impossible. A glass of water beside the workspace holds perfectly still through the episode, so if erased the planner might move the arm straight through it.

We run the same similarity analysis as before and see similar results. The collapse is so fast here that the curves are already swapped at the first measurement.

The anatomy of a collapse

To build some intuition for the collapses, think about this: how does a feature worth a few bits defeat regularizers that were designed to resist collapse? Our starting point was Sobal et al. (2022), who showed with a simple mathematical argument that JEPA training is vulnerable to episode-constant shortcuts. As they show, an encoder that fixates on an episode-constant feature drives the prediction loss to zero.

All that is left in our case is the anti-collapse regularizer, SIGReg: the SOTA's go-to. Sobal's distractors are high-dimensional noise or whole overlaid images, with enough entropy to fill the embedding. The case we've seen carries only a few bits, so the same trick requires a subtler mechanism.

Taking for example a collapsed checkpoint of the LeWorldModel training, we know that its encodings are overwhelmingly determined by the goal's state. The state has only three degrees of freedom, position and angle, so the encodings fill at most a three-dimensional surface inside the 192-dimensional embedding space. Yet the collapsed run's SIGReg term ends no higher than the healthy baseline's, so the random projections look normal to the test, just as they would for a real normal distribution. How can this be?

To make this possible, one would expect a highly irregular embedding geometry, the encoder acting as a kind of hash function that spreads three numbers across 192 dimensions. We can visualize this. We let one T wander smoothly, and record each frame's embedding. We then run the identical trajectory with goal and block T replaced. We then plot the embedded trajectory projected on the dataset's 2 strongest principal components.

This is the collapse seen from the inside: a low-dimensional sheet folded until it looks Gaussian.

See for yourself

We want these results reproduced, challenged, and extended, so we are releasing everything needed to do so. The code and the repository are public and available here: The Obsessed Encoder.

This release includes the watermark code, the small and clearly marked diffs against each published training recipe (their code, our pixels), and the configurations that generated every figure in this post. Each collapse reproduces in a few hours on a single H100. Customize the injected feature and see for yourself.

Conclusion

To summarize, in every system we tested, JEPA-style training badly misallocates latent capacity toward its most predictable features, with little regard for their information content.

If you think about what we've seen, you'll realize that a sufficiently predictable feature can capture almost the entire representation.

We observed the same collapse, with the same measured signature, in DINOv3, LeJEPA, and LeWorldModel. The collapse is not an accident of training but a preference of the objective.

A collapsed solution achieves a lower training loss than an honest one. Short of that extreme, when similar bias operates as a quiet tax on ordinary training, it means latent capacity is over-spent on predictable features at the expense of anything else.

During our research, those insights made us even more bullish on pursuing paradigms of SSL, and not less.

This collapse is not a reason to abandon JEPA-style learning. As we've seen here, it is one of the obstacles between these methods and their promise.

Appendix: DINOv3 implementation details

We train DINOv3 from scratch with the published training code, using the repo's ImageNet-1k pretraining recipe (vitl_im1k_lin834, ViT-L/16, the "Fast setup", 83.3 linear at the published scale), sized down to a single GPU.

Although the published recipe uses a global batch of 2,048, KoLeo (the anti-collapse regularizer) is computed separately on each GPU over an effective batch of 64. Our single-GPU run therefore gives KoLeo more samples, not fewer.

The learning rate follows the recipe's own square-root scaling rule on its 500K-iteration schedule, of which we run the first 50K. Gram anchoring, a later-phase addition in the flagship pipeline, stays off here as in the published ViT-L recipe, and plays no role in these results. We believe the other implications of the smaller batch size are less consequential for these experiments.

Beyond these, the one code-level change is clearly marked: the planted pattern is applied to the source images before augmentation, so every augmented view of an image inherits the same faint, per-image luminance pattern, or an equally faint pattern drawn randomly at every tile.

We evaluate two downstream tracks:

The first is an online ImageNet-1k linear probe, a linear head trained passively on the frozen teacher CLS token as the run progresses (a training-time monitor, not the offline linear-probe protocol).

The second is a dense feature evaluation: an NYU-Depth linear probe on the patch tokens, using the published protocol's head but trained to 4,800 of its 38,400 iterations at each teacher milestone, with a single pass and no test-time augmentation. This makes it a consistent instrument for tracking the groups against each other, not a reproduction of the published depth number.

Footnotes

1. Theory work on feature suppression suggests wider embeddings and better augmentations as mitigations (Xue et al., 2023). The same work also proves that suppression persists at any embedding width when augmentations fail to disrupt the easy feature (their Theorem 5.4). Our planted patterns survive each recipe's augmentations by construction, exactly that regime.
2. SIGReg (Sketched Isotropic Gaussian Regularization) enforces this with the Epps-Pulley test. On each random 1-D projection it compares the empirical characteristic function — the Fourier transform of the projected distribution — against a Gaussian's, and integrates the squared difference over frequencies. The statistic is averaged across many random directions.
3. Toso et al. (2026), train an invariance objective on top of frozen pretrained features, claiming to filter out only what is irrelevant to control. Gulati & Nemenman, (June 2026, concurrent work) , draw the contrastive negatives from within each trajectory. Wang et al. (2021) and Huang et al. (2021) suppress video background shortcuts directly.
4. The feature suppression literature shows that easy shared features can crowd out harder ones in contrastive learning (Chen, Luo & Li, 2021, Li et al., 2020, Robinson et al., 2021). A few injected bits were enough to fully suppress ImageNet features in the experiments of Chen, Luo & Li. The effect is not limited to contrastive losses, their appendix reproduces it on BYOL and with a Gaussian distribution-matching loss.

BibTex

@misc{enigma2026obsessedencoder,
  title={The Obsessed Encoder},
  author={Enigma Team},
  year={2026},
  url={https://enigma.inc/posts/obsessed-encoder}
}