Lemmy: Bestiverse
  • Communities
  • Create Post
  • Create Community
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
RSS BotBM to Hacker NewsEnglish · 8 days ago

Proving and Improving the Platonic Representation Hypothesis with JEPAs

arxiv.org

external-link
message-square
0
link
fedilink
1
external-link

Proving and Improving the Platonic Representation Hypothesis with JEPAs

arxiv.org

RSS BotBM to Hacker NewsEnglish · 8 days ago
message-square
0
link
fedilink
Whether models trained on different modalities converge to the same representation of the world is central to AI. The Platonic Representation Hypothesis (PRH) conjectures that they do as they scale, but without a proof it offers neither a guarantee nor a recipe. We turn the hypothesis into a theorem. First, we prove a \emph{Platonic Representation Theorem}: Joint-Embedding Predictive Architectures (JEPAs) trained separately on any number of modalities learn the same representation up to one rotation whenever their positive pairs are two observations of one scene--\emph{good views}--and their misalignment is bounded otherwise. Second, we show that next-token prediction is a JEPA with \emph{bad views}, parts of a single document, which explains why language models align with vision only through scale. Third, we validate the theory from 145k captions to the texts of 940M web images, and uncover a scaling law of alignment that predicts unseen runs within seed noise. Good views need 42 times fewer training samples than next-token prediction on the same corpus to reach the same alignment, and a 109M-parameter text JEPA aligns with DINOv2 and CLIP's image encoders better than every language model we evaluated, up to 72B parameters, and than CLIP's text encoder: next-token prediction is a remarkably inefficient route to platonic representations. As a showcase, one rotation turns separately trained encoders into an image captioner (CIDEr .838 on COCO). Even that rotation needs no pair: on the latest pair-free benchmark, our text JEPA beats every zero-shot accuracy reported with language models up to 8B parameters, e.g., 64\% against at most 56\% top-5 on ImageNet-100. Positive views, and not scale alone, are thus the lever to build platonic representations.

Comments

alert-triangle
You must log in or register to comment.

Hacker News

hackernews

Subscribe from Remote Instance

You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !hackernews@lemmy.bestiver.se
lock
Community locked: only moderators can create posts. You can still comment on posts.

NOTICE: The bot occasionally has downtime, sorry about that. I’m working on adding better monitoring and error recovery so I can get notified faster.

Posts from the RSS Feed of HackerNews.

The feed sometimes contains ads and posts that have been removed by the mod team at HN.

Source of the RSS Bot.

Visibility: Public
globe

This community can be federated to other instances and be posted/commented in by their users.

  • 599 users / day
  • 2.23K users / week
  • 4.06K users / month
  • 8.84K users / 6 months
  • 2 local subscribers
  • 5.73K subscribers
  • 64.7K Posts
  • 35.6K Comments
  • Modlog
  • mods:
  • patrick
  • RSS BotB
  • BE: 0.19.21
  • Modlog
  • Instances
  • Docs
  • Code
  • join-lemmy.org