Yann LeCun's $1B Bet Against LLMs [Part 1]

My notes

Summary

Yann LeCun raised ~$1B for an alternative to LLMs: JEPA (joint embedding predictive architecture), a non-generative framework that predicts in embedding space rather than reconstructing pixels or tokens. The video traces why generative next-frame prediction fails on video (blurry averaging under uncertainty), how joint-embedding methods solved the “representation collapse” problem, and why LeCun believes world models, not autoregressive LLMs, are the path to planning agents.

Key Insight

Why generative video prediction fails (the core motivation)

• A model forced to output a single next frame under ambiguity (ball bounces left or right) can only predict the average of outcomes, washed-out blur. This compounds autoregressively into “blurry nothingness.”
• Scale framing: full-HD next-frame space is ~10^(15 million) possible frames vs GPT-2’s fixed 50,257-token vocabulary, discrete enumeration is hopeless for video.
• Key reframe: in GPT pre-training, being generative is incidental. What matters are the internal representations learned, not the autocomplete itself. So: can we learn those representations without reconstructing?

Joint embedding lineage (LeCun’s alternative path)

• Roots in Siamese networks (LeCun, Bell Labs, early 1990s, signature-fraud detection): two copies of a net output embeddings; trained to be similar for matching pairs, different for mismatches, never generating any image.
• The trap: representation collapse, network can cheat by outputting a constant vector (e.g. all ones) for any input, maximizing similarity while learning nothing.
• Fix #1: contrastive learning (positive + negative pairs). Problem: needs many negatives; LeCun argues worst-case negative count grows exponentially with representation dimension.
• Fix #2 (the epiphany): Barlow Twins (LeCun + Stéphane Deny, 2020), based on Horace Barlow’s 1961 redundancy-reduction hypothesis. Build the cross-correlation matrix between the two encoders’ neuron outputs and push it toward the identity matrix (diagonal to 1 = same image stays consistent; off-diagonal to 0 = decorrelate neurons). No negatives needed.

Hard benchmark numbers (ImageNet linear-probe, frozen encoder)

• AlexNet (2012, fully supervised): 59.3%
• Barlow Twins (2021, self-supervised + linear probe): 73.2%, beat original AlexNet by >10 pts.
• Supervised SOTA had moved on: ViT (Google, 2020) hit 88.6%.
• DINOv3 (Aug 2025): 88.4% self-supervised, first time SSL essentially matched weakly/fully supervised on image classification. Lineage: Barlow Twins to VICReg to DINO v1/v2/v3.

JEPA = world model

• Map observation at t and t+1 through encoders; a predictor predicts the embedding of t+1 from the embedding of t, optionally conditioned on an action, a learnable world model.
• Payoff: the predictor ignores unpredictable high-pixel noise (LeCun’s example: random leaf motion in a dashcam) and focuses on salient features that survive the encoder.
• V-JEPA 2 conditions on robot-arm control signals, planning by searching action sequences that drive predicted state toward a goal-state embedding. This is classic optimal control (Soviet late-’50s / West early-’60s), but with a learned abstract state representation, that’s the novel twist.
• LeCun’s thesis: LLMs “have no world models,” can’t predict consequences of actions, so can’t be reliable planning/agentic systems; inference must become search, not autoregressive token emission.