Your brain weighs ~1.4 kg. Runs on ~20 watts — roughly 3 bananas per day.

A single NVIDIA DGX H200 — one AI training node — draws 10,200 watts, roughly ~2,000 bananas per day. A GPT-4 training run consumed enough energy to power a small city for a week.

And yet — your brain learns to recognise a cat from 5–10 examples. GPT-4 needed trillions of tokens. Your brain generalises, adapts, reasons about novel situations — and does it all on fruit.

🔑 The gap is not computational power. The gap is algorithmic intelligence. We're burning 510× more energy per node and still can't match the brain's flexibility.

What You'll Take Home Today:

• 🧠 You'll understand what's actually inside a modern LLM — not hand-wavy analogies, the real architecture
• 🗜️ You'll know how to make models 10–100× smaller without destroying them
• 💻 You'll run a state-of-the-art reasoning model on your own laptop — tonight

At every stage I'll tell you where we are in the story. You'll never be lost.

When someone says "transformer" in AI...

❌ Surely not this kind?

❌ Or this kind?

❌ Or even this kind?

🔑 They mean a specific neural network architecture invented at Google in 2017 — Attention Is All You Need (Vaswani et al., arXiv:1706.03762)

Every major model (Llama 3, Gemma 2, Mistral, DeepSeek, Qwen) converged on:

① Tokenize — BPE or SentencePiece (vocabulary ~32K–128K tokens)

② Each transformer block:

• RMSNorm (not LayerNorm) — pre-norm: x / sqrt(mean(x²) + ε) × γ — ~15% fewer ops (Zhang & Sennrich, 2019)
• Grouped Query Attention (GQA) — reduces KV cache by sharing K,V heads across Q groups (Ainslie et al., 2023, arXiv:2305.13245)
• SwiGLU feed-forward — SwiGLU(x) = Swish(xW₁) ⊙ (xV) × W₂ — consistently outperforms ReLU/GELU (Shazeer, 2020)
• Residual connections (unchanged from 2017 — still essential)

③ Autoregressive next-token prediction

Common misconception: "Transformers use LayerNorm and ReLU." No. Not since 2023.

The KV cache problem: During generation, we cache K and V for all previous tokens. For a 70B model with 80 heads, 128 dim/head, 8K context: ~20 GB KV cache per sequence. This is the serving bottleneck.

GQA is the new standard. Llama 2 70B: 64 Q heads, 8 KV heads → 8× KV cache reduction. MHA is effectively deprecated for large-scale deployment.

FlashAttention (Dao et al., 2022, arXiv:2205.14135): Rewrites attention to work in GPU SRAM tiles. 2-4× speedup, linear memory, exact output. Default in PyTorch ≥2.0.

DeepSeek-V3 (Dec 2024): 671B total params, only 37B active per token. Trained for ~$5.57M. Competitive with GPT-4o.

Mamba (Gu & Dao, 2023, arXiv:2312.00752): O(N) time and memory. Strength: long-context streaming. Weakness: in-context learning. SSMs are not a Transformer replacement yet — they are a complement.

The breakthrough idea: Instead of only scaling training compute, scale inference compute.

OpenAI o1/o3 (2024-2025)

• Extended chain-of-thought reasoning before answering, trained with RL
• Can "think harder" on difficult problems by generating more tokens
• o3: >90% on ARC-AGI, ~96% on AIME math competition

DeepSeek-R1 (Jan 2025, arXiv:2501.12948)

• Open-weight 671B MoE reasoning model
• Pure RL (GRPO) on base model produces emergent CoT reasoning without supervised CoT data
• Matches o1-level performance on math and coding — open-weight + recipe published

Pause. o3 exceeded human performance on ARC-AGI in 2024. This happened in your lifetime. What does that mean?

The test-time scaling law: More reasoning tokens → better answers, following a smooth scaling curve. A 7B model thinking for 2000 tokens can outperform a 70B model thinking for 100.

🔑 You don't need a bigger model — you need a model that thinks longer.

The modern fine-tuning pipeline (2025):

Base model (pretrained) → [SFT] → [Preference Alignment] → Aligned model (deployed)

SFT (Supervised Fine-Tuning)

Train on (instruction, response) pairs. Standard cross-entropy. Teaches format/style. "What to say."

RLHF (Ouyang et al., 2022)

Train a reward model on human preferences. Optimise policy with PPO. Complex: 4 models in memory. "What's good."

DPO (Rafailov et al., 2023, arXiv:2305.18290)

Closed-form relationship between optimal RLHF policy and reward function. Eliminates reward model entirely. Simpler, more stable. Now the default for open-source alignment.

Variants: ORPO (Hong, 2024) — SFT + alignment in one step. SimPO (Meng, 2024) — reference-model-free. KTO (Ethayarajh, 2024) — unpaired preferences.

✋ Think: SFT teaches "what to say." RLHF/DPO teaches "what's good" — capturing helpfulness, harmlessness, honesty.

There are three main directions for training LLMs efficiently:

1. Single GPU Training

• Best for models that fit comfortably in one GPU's memory
• Standard DataLoader + optimizer loop, no parallelism overhead
• Limited to models ~7B on 80GB A100; use quantization to push further

2. Multi-GPU Data-Parallel Training

• Each GPU holds a full copy of the model
• Different data batches processed in parallel, gradients averaged
• Does not work for models requiring more than 1 GPU's memory for forward + backward pass

3. FSDP — Fully Sharded Data Parallelism

• Shards model parameters, gradients, and optimizer states across GPUs
• Enables training models that are too large to fit on any single GPU
• Each GPU holds only its shard; parameters are gathered on-demand during compute

🆕 Predicting Training Instability (2026)

• Residual Koopman Spectral Profiling (arXiv:2602.22988): Treats the transformer forward pass as a dynamical system and applies Koopman operator theory to the residual stream
• Can predict training instability ~100 steps before a loss spike happens by reading spectral precursors
• Could eliminate catastrophic training failures — one of the most expensive problems in large-scale training

🔑 FSDP is the key technique for fine-tuning frontier-scale models without an ocean of memory — it's what makes Llama 3 405B fine-tuning practical. And with Koopman profiling, we may soon be able to prevent the training crashes that waste millions in compute.

The Transformer has dominated every modality, domain, and task for 8 years. No architecture in ML history has had this kind of run.

The best efficiency gains come from better ideas, not just better hardware. A breakthrough in idea efficiency (FlashAttention, MoE routing, the Transformer itself) yields orders-of-magnitude computational gains downstream.

The power move: A quantized, pruned, distilled model on an efficient architecture can be 100× smaller with <5% quality loss.

The modern pipeline: Train big → Distill small → Quantize → Deploy

AWQ (Lin et al., 2024, arXiv:2306.00978) — the state of the art for PTQ in 2025.

The Core Insight

• Not all weights are equally important
• ~1% of weight channels carry disproportionate information
• These "salient" channels correspond to large activation magnitudes (not large weight magnitudes!)
• Naive quantization destroys these critical channels → quality collapse

The Solution

• Identify salient channels by measuring activation magnitudes on a small calibration set
• Apply per-channel scaling to protect salient channels before quantization
• Scale factors absorbed into adjacent layers — zero overhead at inference

Why this is elegant: Activation statistics (not weight statistics) determine quantization sensitivity. Now the default quantization method in vLLM and most serving frameworks.

BitNet b1.58 (Ma et al., 2024, arXiv:2402.17764)

Every weight is one of three values: {-1, 0, 1} — that's log₂(3) ≈ 1.58 bits.

How: QAT from scratch. Weights quantized via absmean: w' = Round(w / mean(|w|)) ∈ {-1, 0, 1}. Activations quantized to INT8 per-token.

Key Results

• At 3B params: matches FP16 Transformer perplexity at same model size and training tokens
• Dramatic improvements in latency, memory, throughput, and energy

The Radical Implication

• Matrix multiplication becomes integer addition — no floating point hardware needed
• Opens the door for custom silicon optimised for ternary ops
• Could enable LLM inference on hardware without an FPU

✋ Think: If BitNet works at scale, we don't need FP hardware for AI. What does that imply for chip design? For putting AI in a wristwatch?

GGUF (GPT-Generated Unified Format) — the file format that democratised local AI. Created by the llama.cpp project (Georgi Gerganov, 2023-2024).

• CPU-first inference — runs on x86, ARM, Apple Silicon, RISC-V
• Mixed quantization: different bit-widths per tensor type
• Quality tiers: Q2_K (tiny, lossy) → Q4_K_M (sweet spot) → Q8_0 (near-lossless)

The Ecosystem Built on GGUF

• Ollama — one-command local LLM (ollama run llama3.2)
• LM Studio — GUI for model exploration
• Open WebUI — ChatGPT-like interface, fully local

The Democratisation Story

• 2023: Running a 7B model required ≥$1,000 GPU
• 2025: GGUF Q4 runs 7B on a MacBook Air, a Raspberry Pi 5, or a phone

🔑 The real benchmark isn't MMLU — it's "can a grad student run this on their laptop?"

Classical:

Modern (2024-2025):

🔑 The boundary between "distillation" and "synthetic data generation" has blurred. When GPT-4 generates training data for Phi-3, that's functionally distillation.

The most important distillation result of this era. A 7B model distilled from a 671B can outperform the raw 671B on specific tasks. Let that sink in.

The details (DeepSeek-AI, Jan 2025, arXiv:2501.12948)

The Recipe

• Train DeepSeek-R1 (671B MoE) with pure RL (GRPO) to reason via chain-of-thought
• Generate ~800K reasoning traces across maths, coding, science, logic
• Fine-tune smaller base models (Qwen-2.5, Llama-3) on these traces — pure SFT, no RL on student

🔑 Proves reasoning itself is distillable. A 7B model on a laptop doing competition mathematics. The teacher's CoT is a structured curriculum — the student learns how to think.

The pattern: Teacher generates data → Student trains on it → Student rivals models 10-50× its size

"This is distillation's killer app: not matching logits, but generating wisdom."

The uncomfortable implication: The best small models are all distilled from proprietary large models. Exception: DeepSeek-R1 — open-weight teacher, open recipe.

Process Reward Models (PRMs)

• Instead of judging only the final answer (outcome reward), PRMs score each intermediate reasoning step
• Dense, step-level supervision → more informative training signal
• Enables self-improving loops: model generates reasoning → PRM scores steps → model improves
• Key to making reasoning models (o1, R1) work reliably

Constitutional AI / RLAIF (Anthropic)

• Replace expensive human preference labelling with AI-generated feedback
• Define a "constitution" (principles the model should follow)
• The AI critiques and revises its own outputs against these principles
• Generates preference pairs automatically → train with DPO/RLHF
• Scales supervision without scaling human annotation costs

Connection to distillation: Both PRMs and Constitutional AI are forms of automated knowledge transfer — the system generates its own training signal, philosophically distillation from encoded principles.

FlashAttention (recap for serving context)

• v1 (Dao et al., 2022): 2-4× speedup, O(N) memory — tile computation into GPU SRAM
• v2 (Dao, 2023, arXiv:2307.08691): 50-73% theoretical max FLOPS on A100
• v3 (Shah et al., 2024): H100 features — FP8, TMA, warp specialisation
• Now default everywhere. If you're not using FlashAttention in 2025, you're wasting money.

PagedAttention / vLLM (Kwon et al., 2023, arXiv:2309.06180)

• Problem: KV cache is variable-length and wasteful — pre-allocating for max length wastes ~60-80% of GPU memory
• Solution: Manage KV cache like virtual memory pages — allocated on demand, freed when done
• Result: Near-zero memory waste → 2-4× more concurrent requests per GPU

Continuous Batching: Dynamically add/remove requests mid-generation. No waiting for the longest sequence.

The 2025 minimum viable serving stack: vLLM/TGI + FlashAttention + PagedAttention + continuous batching + (optionally) speculative decoding.

The biggest efficiency gains come from algorithmic breakthroughs, not faster chips. Here are the key directions:

Attention Algorithms

• FlashAttention v1/v2/v3 — Tiled SRAM computation. Exact attention, 2-4× speedup, O(N) memory. Now the default everywhere.
• Linear Attention — Approximate softmax with kernel trick. O(N) time. Trade-off: quality loss for long sequences.
• Sparse Attention — BigBird, Longformer: attend to local windows + global tokens. O(N√N) or O(N).
• Local/Dynamic Attention — Sliding window (Mistral), dilated patterns. Adaptive context depending on content.

Low-Level Optimisation

• Custom CUDA kernels — Hand-optimised fused kernels for critical operations (attention, RMSNorm, rotary embeddings)
• Operation fusion — Combine multiple sequential operations into a single GPU kernel pass. Eliminates memory round-trips. Examples: fused attention+softmax, fused layernorm+residual
• Torch.compile / Triton — JIT compilation of PyTorch graphs to optimised GPU code

🔑 A single algorithmic insight (FlashAttention) delivered more speedup than two hardware generations. The lesson: invest in algorithms, not just silicon.

🍎 Low Hanging Fruit (Available Now)

• Reasoning distillation via CoT traces (DeepSeek-R1 recipe)
• Synthetic data pipelines from frontier models (Phi-3, Orca, Gemma)
• Progressive distillation for multi-scale model variants (e.g., 70B → 32B → 7B → 1.5B)
• Better distillation for local single-GPU deployment — quantization + distillation composition

🍊 Medium Hanging Fruit (Active Research)

• Reducing distillation compute cost itself — distillation is expensive; can we make it cheaper?
• Multi-teacher → single student (ensemble of diverse teachers → one versatile student)
• Distillation at same scale for generalization improvement (teacher and student same size)
• Synthetic data as distillation — the boundary between data generation and distillation has blurred
• Cross-architecture distillation (Transformer teacher → SSM student)

🔮 High Hanging Fruit (Blue Skies)

• Lottery Ticket + distillation — detect redundant connections at training time, not after
• Distill depth-by-depth as layers become "ready" — progressive layer-wise extraction
• Information-theoretic limits: What is the minimum model size to capture a given capability?
• 🆕 Invariant Algorithmic Cores (arXiv:2602.22600): If cores are invariant across random seeds, then the true information content is orders of magnitude smaller than parameter count — perfect distillation may be theoretically achievable
• Self-distillation: A model improving by learning from its own best outputs (connects to RL)

A 7B model doing 85% of a 671B. The capability floor is rising every month. Where does it stop?

Since 2020, the dominant recipe: More parameters → better. More data → better. More compute → better.

This works. And that is precisely the problem.

• Funding flows to scale, not novelty
• Researchers optimise for GPT-N+1 instead of questioning the architecture
• Hardware roadmaps ossify around dense matrix multiply
• Academic labs cannot compete → the field becomes industrially captured

What the scaling laws actually say: They describe one architecture family on one data type. They say this is predictable, not optimal.

Signs of Diminishing Returns (2024-2025)

• GPT-4 → GPT-4o: modest improvements despite enormous investment
• DeepSeek-V3 achieved GPT-4-class with ~$6M training by being smarter, not bigger
• Biggest gains from better data, better algorithms — not bigger models

The marathon analogy: Runner A consumes 10,000 cal/day but trains sloppily. Runner B consumes 2,000 cal/day but trains perfectly. Runner B wins — not more resources, but intelligently used resources.

Current ML metrics (MMLU, Arena Elo) all measure capability-at-any-cost. What if we measured differently?

• 2 billion smartphones can run 3B models today — more aggregate compute than any cloud
• Data centre energy is a geopolitical and environmental issue — Microsoft buying nuclear reactors for AI
• Many applications need AI where there's no cloud: rural clinics, submarines, developing countries

The brain's trick: Compresses experience into reusable representations, maintains persistent state, builds hierarchical models — all on 20W, 3 bananas/day.

The floor of small model capability is rising faster than the ceiling of large models:

🔑 The frontier model's purpose is shifting. It's not the product. It's the teacher. Its job is to discover intelligence; distillation's job is to deliver it to every device on earth.

A personal AI agent running entirely on your hardware:

Runtime: llama.cpp / MLX / ExecuTorch  ·  Power: 5-15W  ·  Latency: <100ms/token  ·  Cost: $0/query  ·  Privacy: 100% local

This is not "ChatGPT on your laptop." This is an AI that knows your documents, learns your patterns, and never phones home.

🔒 Privacy — Data never leaves your device. Period. Medical records, financials, personal conversations — never sent to a third party. GDPR compliance is trivial.

⚡ Latency — No network round trip. 10ms local vs 500ms+ cloud. Critical for interactive use (coding assistants, autocomplete), robotics, real-time control.

💰 Cost — Zero marginal cost per query. A $500 device running 24/7 costs less than moderate API bills. Scales to billions of users without billions in infrastructure.

🌐 Offline — Planes, rural clinics, submarines, developing countries, bad WiFi. Resilient to cloud outages, API deprecation, provider pricing changes. Sovereign — no company can revoke your access.

🎯 Personalisation — Fine-tune on your data. Adapt to your style. Learn your domain. Compound returns: your agent improves the longer you use it.

"The best AI is the one that's always there, always private, and always yours."

① Better Learning Signals

• Process reward models → dense, step-level supervision
• Self-play and verification loops → models that improve by checking their own work
• Curriculum learning → teach efficiently, like a tutor, not a fire hose

② Architecture Innovation

• SSMs (Mamba) → linear-time sequence modelling, natural for streaming
• MoE → activate only relevant parameters (DeepSeek-V3: 671B total, 37B active)
• Neuromorphic / event-driven computation → process only when something changes

③ Distillation as the Bridge

• Frontier models are teachers, not products — their purpose is to bootstrap smaller models
• DeepSeek-R1 proved reasoning is distillable; what about planning? Creativity? Common sense?

④ Efficiency Through Fundamental Insight

• BitNet b1.58: we may not need floating point — just {-1, 0, 1}
• Speculative decoding: 3× free speedup with the right small model
• Sparsity: the brain is >99% sparse at any moment

Fundamental 🧠

• What are the information-theoretic limits of distillation? Minimum model size for a given capability?
• Which capabilities compress cheaply (factual recall) vs expensively (reasoning, planning)?
• Are there phase transitions in model size below which capabilities vanish?
• 🆕 Invariant Algorithmic Cores (arXiv:2602.22600): If trained models converge to the same functional core regardless of seed, what does this mean for the true complexity of intelligence?
• 🆕 Koopman Spectral Profiling (arXiv:2602.22988): Can we treat the forward pass as a dynamical system and predict (or prevent) training instabilities before they happen?

Architectural 🏗️

• Can SSMs match Transformers at 1/10th the compute for general language?
• Modular models: plug in domain-specific modules without retraining?
• Efficient continual learning without catastrophic forgetting?
• 🆕 RMT-guided architecture (arXiv:2602.22345): Can we use Random Matrix Theory to design architectures that are inherently less redundant — building lean models from the start?

Systems 📱

• On-device fine-tuning with limited memory?
• Federated learning for local agents to improve collectively without sharing private data?
• Hardware co-design for ternary/sparse computation? (BitNet implies radically different chips)

Evaluation 📊

• Benchmarks for efficiency: not "score at any cost" but "score at 5W / at 1B params / at $0 API"
• How to evaluate personalisation quality?

These are not incremental questions. They define a research programme for a generation.

After this lecture, do one thing. Not tomorrow. Tonight. Before you go to sleep.

# Install Ollama (macOS/Linux — one command)
curl -fsSL https://ollama.ai/install.sh | sh

# Run a 7B reasoning model on your laptop
ollama run deepseek-r1:7b

# Ask it to solve a maths problem. Watch it think.

Then reflect:

• This model was distilled from a 671B-parameter system
• It's running on your hardware, with no internet, no API key, no cost per query
• Two years ago, running a state-of-the-art reasoning model locally was impossible. Today it's one command
• The gap between frontier and local is closing faster than anyone predicted

The best time to start building local AI was two years ago. The second best time is right now, tonight, before you close this laptop.

Architecture & Training

• "Attention Is All You Need" — Vaswani et al. (2017) — arXiv:1706.03762
• "Training Compute-Optimal LLMs" (Chinchilla) — Hoffmann et al. (2022) — arXiv:2203.15556
• "RoFormer: Rotary Position Embedding" — Su et al. (2021) — arXiv:2104.09864
• "GLU Variants Improve Transformer" — Shazeer (2020) — arXiv:2002.05202

Alignment

• "Direct Preference Optimization" — Rafailov et al. (2023) — arXiv:2305.18290
• "Training LMs to Follow Instructions" (InstructGPT) — Ouyang et al. (2022)

Efficiency & Compression

• "FlashAttention-2" — Dao (2023) — arXiv:2307.08691
• "GPTQ" — Frantar et al. (2022) — arXiv:2210.17323
• "AWQ" — Lin et al. (2024) — arXiv:2306.00978
• "BitNet b1.58" — Ma et al. (2024) — arXiv:2402.17764
• "SparseGPT" — Frantar & Alistarh (2023) — arXiv:2301.00774

Distillation & Small Models

• "Distilling the Knowledge in a Neural Network" — Hinton et al. (2015) — arXiv:1503.02531
• "The Lottery Ticket Hypothesis" — Frankle & Carlin (2018) — arXiv:1803.03635
• "DeepSeek-R1" — DeepSeek-AI (2025) — arXiv:2501.12948
• "Phi-3 Technical Report" — Microsoft (2024) — arXiv:2404.14219
• "Mamba" — Gu & Dao (2023) — arXiv:2312.00752

2026 Frontier Research

• "Invariant Algorithmic Cores" — (2026) — arXiv:2602.22600 — models discover unique algorithmic solutions
• "Residual Koopman Spectral Profiling" — (2026) — arXiv:2602.22988 — predicting training instability
• "Structure & Redundancy via RMT" — (2026) — arXiv:2602.22345 — spectral pruning via Random Matrix Theory

Section 1 — LLM 101: The Transformer: LEGO bricks stacked into a sandwich, trained by predicting the next word trillions of times. The 2025 recipe: RMSNorm + SwiGLU + RoPE + GQA. Deceptively simple. Wildly powerful — and fundamentally a meta-learning system (Antreas's hot take!). Plus: how to train the beast at scale (FSDP, Koopman spectral profiling) and fine-tune it efficiently (LoRA, QLoRA, DPO).

Section 2 — Compression & Distillation: The scaling monoculture is unsustainable. But we have tools: pruning (carve the statue — now with RMT-guided spectral methods), quantization (approximate π), distillation (teach the student). Combined: 100×+ compression. DeepSeek-R1 proved even reasoning is distillable. And 2026 research on Invariant Algorithmic Cores suggests perfect distillation may be theoretically achievable.

Section 3 — Smarter, Not Bigger: Small models trained with distillation now rival giants from 12 months ago. The future is local, private, efficient. Intelligence per watt, not intelligence per dollar.

Section 4 — Resources: Tools, papers, and the big picture diagrams to navigate the field.

The arc of this lecture:

• Beginning: We started with a gap — 20W vs 10,000W. A brain running on fruit vs a cluster running on a power station.
• Middle: We learned the tools to close that gap — pruning, quantization, distillation, speculative decoding, architectural innovation.
• Resolution: The gap is closing. You're watching it happen in real time. A 7B model on your laptop today does what a data centre couldn't do two years ago.

The brain runs on 3 bananas. AI runs on 1,500. Closing that gap is the most important research challenge of our generation.