# Colibri Streams a 370 GB MoE Model Off SSD on 25 GB of RAM

> Colibri, a 1,300-line C engine, runs a 370 GB mixture-of-experts model on 25 GB of RAM by streaming experts off SSD at 0.1 to 1 token per second.

Published: 2026-08-25
URL: https://daniliants.com/insights/colibri-streams-a-370-gb-moe-model-off-ssd-on-25-gb-of-ram/
Tags: local-llm, mixture-of-experts, ssd-inference, on-device-ai

---

## Summary

Colibri is a new 1,300-line C inference engine that runs GLM 5.2, a 370 GB open-weight mixture-of-experts model, on a laptop with only 25 GB of RAM by streaming individual "expert" weights off an SSD instead of loading the full model into memory. It exploits the fact that MoE models only activate about 5% of their parameters (roughly 40B of the total) per token, keeping the always-used core in RAM and fetching the other 21,500+ experts (19 MB each) from disk on demand, achieving 0.1-1 token/second depending on hardware, versus being unable to run at all otherwise.

## Key Insight

- **Why this beats mmap:** llama.cpp can already memory-map oversized models, but OS-level paging is blind. It faults pages in one at a time, blocking the thread, with no idea which weights matter next. Colibri's router-aware prefetch requests all 8 needed experts asynchronously the moment they're selected, and speculatively pre-loads the *next* layer's likely experts (72% prediction accuracy) while the CPU is still crunching the current one.
- **The math that makes it possible:** GLM 5.2 has 256 experts per layer; the router picks only 8 per token (about 5% of total parameters active). A fully cold token still touches roughly 11 GB of disk reads across 75 layers, expensive, but nowhere near the 30 seconds/word a dense model would require reading 370 GB per token off disk.
- **Real throughput numbers (community-benchmarked):** budget SSD 0.1 tok/s, top PCIe 5 drive 0.28 tok/s, 128 GB Framework laptop 0.37 tok/s, M5 Max with a fast internal SSD tops out at 1 tok/s. That's the practical ceiling as of this video.
- **It gets faster with use:** Colibri logs which experts get hit most and pins those "hot" experts into any spare RAM on subsequent launches, so throughput improves the more you use a given workload.
- **Read-only workload, so SSD wear isn't the risk.** Streaming is pure reads (reads don't degrade flash); the actual risk is a cheap drive thermally throttling under sustained load, not wear-out.
- **Not alone:** the same season, Redis creator antirez shipped DS4 (a Metal/C engine doing the same trick on Apple Silicon), and both llama.cpp and vLLM have open proposals for expert-level disk caching, signals this is becoming a recognized architecture pattern (treating SSD as a memory tier), not a one-off hack.
- **Reframe the "0.1 tokens/sec" mockery:** that's still roughly 8,000 tokens/day unattended, several long frontier-quality answers daily from an offline laptop with no data leaving the building.