Apple Silicon LLM Tracker Surfaces Silent Correctness Bugs

My notes

Summary

This is a live, query-driven tool that maps which open-weight LLM, image, and audio models actually run well on a given Apple Silicon Mac, and which serving engine (MLX, llama.cpp, Ollama, LM Studio, vLLM, ds4) to trust for each one. Its core value isn’t the model list. It is the twice-daily-scraped open-issue tracker per model-per-engine pairing, which surfaces silent correctness bugs, meaning wrong output rather than crashes, that benchmark numbers never show.

Key Insight

• Google ships its own QAT GGUF for Gemma 4 (17.7 GB q4_0, plus 1.2 GB mmproj for vision). A quantization-aware-trained build from the model’s own authors loses less quality than the usual post-hoc 4-bit conversion most communities publish. Per the tracker it posts the highest tool-use score on the page (tau2-Bench 86.4%).
• Gemma 4’s headline 256k context window has a quality bug, not a crash. Sliding-window attention is reported to silently drop earlier context. That’s worse than an error because nothing tells you it happened, and it undercuts the exact feature, long context, the model is chosen for.
• The MLX ecosystem is not uniformly mature per-model. mlx-lm carries more open issues against Gemma 4’s architecture than any other model tracked (generation hangs at 0% CPU after prompt processing, thinking-mode turns return empty content, one variant won’t load, KV quantization blocked on sliding-window layers). The tracker’s explicit conclusion: for this specific model, GGUF and llama.cpp are the safer default, not MLX, despite Apple Silicon’s MLX-first reputation.
• Speculative decoding is a decode-speed trade that can cost throughput on agent workloads. Enabling a draft head commonly disables the prefix cache, forcing full re-prefill every turn. For agentic use, meaning long, mostly-repeated context across turns, that’s a net loss even though single-turn decode looks faster in isolation.
• A roughly 50 KB request has been reported to crash llama-server outright when serving Meta’s agentic model. 50 KB is an ordinary agent turn once a file or diff lands in the prompt, so this is a production-relevant ceiling, not an edge case.
• That model’s own benchmark profile is a mismatch warning. It leads tool-orchestration (MCP Atlas 75.5) and SWE-bench Verified (76.0) but its Terminal-Bench 2.1 score of 51.7 trails Qwen3.8-27B’s 73.0, meaning it’s the better pick for multi-tool orchestration, not raw terminal or coding work. It also posts a 28.4% prompt-injection attack-success rate on Siren AgentDojo, worth weighing before exposing it to untrusted tool output.
• GLM-4.7-Flash is the cheap, easy-mode tier: 3B active parameters out of 31B total, MIT licensed, and described as the single easiest model on the page to get running, making it the natural default for low-stakes, high-volume routing.
• DeepSeek V4 and GLM-5.2 need a purpose-built engine, not general GGUF loading. GLM-5.2 at 4-bit is 372-475 GB. ds4, a single-purpose C and Metal engine rather than a general loader, adds a disk KV cache that survives restarts, SSD expert-streaming for machines too small to hold full weights, and tensor parallelism across two Macs over Thunderbolt RDMA. It is the fastest option on the page where it applies, but only for these two published checkpoints.
• A structural mlx-lm bug appears at roughly 390 GB models. load_model() builds one giant Metal command buffer via a single mx.eval() call, which hits a command-buffer timeout and hard-aborts uncaught, relevant for anyone pushing GLM-5.2-class models through stock mlx-lm rather than ds4.
• Explicit accuracy caveats worth carrying into any “which model should I run” decision: benchmark scores are vendor or aggregator reported and not independently reproduced by the tracker, issue states are a twice-daily snapshot, and hardware throughput figures are arithmetic estimates rather than measured runs.