opencode-senses Adds Local Vision to Text-Only Models

My notes

Summary

opencode-senses is an open-source plugin that gives text-only OpenCode coding models a vision layer, running fully local on a 6 GB GPU with no API keys. It auto-injects structured evidence (scene description, caption, exact OCR) whenever an image is attached, and exposes 13 additional tools (detect, zoom, diff, colors, reverse-search, and others) the model can call to dig deeper.

Key Insight

• Uses Moondream 2 (default) as the local vision model, which fits comfortably in 6 GB VRAM and peaks around 4.5 GB. Moondream 3.1 (9B) is supported for bigger cards but needs roughly 16 GB.
• Auto-provisions its own Python venv and downloads roughly 3.9 GB of model weights on first use via uv (10-100x faster than pip), or falls back to python3 -m venv plus pip.
• Architecture separates concerns cleanly: a TypeScript plugin runs inside the OpenCode session and talks to a Python runtime over line-delimited JSON-RPC on stdio, keeping the vision workload out of the text model’s context.
• Notable prompt-injection defense: everything read from an image is wrapped in an explicit <SENSES> “untrusted data” guard, so text like “ignore previous instructions” embedded in a screenshot is treated as evidence, not a command.
• Deterministic, non-hallucinatable tools exist alongside model-based ones: senses_colors (dominant palette and RGB) and senses_metadata (dimensions, EXIF, byte size) run without invoking the model at all, useful for ground-truth checks a vision model might get wrong.
• senses_zoom upscales a region (LANCZOS, 1-8x) and re-runs OCR, caption, or query on just that crop, recovering small text the model misses when reading a full screenshot at once.
• senses_diff produces a pixel-level change map between two images (with anti-aliasing noise filtered out) plus an optional model-generated summary, built for UI and render iteration QA loops.
• Any tool accepts an https:// URL directly as the path argument; images are downloaded verbatim (original bytes and type preserved, no re-encode) and cached locally for reuse across calls.
• Reverse image search works two ways: local perceptual-hash search across cached files (always free, no network) and an opt-in Yandex CBIR upload (no API key, but occasionally bot-blocked).