# opencode-senses Adds Local Vision to Text-Only Models

> An OpenCode plugin running Moondream locally on a 6 GB GPU, auto-injecting OCR and scene evidence wrapped in an untrusted-data guard against prompt injection.

Published: 2026-08-19
URL: https://daniliants.com/insights/opencode-senses-adds-local-vision-to-text-only-models/
Tags: coding-agents, local-ai, multimodal, prompt-engineering

---

## Summary

opencode-senses is an open-source plugin that gives text-only OpenCode coding models a vision layer, running fully local on a 6 GB GPU with no API keys. It auto-injects structured evidence (scene description, caption, exact OCR) whenever an image is attached, and exposes 13 additional tools (detect, zoom, diff, colors, reverse-search, and others) the model can call to dig deeper.

## Key Insight

- Uses Moondream 2 (default) as the local vision model, which fits comfortably in 6 GB VRAM and peaks around 4.5 GB. Moondream 3.1 (9B) is supported for bigger cards but needs roughly 16 GB.
- Auto-provisions its own Python venv and downloads roughly 3.9 GB of model weights on first use via `uv` (10-100x faster than pip), or falls back to `python3 -m venv` plus pip.
- Architecture separates concerns cleanly: a TypeScript plugin runs inside the OpenCode session and talks to a Python runtime over line-delimited JSON-RPC on stdio, keeping the vision workload out of the text model's context.
- Notable prompt-injection defense: everything read from an image is wrapped in an explicit `<SENSES>` "untrusted data" guard, so text like "ignore previous instructions" embedded in a screenshot is treated as evidence, not a command.
- Deterministic, non-hallucinatable tools exist alongside model-based ones: `senses_colors` (dominant palette and RGB) and `senses_metadata` (dimensions, EXIF, byte size) run without invoking the model at all, useful for ground-truth checks a vision model might get wrong.
- `senses_zoom` upscales a region (LANCZOS, 1-8x) and re-runs OCR, caption, or query on just that crop, recovering small text the model misses when reading a full screenshot at once.
- `senses_diff` produces a pixel-level change map between two images (with anti-aliasing noise filtered out) plus an optional model-generated summary, built for UI and render iteration QA loops.
- Any tool accepts an `https://` URL directly as the `path` argument; images are downloaded verbatim (original bytes and type preserved, no re-encode) and cached locally for reuse across calls.
- Reverse image search works two ways: local perceptual-hash search across cached files (always free, no network) and an opt-in Yandex CBIR upload (no API key, but occasionally bot-blocked).