Claude Real Video: let Claude actually watch a video

My notes

Summary

claude-real-video is a local, open-source CLI (MIT license) that turns any video URL or file into a folder of meaningful still frames plus a transcript, so a text-only model like Claude can effectively “watch” it. Instead of sampling at a fixed interval, it detects scene changes, drops near-duplicate frames with a sliding-window comparison, and transcribes audio via existing subtitles or Whisper as a fallback, all without uploading the video anywhere.

Key Insight

• Most “AI watches video” tools aren’t actually watching: pasting a YouTube link into ChatGPT reads the transcript only; Claude doesn’t accept video files at all; even Gemini, which does ingest video natively, uploads it to Google and samples at a fixed 1 fps, so fast cuts between samples are missed.
• Fixed-interval sampling wastes budget on static content and misses content on dynamic content: a 10-minute static slide produces ~600 near-identical frames under 1 fps sampling; claude-real-video’s dedup collapses that to effectively 1 frame, while a fast-cut reel still gets every visual change caught.
• Frame selection combines scene-change detection with a density floor (--fps-floor, default 1.0s) so both slow screencasts and fast cuts are covered from a single pass.
• Deduplication uses actual pixel difference on downscaled RGB (not a perceptual hash, which the author notes “goes blind” on flat colors and equal-luma hue shifts), compared against a sliding window of the last N kept frames (--dedup-window, default 4), so an A-B-A cutaway doesn’t resend a shot the model already saw.
• Transcription prefers existing subtitles: if a local file has a sidecar .srt/.vtt or an embedded subtitle track, that’s used directly (faster, more accurate). Whisper is only invoked as a fallback when no subtitles exist.
• Everything runs on the local machine; nothing is uploaded to a cloud API, which matters for private or login-gated video (a Netscape cookie file can be passed for authorized access to gated content).
• A --report flag writes an HTML visualization of every keep/drop decision with its diff percentage, useful for tuning --scene and --dedup-threshold sensitivity on a per-video basis.
• Hard cap --max-frames (default 150) keeps output bounded regardless of video length, which matters for controlling downstream LLM context cost.