# Skill Recorder Builds Agent Skills From Screen Recordings

> A Microsoft desktop app that records one on-screen work session and uses the Copilot CLI to reconstruct it as a reusable skill file or scheduled automation.

Published: 2026-08-14
URL: https://daniliants.com/insights/skill-recorder-builds-agent-skills-from-screen-recordings/
Tags: screen-recording, skills, workflow-automation, microsoft

---

## Summary

Skill Recorder is a Microsoft desktop app (macOS primary, Windows 11 supported) that records a single on-screen work session, including clicks, app and window switches, URLs, and optional spoken narration, then uses the GitHub Copilot CLI to reconstruct the session into an intent plus an ordered list of steps. From that reconstruction, one click generates a reusable `SKILL.md` procedure or a scheduled Automation that an AI agent (Microsoft Scout, Copilot Cowork, Copilot Studio) can run going forward.

## Key Insight

- **Show-once, generalize pattern**: recording yourself filling in one form teaches the agent to submit all similar forms. It does not just replay clicks, it infers intent and generalizes.
- **Native-tool preference**: generated skills and automations prefer the agent's own tools (`gh` CLI, `web_fetch`) over literal UI-click replay, making them more robust to layout changes than pure RPA-style recorders.
- **Local-first capture, cloud-only on demand**: recording, storage, frame extraction, and on-device Whisper narration transcription happen entirely locally. Nothing leaves the machine until the user explicitly clicks "Analyze", at which point the event timeline (window and doc titles, URLs, clipboard previews), extracted screen images, and narration text are sent to GitHub's cloud for Copilot processing. This is a deliberate consent gate, not just a privacy footnote.
- **Explicit secrets warning baked into the UX**: the app reminds users before every recording not to type, paste, or narrate passwords, tokens, or API keys, because clipboard previews and screen frames are captured by default.
- **Distribution model is a source release, not a signed binary**: the install script downloads a pinned Node.js runtime and builds the exact release commit locally (commit hash embedded in the install URL), with nothing installed globally. This sidesteps code-signing and notarization overhead for a fast-moving internal tool while keeping provenance verifiable, since the script can be inspected before running.
- **Whisper on-device narration** supports 99 languages via a one-time 252 MB model download, a useful data point for anyone evaluating local speech-to-text options without a cloud dependency.
- Requires a GitHub account with Copilot access. The Copilot CLI ships bundled with the app, so there is no separate CLI install step.