How Token Probability Skew Makes AI Text Detectable

My notes

Summary

The video opens with a whiteboard demo of how an LLM assigns different probabilities to near-synonyms when filling in a blank, and this skew toward high-probability words is what creates the detectable statistical pattern in AI-generated text. It then pivots to a full, on-screen build script for a local Ollama-based rewriting tool that flattens that pattern by banning AI-typical words and matching the user’s own voice, packaged as an installable Claude Code skill.

Note that the source explicitly frames the tool as removing a statistical signal only. It does not change facts and does not remove any disclosure obligation the user has to a client, platform, or law.

Key Insight

Why AI text is detectable

The presenter illustrates that when a model completes a sentence with a set of near-synonyms, it doesn’t pick uniformly. It has a skewed internal probability distribution. Consistently sampling from the same skewed distribution across thousands of words is what produces a statistically fingerprintable pattern in AI output.

The counter-tool, as built on screen

• A style-rules section bans an explicit word list and adds rules: no rhetorical questions unless the original was a question, no new intensifiers or flourish, plain words beat vivid ones, no inversions or fragments for effect, and match the original’s register exactly rather than upgrading plain text to marketing copy. Output discipline is to return only the rewritten block, with no preamble, explanation, or code fence.
• Voice anchor: ask the user to paste 2-3 paragraphs of their own writing. If provided, add a voice-anchor section to the system prompt that the rewrite must match on register and rhythm, stored in a separate prompt file so it’s editable without touching code.
• Smoke test: generate a 3-paragraph test markdown file (heading, one code block, one number, one URL), run the rewrite script on it, confirm the code block, number, and URL survive untouched, and check the similarity score stays under a 25% change threshold. Show the diff, and if the model reintroduced rhetorical questions or flourish, tighten the prompt and re-run.
• Ship as a Claude Code skill: create .claude/skills/rewrite-local/SKILL.md with trigger phrases. The skill takes a file path, runs the rewrite script against it, and reports the survival score, word counts, the fact-check result, and the rewritten text. The skill must explicitly disclose what it does and does not do.
• Finish: the model summarizes which local model was installed, where the script and prompt live, how to run it, how long a 1,500-word post took in the smoke test, and which parts were never modified.

Net technique

A local LLM, driven by a strict banned-word and register-matching system prompt plus an optional voice anchor from the user’s own writing, rewrites AI-drafted text to flatten the probability skew, while explicitly preserving code, numbers, and links.