Cloudflare's Agent Readiness Score: how agent-ready is your site?

My notes

Summary

Cloudflare launched isitagentready.com, a tool that scores websites on how well they support emerging AI agent standards (discovery, content, access control, interactivity). Only 4% of the top 200,000 sites declare AI content preferences, and fewer than 15 support MCP Server Cards or API Catalogs. Cloudflare also overhauled its own developer docs using these standards and measured 31% fewer tokens consumed and 66% faster answers from AI agents as a result.

Key Insight

• Current adoption is near-zero for new standards:
  • robots.txt exists on 78% of sites, but almost none are written for AI agents
  • Only 4% declare Content-Signal preferences (ai-train/ai-input/search)
  • Markdown content negotiation (Accept: text/markdown) supported on only 3.9% of sites
  • MCP Server Cards and API Catalogs (RFC 9727): fewer than 15 sites total in a 200k-site dataset
• 80% token reduction possible by serving markdown instead of HTML, Cloudflare measured this directly on their docs
• The “grep loop” problem: massive single llms.txt files exceed agent context windows, forcing iterative keyword searches that reduce accuracy and increase latency. Fix: one llms.txt per top-level directory, root file points to subdirectories
• Hidden agent directives work: embedding a plain-text message in HTML (hidden from visual rendering) that tells LLMs to request the markdown version is a practical zero-cost trick
• Redirecting AI training crawlers away from deprecated content prevents LLMs from recommending outdated implementation patterns, a real, underappreciated SEO-equivalent problem for AI
• Only 3 agents request markdown by default (as of Feb 2026): Claude Code, OpenCode, Cursor. The rest need URL-based fallbacks like /index.md
• x402 (HTTP 402 Payment Required) is being revived for machine-to-machine payments, agents can now pay for resources autonomously at the protocol level
• Cloudflare Docs benchmark: agent answered questions 31% cheaper and 66% faster compared to unoptimized documentation sites