TIME Serves AI Bots a Different Website With Ads Built In

My notes

Summary

TIME.com serves AI crawlers (ClaudeBot, PerplexityBot, OAI-SearchBot) a stripped markdown copy of its articles roughly one twenty-third the size of the human HTML page, with advertiser-written “sponsored FAQ” content injected that no human reader ever sees. Each AI fetch is logged as a fresh, billable ad impression via a vendor called Mobian, while Googlebot still receives the same page a human gets.

Key Insight

• User-agent-based content forking, verified by direct testing: same URL, same second, 303,235 bytes of full HTML for Chrome, Safari, and Googlebot versus 13,409 bytes of markdown for ClaudeBot, PerplexityBot, and OAI-SearchBot. About 23x smaller and a different format entirely.
• Selective, not blanket, bot policy: GPTBot and ChatGPT-User (OpenAI’s training and live-fetch agents) get 406 Blocked. OAI-SearchBot, which feeds ChatGPT’s search index, gets waved through to the ad-laden markdown. TIME is choosing per-bot whether to monetize or block.
• Ad-tech headers reveal the billing model: x-mobian-impression is a fresh UUID on every single request (paired with cache-control: no-store), meaning every AI read of the page counts as a new ad impression. x-mobian-tokens shows the billing unit is tokens fed into a model, not pageviews or humans.
• Sponsored content is written to be quoted back by chatbots: on list and collection pages, injected copy for advertisers (Ally Bank, Project Management Institute) is phrased as Q&A, for example “Who is Ally Bank?” and “Which banks offer early direct deposit?”, mimicking exactly how a model would answer a user asking for a recommendation. A PMI stat claims certified project managers earn 16% more.
• Total invisibility to human readers: zero hits for “Ally Bank” or “Mobian” when loading the same pages as a person. The sponsored blocks, and the entire machine-only layer of the site, are undetectable without changing your user-agent.
• Googlebot is exempted: the ranking crawler still sees the real page, so search rankings are not directly distorted. Only assistant and answer-engine crawlers get the forked, monetized version.
• Scale context: TIME states its bot traffic already outnumbers human traffic on most days, which is likely to become common across major publishers soon.