Websites are identifying AI visitors, opening a cloaking risk for information access
A 217-million-response scan finds routine agent signposts alongside the technical precondition for hidden machine-facing content.
Signals Corps scanned 217,097,223 Common Crawl responses and found 7,959 sites sending headers aimed at AI clients, mostly pointers to llms.txt files or MCP servers; 136 said they had classified CCBot as an AI bot. The researchers could not measure agent-only cloaking in the crawl, but demonstrated that sites can use client identifiers to serve agents information that a person checking the same URL would not see. For publishers and information services, this creates a machine-facing distribution layer that can route agents efficiently while making access, integrity and auditability depend on what servers reveal to each intermediary.