{"id":43837,"date":"2024-06-24T15:53:28","date_gmt":"2024-06-24T19:53:28","guid":{"rendered":"https:\/\/mjtsai.com\/blog\/?p=43837"},"modified":"2025-08-06T14:52:33","modified_gmt":"2025-08-06T18:52:33","slug":"ai-companies-ignoring-robots-txt","status":"publish","type":"post","link":"https:\/\/mjtsai.com\/blog\/2024\/06\/24\/ai-companies-ignoring-robots-txt\/","title":{"rendered":"AI Companies Ignoring Robots.txt"},"content":{"rendered":"

Mark Sullivan<\/a>:<\/p>\n

The AI search startup Perplexity is in hot water in the wake of a Wired<\/em> investigation<\/a> revealing that the startup has been crawling content from websites that don’t want to be crawled.<\/p>

[…]<\/p>

“Perplexity is not ignoring the Robot Exclusions Protocol and then lying about it,” said Perplexity cofounder and CEO Aravind Srinivas in a phone interview Friday. “I think there is a basic misunderstanding of the way this works,” Srinivas said. “We don’t just rely on our own web crawlers, we rely on third-party web crawlers as well.”<\/p>

Srinivas said the mysterious web crawler that Wired<\/em> identified was not owned by Perplexity, but by a third-party provider of web crawling and indexing services. Srinivas would not say the name of the third-party provider, citing a Nondisclosure Agreement. Asked if Perplexity immediately called the third-parter crawler to tell them to stop crawling Wired<\/em> content, Srinivas was non-committal. “It’s complicated,” he said.<\/p>

Srinivas also noted that the Robot Exclusion Protocol, which was first proposed in 1994, is “not a legal framework.” He suggested that the emergence of AI requires a new kind of working relationship between content creators, or publishers, and sites like his.<\/p><\/blockquote>\n\n

Nick Heer<\/a> (Mastodon<\/a>, Hacker News<\/a>):<\/p>\n

\n

Srinivas is creating a clear difference between laws and principles because the legal implications are so far undecided, but it sure looks unethical that its service ignores the requests of publishers — no matter whether that is through first- or third-party means.<\/p>\n<\/blockquote>\n\n

Tim Marchman<\/a>:<\/p>\n

Earlier this week, WIRED published a story<\/a> about the AI-powered search startup Perplexity, which Forbes has accused<\/a> of plagiarism. In it, my colleague Dhruv Mehrotra and I reported that the company was surreptitiously scraping, using crawlers to visit and download parts of websites from which developers had tried to block it, in violation of its own publicly stated policy<\/a> of honoring the Robots Exclusion Protocol.<\/p>

[…]<\/p>

After we published the story, I prompted three leading chatbots to tell me about the story. OpenAI’s ChatGPT<\/a> and Anthropic’s Claude<\/a> generated text offering hypotheses about the story’s subject but noted that they had no access to the article. The Perplexity chatbot produced a six-paragraph, 287-word text<\/a> closely summarizing the conclusions of the story and the evidence used to reach them. (According to WIRED’s server logs, the same bot observed in our and Knight’s findings, which is almost certainly linked to Perplexity but is not in its publicly listed IP range, attempted to access the article the day it was published, but was met with a 404 response. The company doesn’t retain all its traffic logs, so this is not necessarily a complete picture of the bot’s activity, or that of other Perplexity agents.) The original story is linked at the top of the generated text, and a small gray circle links out to the original following each of the last five paragraphs. The last third of the fifth paragraph exactly reproduces a sentence from the original: “Instead, it invented a story about a young girl named Amelia who follows a trail of glowing mushrooms in a magical forest called Whisper Woods.”<\/p>

This struck me and my colleagues as plagiarism.<\/p><\/blockquote>\n\n

Kali Hays<\/a> (via John Voorhees<\/a>):<\/p>\n

\n

OpenAI and Anthropic have said publicly they respect robots.txt and blocks to their web crawlers.<\/p>\n

Yet, both companies are ignoring or circumventing such blocks, BI has learned.<\/p>\n<\/blockquote>\n\n

Katie Paul<\/a>:<\/p>\n

TollBit said its analytics indicate “numerous” AI agents are bypassing the protocol, a standard tool used by publishers to indicate which parts of its site can be crawled.<\/p>

“What this means in practical terms is that AI agents from multiple sources (not just one company) are opting to bypass the robots.txt protocol to retrieve content from sites,” TollBit wrote. “The more publisher logs we ingest, the more this pattern emerges.”<\/p><\/blockquote>\n\n

Previously:<\/p>\n