AssociationAI / AI Literacy
AssociationAI team Published

Article

Beginner

robots.txt Asks AI Crawlers to Stay Out. It Doesn't Make Them.

Adding AI crawler lines to robots.txt sends a request, not a lock. What two research teams found, why member pages need a login, and what to tell your board.

AI Literacy Websites Publications Member Data

The request usually lands a few days before a board meeting: can we stop AI companies from taking our journal archive? Someone answers that it’s a quick fix, adds a few lines to the website’s robots.txt file and reports the job done. The lines are worth adding, but the report overstates what they do, and the gap matters most for the pages your members pay to see.

A teaching example to keep in mind

The Invented Riverbend Association of Map Librarians publishes a quarterly journal, keeps a public resource library of map-cataloging guides and runs a members-only directory. Its executive director asks the communications manager to “block AI” from the journal archive before the spring board meeting. This is a teaching example, not a case study.

The manager finds a list of AI crawler names on a forum, pastes them into robots.txt with a line telling each one to skip the journal folder, and while she’s there adds the member directory too, to be safe. Two of those choices are reasonable. One of them makes things slightly worse. To see which, it helps to know what the file actually is.

What the file is, in the researchers’ words

robots.txt is a plain text file at the top of a website that tells automated visitors which parts of the site the owner would rather they skip. The team behind Somesite I Used To Crawl, a preprint from UC San Diego and the University of Chicago, describes it plainly: “The most prominent and popular tool is robots.txt, a voluntary (and non-enforceable) protocol that enables site owners to specify crawling restrictions.”

The same paper explains why it works at all: “As an honor-based system, compliant crawlers follow its directives, but adherence is not mandatory.” Well-behaved crawlers read the file and do what it says. Nothing makes the others read it.

Consent in Crisis, a preprint from the MIT-led Data Provenance Initiative, gives the history of why so many crawlers do honor it: “While it is not legally enforceable, it is respected by all major search engines, as it prevents website servers from getting overloaded by crawlers, it allows websites to signal pages that are undesirable to crawl (for example, calendar sites that could lead to infinite loops), and by respecting it, crawlers disincentivize adversarial tactics designed to impede crawlers.” That is the authors’ description of a long-standing custom, not a legal finding.

So the honest summary for your director is that robots.txt is a request, not a lock. It’s a sign on the door. Most of the visitors you’d expect to read signs will read it.

What one team found when it tested crawlers

The Somesite team didn’t stop at describing the file. It set up its own tests to see which AI crawlers followed robots.txt, and it also asked professional artists what they wanted from tools like it. In the abstract the team says “we find strong demand for tools like robots.txt, but significantly constrained by critical hurdles in technical awareness, agency in deploying them, and limited efficacy against unresponsive crawlers.”

One result is easy to repeat too broadly, so read it with its limits. In the Somesite team’s tests, “the majority of AI assistant crawlers do not respect robots.txt.” That finding is about one kind of crawler, the kind that fetches pages when someone asks an AI assistant a question, in the crawlers this team tested at the time it tested them. It is not a finding that all AI crawlers ignore the file, and the paper reports a different picture for other kinds. It’s still enough to tell the director that some of the visitors she’s worried about may never look at the sign.

There’s a second limit that matters for a journal archive. The same paper notes that “While companies claim that the content of a website retrieved by AI search crawlers is not directly used for training, the user or owner of a website cannot enforce nor verify this claim.” Even when a crawler does fetch your pages for a stated purpose, you have no way to check what happens next.

Why mixed signals weaken the request

Many association sites say one thing in their terms of use and another in robots.txt, or say nothing in one of them. The Consent in Crisis team looked at this across a large set of websites. Over a single year, from 2023 to 2024, the authors found a rapid rise in restrictions that sites placed on crawling, as reported in Consent in Crisis. They also found the signals often disagreed: “The inconsistencies and omissions between robots.txt and terms of service pages suggest a data ecosystem ill-equipped to signal or enforce preferences.” For the 2,000 web domains that supply the most text to C4, one of the training datasets it studied, Longpre et al.’s preprint compared terms pages with robots.txt files, crawled through April 2024. Its figures are shares of that sample’s text, not counts of websites. Consent in Crisis reports: “We also see many websites write anti-crawling terms but have no robots.txt file (35.1%), or have no ToS but a restrictive robots.txt (20.3%) that disallows at least some crawlers.” In that sentence, ToS means terms of service.

For an association, the fix is ordinary housekeeping. If your terms of use say the journal shouldn’t be used to train AI systems, your robots.txt should ask the same thing, and the other way round. Whether either statement holds up against a particular company is a question for counsel. This isn’t legal advice.

Where the teaching example went wrong

Back at Riverbend, asking crawlers to skip the journal archive was fine, and so was being honest that it’s a request. The member directory is the problem. It’s a private page, and putting its address in robots.txt does nothing to protect it, since the file is public and anyone can read it. If the directory has no member login, it’s open to every crawler that ignores the file. If it does have a login, the login is doing the protecting, and the robots.txt line only advertises where the private pages live.

We think the rule that follows is simple: private member pages need a login, and they don’t belong in robots.txt at all. robots.txt is for public pages you’d prefer crawlers skip, like a print view, a site search page or an archive you’ve decided to keep out of AI tools.

For the other direction, when staff paste content into a tool instead of a crawler fetching it, see what an AI system does with what you type.

What to tell the board

Here’s wording the communications manager could use, adapted to your own site: “We’ve asked AI crawlers not to collect the journal archive. Compliant crawlers should follow that request, but one research team’s tests found that most AI assistant crawlers it checked did not. We can’t verify what happens to pages that are collected. Our members-only pages are protected by a login, not by robots.txt, and we’ve checked that each one has one.”

That last sentence is the one worth earning before the meeting. Our companion tutorial on checking your robots.txt lines before you count on them with AI crawlers walks through the file line by line, and its AI crawler rule check lab lets you practice on invented lines first.

Diagram showing three kinds of visitor to an association website: a compliant crawler that reads robots.txt and skips the listed public pages, a crawler that never reads the file and collects pages anyway, and a member login that keeps everyone without an account out of private pages.

Sources (2)