Live demo
BeginnerTry it: AI crawler rule check lab
This is the working lab from our tutorial on checking your robots.txt lines before you count on them with AI crawlers. Staff at the Invented Riverbend Association of Map Librarians plan to add six lines to their robots.txt. Each line gets STOP, FIX or PASS.
Invented organization, crawler names and pages for practice. The lab checks only the format of what you enter on this page. It does not visit any website and cannot tell whether a real crawler follows a rule.
Pick a case
Planned lines in this case
| Line | Crawler name | Rule | Path | Under a user-agent line | Staff call it private | Member login |
|---|
Rules, in order (first match wins)
- R1, STOP: staff call the page private and it has no member login, so nothing but a request keeps crawlers out.
- R0, FIX: the crawler name is not "*" and has characters other than letters, underscores and hyphens (a blank name counts).
- R2, FIX: the rule is not under any user-agent line.
- R3, FIX: the path is not empty and does not start with "/".
- R4, FIX: the page already has a member login, and listing it tells anyone where it is.
- R5, PASS: none of the above.
R0, R2 and R3 are adapted from IETF RFC 9309, Robots Exclusion Protocol (2022). R1's STOP level, R4's advice to leave guarded pages out, treating a blank name as invalid and the rule order are teaching choices for this lab, not a standard.
Result
| Line | Crawler name | Rule | Path | Verdict | Rule ID | Message |
|---|---|---|---|---|---|---|
| Pick a case. | ||||||
Not checked yet.
Plain-text record (select and copy it by hand)
Nothing checked yet.
This is a teaching tool. A PASS means a line follows the format rules checked here. It does not mean any crawler will obey it, and robots.txt never protects a private page.