AI & the web
robots.txt
A file at a website's root that gives participating crawlers instructions about fetching URLs.
In everyday use
A site allows a search crawler to fetch its public dictionary.
What to keep in mind
It is not an access-control system and cannot protect private content.
Also searched as: robots exclusion protocol
Further reading
Original explanation and example by Perspectism. Further reading may cover the wider subject, rather than this exact wording. Edition updated 7 October 2026.
More in ai & the web
Page updated .