AI & the web
Crawler
A program that visits URLs to discover or retrieve content.
In everyday use
A search crawler follows a link to a public dictionary entry.
What to keep in mind
Crawling, indexing and showing a result are separate steps.
Also searched as: web crawler, bot
Further reading
Original explanation and example by Perspectism. Further reading may cover the wider subject, rather than this exact wording. Edition updated 7 October 2026.
More in ai & the web
Page updated .