The term used for the tools that search engines automatically send out to find websites, record them and index them within their databases. Also known as robots or spiders. Some crawlers will only visit the home page of a website while others may 'deep crawl' and index many sub-pages, depending on the structure of the site.

Many webmasters seem unaware that crawlers are not able to view images, company logos, Flash presentations and many other facets of their web pages. Crawlers strictly see the source code of a page - they "read" the source code, making a record of what content is contained there and indexing it.

Probably the most famous crawler is Googlebot.