Crawling and categories
Why a large site is crawled category by category, why the automatic crawl deliberately leaves it alone, and how to tell an unmapped site from a broken one.
Sites are crawled in one of two ways, and which one applies is decided by size. The difference decides whether a site is crawled without any action on your part.
Under 5,000 URLs: it runs itself
The automatic pass takes the whole backlog and works through it. You do not have to do anything, and mapping is about templates rather than about getting pages fetched.
Over 5,000 URLs: you choose what matters
Above that threshold the automatic pass never starts, and the site is crawled category by category.
Fetching a large site in full before any part of it has been selected would load the site without purpose. So a large site waits until you have mapped something, and then crawls what you mapped.
The consequence: on a large site, a crawl with nothing mapped fetches nothing. The button works, the job runs, and zero pages are read, because zero pages were selected.
Mapping does not need a prior crawl
Categorising does not require a crawl.
Category suggestions are derived from URL paths, which discovery already collected. A site that has never been crawled once can be mapped on day one. On a large site, roughly a dozen patterns typically cover nearly all of it.
A large site with nothing mapped stays uncrawled, so the first thing to check is whether categories exist.
The three counts, and what each one is not
Discovered: URLs we know exist. The largest number, and it says nothing about whether we have looked at them.
Mapped: URLs assigned to a category, that is, ready to crawl.
Crawled: URLs we have fetched and stored data for. Only these are scored.
A gap between discovered and mapped is a decision you have not made yet. A gap between mapped and crawled is work in progress.
Discover more adds URLs, not crawls
"Discover more" finds newly published URLs and adds them as discovered. They arrive unmapped, so nothing fetches them until you assign them.
The counts move while the crawl does not.
One crawl reads everything
A crawl fetches each page once and stores both the head and the body. The head holds the title, description, canonical, robots and structured data. The body holds the headings, text, images and internal links. The score reads mostly the head; keyword inference, content quality and link analysis read the body.
Pages crawled before the body was read show as Meta only. Re-crawl them before those views can include them.
Read next
Internal linking