Crawling and categories

Sites are crawled in one of two ways, and which one applies is decided by size. The difference is the single most common source of "nothing is happening".

Under 5,000 URLs: it runs itself

The automatic pass takes the whole backlog and works through it. You do not have to do anything, and mapping is about templates rather than about getting pages fetched.

Over 5,000 URLs: you choose what matters

Above that threshold the automatic pass never starts, and the site is crawled category by category.

That is the design, not a fault. Fetching a hundred thousand pages before anyone has said which parts of the site matter is expensive for us and rude to your origin. So a large site waits until you have mapped something, and then crawls what you mapped.

The consequence catches people out: on a large site, a crawl with nothing mapped fetches nothing. The button works, the job runs, and zero pages are read — because zero pages were selected.

Mapping does not need a prior crawl

The natural assumption is that you must crawl before you can categorise, and it is wrong.

Category suggestions are derived from URL paths, which discovery already collected. A site that has never been crawled once can be mapped on day one. On a 24 000-URL site, roughly a dozen patterns typically cover essentially all of it.

So when a large site has sat untouched for months, the first question is not "what broke" — it is "did anyone map it".

The three counts, and what each one is not

Discovered — URLs we know exist. The largest number, and it says nothing about whether we have looked at them.

Mapped — URLs assigned to a category, which is to say ready to crawl. This is the one most often missing.

Crawled — URLs we have fetched and stored data for. Only these are scored.

A gap between discovered and mapped is a decision you have not made yet. A gap between mapped and crawled is work in progress.

Discover more adds URLs, not crawls

"Discover more" finds newly published URLs and adds them as discovered. They arrive unmapped, so nothing fetches them until you assign them.

Expect the counts to move while the crawl does not. That is the feature working.

The two crawl types

A metadata crawl reads the head — title, description, canonical, robots, structured data. Most of the score reads this.

A content crawl also extracts the body: headings, text, images and internal links. Keyword inference, content quality and link analysis all need it, so run one before expecting those views to have anything to say.