URLs and mapping

The URLs page is where a site becomes a set of categories. Everything downstream reads from what you decide here: which pages get crawled, which share a template, and which are left out of the measurement. Crawling and categories explains why the decisions matter; this page explains the controls.

The page

The table on the left lists the site's URLs grouped by depth, fifty at a time per level. The panel on the right has four sections: Filter the view, Crawl, Suggested categories and Categories. On a narrow screen the panel becomes a second tab, Categories & Crawl.

There is no Save button for categories and filter rules. A change is saved about a second and a half after you stop editing; "Saving…" appears in the Crawl section while it happens.

What a category does

A category does two jobs. It selects pages for crawling, and it is what gets a metadata template on the Content page. A page in no category is crawled only on a small site, and templated nowhere.

A category is filled in one of three ways, chosen when you create it:

• URL pattern matches on the path, /blog/ for example. The rule is re-applied whenever new pages are found, so a category keeps filling itself.
• Page content matches on what the page says: a full-text query, re-run after every crawl. It has nothing to read until the pages are crawled.
• Hand-pick is you ticking rows in the table.

A hand-picked page stays where you put it; a content match beats a pattern match; among pattern rules the highest priority wins, and only the first rule that matches counts. CSV import is a fourth way in, covered below.

Creating a category

The plus in the Categories header asks how the category is filled, then opens the editor in the panel. Give it a name and a colour. Pages are written in takes a language code, sv or sv-SE, and matters only for hreflang; leave it empty for a category that is not one language. As you type a pattern or query the editor counts the URLs it captures and the table marks them. For a pattern the count is taken from the loaded rows, so on a large site it is a lower bound.

Suggested categories offers patterns derived from the URL paths, and a language rule when the site's URLs follow one. Accept adds the category and its rule; Dismiss only hides the suggestion until the page is reloaded. The suggestions work before the first crawl, so a large site is mapped from its paths rather than from crawled content.

Match types are contains, starts with, ends with, exact and regex, all case-insensitive. Starts-with also accepts the pattern with the protocol in front, so /shoes/ and https://example.com/shoes/ behave the same.

The crawl section

One line says where the site is, and one button does the thing that line asks for.

• No pages discovered yet → Find pages reads the sitemap and follows links.
• Categorize URLs to start crawl appears on a large site with nothing mapped; Crawls automatically, no categorizing needed on a small one.
• N pages ready to crawl → Start crawl.
• Crawling → Pause or Cancel. The bar shows progress through the mapped pages.
• N pages failed → Retry resets them and crawls again.
• N pages found but never crawled → Crawl them fetches the pages the site has never read, and nothing else.
• Up to date → Re-crawl reads everything again, after a confirmation that says how many pages and roughly how much data that is.

Each category card also has a crawl icon, for crawling that category alone.

The menu beside the line holds the rest: Find new pages, the link depth to follow when there is no sitemap, Crawl pages not yet crawled, Re-crawl all pages, and Find new pages automatically. That last toggle is the rediscovery schedule, daily or every so many hours, and it is not the same as automatic crawling: turning it off does not stop a small site from crawling itself, and turning it on does not make a large one start.

The counts

Under the status line: N discovered · N crawled, with the sources, the categorized count and the last crawl in the tooltip. Discovered is every URL we know exists. Crawled is every URL we have read. Categorized is every URL in a category.

N not crawled is the pages that have never been fetched. It is not simply discovered minus crawled, because rediscovery puts a page back in the queue when its sitemap date changes, and such a page has been crawled even though it is waiting again.

Searching and filtering

The search box has two modes. URL filters the loaded rows by URL, title and category as you type. Content searches the text of every crawled page on the site and shows a snippet under each match; pages not yet crawled cannot match. From a content search, Turn these into a category opens the editor with the query filled in.

The Filter section narrows by language, when the site has more than one, and to Only mapped URLs. Clicking a category card filters the table to that category, and the Uncategorized row to pages in none. The columns button shows the columns hidden by default: source, HTTP status and crawl status, which the dot beside each URL already summarises.

Working with rows

Expanding a row shows what the crawl read from the page and, with Search Console connected, the queries it ranks for. The category cell assigns or removes a category in place. The row menu opens the page, copies the URL, re-crawls it, or deletes it.

Ticking rows opens a bar for the selection: assign a category, re-crawl, export as CSV, or delete. Assigning and re-crawling take up to 500 pages per request.

Delete does not delete. It excludes the page from this site: it leaves every list and count at once and stays out of the score. The crawl data itself is untouched.

System pages

When the crawl finds pages that are not really pages, a card above the table groups them: admin and login pages, API and system files, feeds, cart and checkout, internal search results, WordPress ID links, tracking parameters, and files rather than pages. They count toward the score until excluded, but they cannot be optimised.

Every group is ticked; untick what you want to keep, then Exclude. The toast offers Undo for eight seconds. Hiding the card keeps it hidden until new system pages appear.

CSV in and out

The panel header exports every URL with its category, title, description and the other SEO fields; the import dialog offers the same file as a starting point, or a blank template. Edit it in a spreadsheet and import it back.

Two columns are required, url and category; the rest are optional. Every row must match a URL the site already knows, on its path, so http and https, with and without www, line up. A category that does not exist yet is created. Fill in blanks writes only the non-empty cells; Replace everything treats the file as the truth and blank cells clear the field. Preview first: the dialog says how many rows will import, which categories are new, which rows will be skipped, and warns when two category names look like typos of each other. Files are capped at 10 MB and 5,000 rows.

Titles and descriptions imported this way become the pages' own values, which the category template then leaves alone; Templates and publishing explains how to hand them back.

When a crawl reads nothing

The crawl finished and the toast says why:

• Every discovered URL has already been crawled. Nothing to do.
• URLs are waiting. Map URLs to categories to choose what gets crawled. A large site with nothing mapped.
• Nothing waiting in this category. The category you started from is empty; the count says where the waiting pages are.
• A site this size should crawl automatically. The automatic pass did not run. Nothing for you to fix, but let us know.

A crawl can also be refused outright when it would write more than one job is allowed to. Crawl a category at a time instead.

Arriving from the dashboard

A link from a dashboard finding opens this route as a plain filtered list with a back link and no panel. It is the same URLs, narrowed to one issue; the full page is one click away.