Filter / sort / pagination params on category pages can balloon a crawl from thousands to millions of near-duplicate URLs. We crawl + group URLs by query shape and flag the ones that need canonical/noindex protection.
Crawls up to 60 pages from the homepage. Groups URLs by their query-string shape and classifies each group as filter / pagination / tracking / clean.