Crawl a site and produce sitemap.xml, a plain-text URL list, and a human-readable HTML index. Same crawler the auditor uses — respects robots.txt by default.
Crawls only HTML pages on the start host. May take 1–3 minutes for medium sites.