Start crawl from linked pages

5 votes

Rather than crawl the site starting with the homepage, use most linked to pages, only crawling no external link pages after the linked pages have been crawled and internal links prioritized.

This will create a crawl of the site shaped more like Google's. Crawl depth will reflect distance from external links rather than merely distance from homepage.

Not planned Suggested by: <marquee>Russ</marquee> Upvoted: 17 Jun, '21 Comments: 1

Comments: 1