Start crawl from linked pages
Rather than crawl the site starting with the homepage, use most linked to pages, only crawling no external link pages after the linked pages have been crawled and internal links prioritized.
This will create a crawl of the site shaped more like Google's. Crawl depth will reflect distance from external links rather than merely distance from homepage.
Not planned Suggested by: <marquee>Russ</marquee> • Upvoted: 17 Jun, '21 • Comments: 1
Thanks for the suggestion Russ. We don't currently bring in any external link data, so would need to do a pile of work to even get to place where this might be feasible, unfortunately.