URL crawl limit by folder option
1
votes
URL crawl limit by url first path, where you can specify the maximum number of crawled URLs per folder.
This would allow crawling of larger complex sites, getting a sample form all page types.