Irresponsible crawling speed
5
votes
When crawling one's own site (as verified by GA/search console), allow higher crawl limits. The 25/sec limit should be lifted.