Responsible crawling: react to status 429 by throttling back or pausing
2
votes
(Perhaps offered as an option) when a server responds with a 429 (too many requests) throttle back a bit eg. fewer requests per second, or if that's not monitored any more, add delay between requests or pause briefly.
I guess the specific reaction doesn't matter as long as it's reducing server load when it complains.