Set cookies for crawl
13
votes
Sometimes preprod environments can be protected by cookies : the user is redirected if he doesn't have the cookie. Some other websites can be concerned (big SEO problem if cookies are mandatory to navigate). Therefore it would be a good idea to add a feature that can let us set some custom cookies to allow the crawl.
FYI, Screaming Frog's "allow cookie"'s feature is not enough since we need to set custom cookies. I had a good experience with Apify with this feature.
Thank you !