Respect each subdomains robots.txt rules
1
votes
Hi, during the analysis of websites segmented in different subdomains, I flagged the "treat subdomain as internal" function, but the robots.txt file of the individual subdomains is not taken into account but only the one of the root domain. It would be very useful to correctly analyse this type of websites to be able to enforce the rules of the individual robots.txt of each subdomain.