The robots.txt file is a text file that instructs web robots (like search engine bots) how to crawl pages on your site. One main use for it is to tell robots not to visit certain pages of your site. An error in the robots.txt file can be a big problem -- if the file is telling search engine bots not to crawl your site, it won't show up in search results.
By default, we block all bots from wp-admin/ and wp-includes/ folders, as no bot needs to access those administrative directories.
Message from Amazon
If you receive a message from Amazon about robots.txt saying that their crawler doesn't have access to your pages, please forward the message on to us so we can look into it. However, it's very likely to be a false positive -- we've had several of these recently.
Errors in Google's mobile-friendly test tool
If you run a test in Google's mobile-friendly test tool and come up with errors, this is also unlikely to be a problem. There are some pages of your site that we disallow access for security reasons, or because the bots just don't need access to them. This is the case for files within the wp-admin and wp-includes directories.
However, if you do run a test like this and see errors that don't include wp-admin and wp-includes, please take a screenshot of the results and forward it on to us at support@performancefoundry.com. We'll look into it and let you know if it's a false positive or if action needs to be taken.
Related Articles
How to Update ads.txt file for Mediavine Users
The Mediavine ad network uses an ads.txt file to enhance security and authenticate publishers' sites. Occasionally it sends out emails to clients notifying them that their ads.txt file needs updating, and the tone can make it a bit scary if you ...
Why isn’t Google Search Console indexing all my pages?
Short answer: Not all of your pages should be indexed. There are many common reasons why a page is not indexed, and should not be indexed, for example if the URL doesn’t exist, is a duplicate, or is a categories or tags page. Check out Google’s ...
Google says to fix my Adsense code
Occasionally Google sends out emails similar to the text below: "We've noticed that the ads.txt file on one or more of your sites that you monetize through this AdSense account is missing the correct publisher code. From mid-October, Google will stop ...
Issues with AMP
If you're having issues with AMP, these resources may help! (Want to know what AMP is? Check out this doc.) Validate AMP for Google Search If you’ve created AMP content and the page isn’t appearing in Google Search, there are a couple of things you ...
I got a 503 error :(
503 errors are normally sporadic and temporary. If you see one, please wait five seconds then try again. If you see one regularly or repeatedly, please send an email to support@performancefoundry.com including the time, timezone and exact URL you ...