See which pages your robots.txt file blocks
A robots.txt file tells cooperating crawlers which parts of a site they may request. A rule meant for a test site can stop them from reading pages you want people to find.
Read the rules in your robots.txt file
User-agent chooses the crawler
A group starts by naming the crawler its rules apply to. An asterisk applies to crawlers that don't have a more specific matching group.
Allow and Disallow name paths
These lines allow or block crawling under the paths they name.
Sitemap names the sitemap address
Give the full sitemap URL so crawlers can find the file.
Keep the file at the site's root, such as https://yoursite.com/robots.txt. Google's file guide explains the syntax.
Check for a robots.txt disallow all rule after launch
A group with User-agent: * and Disallow: / blocks crawling across the site for crawlers that use that group. Keep that rule only if you mean to block them.
Why does Google say indexed though blocked by robots.txt?
Google may find a blocked URL through links elsewhere and index the address without fetching its content. Disallow doesn't remove a page from search, and blocking access can stop Google from seeing a page's noindex rule. See Google's robots.txt introduction.
Frequently asked questions
Can robots.txt keep a page private?
No. robots.txt doesn't enforce access control. Protect private content with authentication rather than relying on a crawler's choice to obey the file.
Check indexability and the sitemap behind robots.txt
SEO
Find what's keeping your pages out of search results.
Sitemap
Make your sitemap readable and investigate Search Console's fetch errors.
Indexability
Find the noindex tags and other blocks that stop a page from being indexed.
AI crawlers
Choose which AI crawlers may use your pages for training, search or user requests.