Read the rules in your robots.txt file

  • User-agent chooses the crawler

    A group starts by naming the crawler its rules apply to. An asterisk applies to crawlers that don't have a more specific matching group.

  • Allow and Disallow name paths

    These lines allow or block crawling under the paths they name.

  • Sitemap names the sitemap address

    Give the full sitemap URL so crawlers can find the file.

Keep the file at the site's root, such as https://yoursite.com/robots.txt. Google's file guide explains the syntax.

Check for a robots.txt disallow all rule after launch

A group with User-agent: * and Disallow: / blocks crawling across the site for crawlers that use that group. Keep that rule only if you mean to block them.

Why does Google say indexed though blocked by robots.txt?

Google may find a blocked URL through links elsewhere and index the address without fetching its content. Disallow doesn't remove a page from search, and blocking access can stop Google from seeing a page's noindex rule. See Google's robots.txt introduction.

Frequently asked questions

Can robots.txt keep a page private?

No. robots.txt doesn't enforce access control. Protect private content with authentication rather than relying on a crawler's choice to obey the file.

  • SEO

    Find what's keeping your pages out of search results.

  • Sitemap

    Make your sitemap readable and investigate Search Console's fetch errors.

  • Indexability

    Find the noindex tags and other blocks that stop a page from being indexed.

  • AI crawlers

    Choose which AI crawlers may use your pages for training, search or user requests.