What Google says
We read Google's robots.txt introduction on 9 October 2026. It was last updated on 10 December 2025.
- A robots.txt file tells search engine crawlers which URLs they can access on your site. It is mainly used to manage crawler traffic, so a site is not overloaded with requests.
- It is not a way to keep a web page out of Google. A blocked URL (web address) can still appear in results, without a description, if other pages link to it.
- To keep a page out of Google, Google points to the noindex rule or a password.
- Not every crawler follows robots.txt. Googlebot and other respectable crawlers do. Others might not.
- Different crawlers may read the rules in slightly different ways.
How the file is written
Google's page on creating a robots.txt file was last updated on 21 November 2025. It says the file must be named robots.txt and placed at the root of the site. It must be saved as plain text in UTF-8 (a standard way to store text). A file in a subfolder will not work. Its example looks like this:
User-agent: GooglebotandDisallow: /nogooglebot/keep Googlebot out of that folder.User-agent: *andAllow: /let all other crawlers visit the whole site.Sitemap: https://www.example.com/sitemap.xmltells crawlers where your sitemap is. The address must be a full URL.
User-agent names the crawler a group of rules is for. Disallow blocks a file or folder. Allow overrides a Disallow for something inside it. Google suggests opening the file in a private browser window to confirm it is public. Then check it with the robots.txt report in Search Console.
A made-up example
A pump maker in Coimbatore has a login area at /admin/. It also has a search page at /search/ that makes thousands of useless addresses. The developer blocks both in robots.txt, so crawlers spend their visits on the real product pages.
Now picture a different slip. A developer blocks the whole site in robots.txt while it is being built. The developer then forgets to remove the rule at launch. A rule that blocks the path / covers every page. Crawlers cannot visit the site, and the business wonders why it is not showing in search. We suggest opening the file in a browser after every launch.
Common mistakes
- Using robots.txt to hide a page. Google says not to.
- Blocking a page and then adding noindex to it. Google warns that mixing crawling and indexing rules can make some rules cancel others.
- Writing the file in a word processor, which can add characters crawlers cannot read.
- Putting the file in a subfolder instead of the root.
- Leaving a launch-day block in place.
- Forgetting that AI (artificial intelligence) features follow the same file. Google's AI features page says robots.txt rules for Googlebot are how site owners manage Google's access. It points to a separate token, Google-Extended, for limiting some AI uses in other Google systems.
Related terms
- Sitemap: a file that lists the pages you want crawlers to find.
- Crawl budget: how many pages a search engine will fetch from your site in a given time.
- Indexing: a search engine saving a page so it can be shown in results.
- Technical SEO (search engine optimisation): the set-up work that lets search engines read your site.
- Canonical tag: a tag that names the main address when several pages show the same content.
To have your file and site set-up checked, see our technical SEO work, or begin with an SEO audit.
Sources
- Google Search Central, "Introduction to robots.txt" (developers.google.com/search/docs/crawling-indexing/robots/intro), last updated 10 December 2025, read 9 October 2026.
- Google Search Central, "How to write and submit a robots.txt file" (developers.google.com/search/docs/crawling-indexing/robots/create-robots-txt), last updated 21 November 2025, read 9 October 2026.
- Google Search Central, "AI features and your website" (developers.google.com/search/docs/appearance/ai-features), last updated 10 December 2025, read 9 October 2026.
