Robots.txt Guide: What It Is, Examples and How to Block AI Bots

Robots.txt is a tiny text file with a big job: it tells search engines and other bots which parts of your website they may crawl. Get it right and crawlers focus on your important pages. Get it wrong and you can accidentally hide your whole site from Google. This guide explains robots.txt in plain English, with ready-to-use examples.

Free tool
Create your robots.txt
Use a WordPress or store preset, block AI crawlers, add your sitemap and test any URL – free robots.txt generator.
Open Robots.txt Generator →

What robots.txt does (and does not do)

Robots.txt lives at the root of your domain: https://yourdomain.com/robots.txt. Well-behaved bots read it before crawling.

  • It controls crawling – which URLs bots may request.
  • It does not control indexing. A blocked page can still appear in Google if other sites link to it (usually without a description).
  • It is not security. The file is public, and bad bots ignore it.

To keep a page out of search results, use a noindex meta tag and allow crawling, so Google can see the tag.

The basic syntax

LineMeaning
User-agent: *The following rules apply to all bots
User-agent: GooglebotRules only for Google’s crawler
Disallow: /admin/Do not crawl URLs starting with /admin/
Allow: /admin/help.htmlException inside a disallowed folder
Disallow: /*.pdf$Block URLs ending in .pdf (* = anything, $ = end)
Sitemap: https://…Location of your XML sitemap

Rules are grouped by user-agent. A bot follows the most specific group that matches its name; for Google, the longest matching rule wins when Allow and Disallow conflict.

Ready-to-use examples

Allow everything

User-agent: * then Disallow: (empty) and your Sitemap: line.

A good WordPress robots.txt

  • User-agent: *
  • Disallow: /wp-admin/
  • Allow: /wp-admin/admin-ajax.php
  • Disallow: /?s= and Disallow: /search/ (internal search results)
  • Sitemap: https://yourdomain.com/sitemap_index.xml (or /wp-sitemap.xml)

Do not block /wp-content/ or /wp-includes/ – Google needs your CSS and JavaScript to render pages.

Block a staging site completely

User-agent: * then Disallow: / – but better, protect staging sites with a password.

How to block AI crawlers

Many site owners now choose whether AI companies may use their content for training. Add a separate group for each bot:

User-agentCompany / purpose
GPTBotOpenAI – model training
ClaudeBotAnthropic
Google-ExtendedGoogle – Gemini training (does not affect Search)
CCBotCommon Crawl dataset
PerplexityBotPerplexity
Applebot-ExtendedApple – AI training

Each group looks like User-agent: GPTBot followed by Disallow: /. Blocking these does not affect your Google or Bing rankings. Reputable AI crawlers follow robots.txt; others may not.

Common mistakes

  • Disallow: / left on after launch – the classic way to vanish from Google.
  • Blocking CSS/JS files – Google cannot render your pages properly.
  • Using robots.txt to hide private pages – it advertises them instead. Use logins.
  • Blocking pages you want removed – Google cannot see the noindex tag.
  • Wrong location – it must be at the root, not in a subfolder; each subdomain needs its own file.

How to test it

  • Open yourdomain.com/robots.txt in a browser.
  • Check Settings › robots.txt in Google Search Console for fetch errors.
  • Use the URL tester in our Robots.txt Generator to see whether a specific URL is allowed for Googlebot or GPTBot.

Then make sure your sitemap is complete – see how to create a sitemap.

FAQ

Where is my robots.txt file?

At https://yourdomain.com/robots.txt. WordPress serves a virtual one if no file exists.

Does robots.txt stop a page from being indexed?

No. Use a noindex meta tag for that, and keep the page crawlable.

How do I block ChatGPT from my website?

Add “User-agent: GPTBot” with “Disallow: /”. OpenAI’s other agents, such as ChatGPT-User, have their own names.

Will blocking AI bots hurt my SEO?

No. Googlebot and Bingbot are separate from AI training crawlers like Google-Extended and GPTBot.

What happens if I have no robots.txt?

Bots assume they may crawl everything. That is fine for many small sites.

Free tool
Try the free Robots.txt Generator
No signup and nothing uploaded – it runs right in your browser. Browse all 74 free tools on our Free Tools page.
Open Robots.txt Generator →

Need a website or app for your business? Talk to Techpass Master – we design and build websites and apps for clients worldwide.

WhatsApp us📞 Call now