Robots.txt is a tiny text file with a big job: it tells search engines and other bots which parts of your website they may crawl. Get it right and crawlers focus on your important pages. Get it wrong and you can accidentally hide your whole site from Google. This guide explains robots.txt in plain English, with ready-to-use examples.
Page Contents
What robots.txt does (and does not do)
Robots.txt lives at the root of your domain: https://yourdomain.com/robots.txt. Well-behaved bots read it before crawling.
- It controls crawling – which URLs bots may request.
- It does not control indexing. A blocked page can still appear in Google if other sites link to it (usually without a description).
- It is not security. The file is public, and bad bots ignore it.
To keep a page out of search results, use a noindex meta tag and allow crawling, so Google can see the tag.
The basic syntax
| Line | Meaning |
|---|---|
| User-agent: * | The following rules apply to all bots |
| User-agent: Googlebot | Rules only for Google’s crawler |
| Disallow: /admin/ | Do not crawl URLs starting with /admin/ |
| Allow: /admin/help.html | Exception inside a disallowed folder |
| Disallow: /*.pdf$ | Block URLs ending in .pdf (* = anything, $ = end) |
| Sitemap: https://… | Location of your XML sitemap |
Rules are grouped by user-agent. A bot follows the most specific group that matches its name; for Google, the longest matching rule wins when Allow and Disallow conflict.
Ready-to-use examples
Allow everything
User-agent: * then Disallow: (empty) and your Sitemap: line.
A good WordPress robots.txt
User-agent: *Disallow: /wp-admin/Allow: /wp-admin/admin-ajax.phpDisallow: /?s=andDisallow: /search/(internal search results)Sitemap: https://yourdomain.com/sitemap_index.xml(or /wp-sitemap.xml)
Do not block /wp-content/ or /wp-includes/ – Google needs your CSS and JavaScript to render pages.
Block a staging site completely
User-agent: * then Disallow: / – but better, protect staging sites with a password.
How to block AI crawlers
Many site owners now choose whether AI companies may use their content for training. Add a separate group for each bot:
| User-agent | Company / purpose |
|---|---|
| GPTBot | OpenAI – model training |
| ClaudeBot | Anthropic |
| Google-Extended | Google – Gemini training (does not affect Search) |
| CCBot | Common Crawl dataset |
| PerplexityBot | Perplexity |
| Applebot-Extended | Apple – AI training |
Each group looks like User-agent: GPTBot followed by Disallow: /. Blocking these does not affect your Google or Bing rankings. Reputable AI crawlers follow robots.txt; others may not.
Common mistakes
Disallow: /left on after launch – the classic way to vanish from Google.- Blocking CSS/JS files – Google cannot render your pages properly.
- Using robots.txt to hide private pages – it advertises them instead. Use logins.
- Blocking pages you want removed – Google cannot see the noindex tag.
- Wrong location – it must be at the root, not in a subfolder; each subdomain needs its own file.
How to test it
- Open
yourdomain.com/robots.txtin a browser. - Check Settings › robots.txt in Google Search Console for fetch errors.
- Use the URL tester in our Robots.txt Generator to see whether a specific URL is allowed for Googlebot or GPTBot.
Then make sure your sitemap is complete – see how to create a sitemap.
FAQ
Where is my robots.txt file?
At https://yourdomain.com/robots.txt. WordPress serves a virtual one if no file exists.
Does robots.txt stop a page from being indexed?
No. Use a noindex meta tag for that, and keep the page crawlable.
How do I block ChatGPT from my website?
Add “User-agent: GPTBot” with “Disallow: /”. OpenAI’s other agents, such as ChatGPT-User, have their own names.
Will blocking AI bots hurt my SEO?
No. Googlebot and Bingbot are separate from AI training crawlers like Google-Extended and GPTBot.
What happens if I have no robots.txt?
Bots assume they may crawl everything. That is fine for many small sites.
Need a website or app for your business? Talk to Techpass Master – we design and build websites and apps for clients worldwide.
