Robots.txt generator – control how search engines and AI bots crawl your site
Create a correct robots.txt in a minute: allow or block search engines, keep admin and private folders out of crawling, add your sitemap and choose whether AI crawlers like GPTBot may use your content.
What is robots.txt?
robots.txt is a small text file at the root of your website (example.com/robots.txt) that tells crawlers which pages they may visit. Search engines like Google and Bing read it before crawling, and so do AI crawlers such as GPTBot and ClaudeBot. It controls crawling, not indexing: a blocked page can still appear in Google if other sites link to it. To keep a page out of search results, use a noindex meta tag instead – and do not block that page in robots.txt, or Google will never see the tag.
How to add robots.txt to your site
- Any website: upload the downloaded
robots.txtfile to the root folder (next to your home page). - WordPress: most SEO plugins have a robots.txt editor under their Tools or Settings menu – paste the text there. WordPress creates a basic virtual file if you have none.
- Shopify: edit the
robots.txt.liquidtheme template. - Check it in Google Search Console under Settings › robots.txt.
Robots.txt rules explained
| Rule | Meaning |
|---|---|
| User-agent: * | The rules below apply to all crawlers |
| Disallow: /private/ | Do not crawl anything that starts with /private/ |
| Disallow: | (empty) Nothing is blocked |
| Allow: /wp-admin/admin-ajax.php | An exception inside a blocked folder |
| Disallow: /*.pdf$ | Block every URL that ends in .pdf |
| Sitemap: https://…/sitemap.xml | Where to find your XML sitemap |
Common mistakes
- Disallow: / left on after launch – this blocks your whole site from Google.
- Blocking CSS and JavaScript files – Google needs them to render your pages.
- Using robots.txt to hide private data – the file is public, so it actually points people to those folders. Protect private areas with a login instead.
Frequently asked questions
Where do I put robots.txt?
In the root of your domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
Does robots.txt stop a page from being indexed?
No, it only stops crawling. Use a noindex meta tag (and allow crawling) to remove a page from search results.
How do I block ChatGPT and other AI bots?
Tick the AI crawlers you want to block. The generator adds a separate group such as “User-agent: GPTBot” with “Disallow: /”. Well-behaved AI crawlers follow these rules.
Will blocking AI crawlers hurt my Google rankings?
No. Google-Extended only controls use of your content for Gemini training. Googlebot, which handles search, is not affected.
What is the best robots.txt for WordPress?
Block /wp-admin/ but allow /wp-admin/admin-ajax.php, block internal search results, and add your sitemap. The WordPress preset does exactly this.
Need a website or app built?
We design and build websites and Android apps for clients worldwide – clear scope, fixed price, fast delivery.
Techpass Master Technologies · Websites, Android apps and digital marketing
