Home › Free tools › Robots.txt generator
Free SEO tool · WordPress-ready

Robots.txt generator – control how search engines and AI bots crawl your site

Create a correct robots.txt in a minute: allow or block search engines, keep admin and private folders out of crawling, add your sitemap and choose whether AI crawlers like GPTBot may use your content.

WordPress and Shopify presetsAI crawler controlsSitemap and tester
1. Start from a preset
2. Your site
Start each path with /. Use * as a wildcard and $ for “ends with”, for example /*.pdf$
3. AI crawlers
Blocking AI training crawlers does not affect your normal Google or Bing rankings.
4. Other options
Test a URL

What is robots.txt?

robots.txt is a small text file at the root of your website (example.com/robots.txt) that tells crawlers which pages they may visit. Search engines like Google and Bing read it before crawling, and so do AI crawlers such as GPTBot and ClaudeBot. It controls crawling, not indexing: a blocked page can still appear in Google if other sites link to it. To keep a page out of search results, use a noindex meta tag instead – and do not block that page in robots.txt, or Google will never see the tag.

How to add robots.txt to your site

  • Any website: upload the downloaded robots.txt file to the root folder (next to your home page).
  • WordPress: most SEO plugins have a robots.txt editor under their Tools or Settings menu – paste the text there. WordPress creates a basic virtual file if you have none.
  • Shopify: edit the robots.txt.liquid theme template.
  • Check it in Google Search Console under Settings › robots.txt.

Robots.txt rules explained

RuleMeaning
User-agent: *The rules below apply to all crawlers
Disallow: /private/Do not crawl anything that starts with /private/
Disallow:(empty) Nothing is blocked
Allow: /wp-admin/admin-ajax.phpAn exception inside a blocked folder
Disallow: /*.pdf$Block every URL that ends in .pdf
Sitemap: https://…/sitemap.xmlWhere to find your XML sitemap

Common mistakes

  • Disallow: / left on after launch – this blocks your whole site from Google.
  • Blocking CSS and JavaScript files – Google needs them to render your pages.
  • Using robots.txt to hide private data – the file is public, so it actually points people to those folders. Protect private areas with a login instead.

Frequently asked questions

Where do I put robots.txt?

In the root of your domain, so it opens at https://yourdomain.com/robots.txt. Each subdomain needs its own file.

Does robots.txt stop a page from being indexed?

No, it only stops crawling. Use a noindex meta tag (and allow crawling) to remove a page from search results.

How do I block ChatGPT and other AI bots?

Tick the AI crawlers you want to block. The generator adds a separate group such as “User-agent: GPTBot” with “Disallow: /”. Well-behaved AI crawlers follow these rules.

Will blocking AI crawlers hurt my Google rankings?

No. Google-Extended only controls use of your content for Gemini training. Googlebot, which handles search, is not affected.

What is the best robots.txt for WordPress?

Block /wp-admin/ but allow /wp-admin/admin-ajax.php, block internal search results, and add your sitemap. The WordPress preset does exactly this.

Need a website or app built?

We design and build websites and Android apps for clients worldwide – clear scope, fixed price, fast delivery.

Techpass Master Technologies · Websites, Android apps and digital marketing

WhatsApp us📞 Call now