Generate clean, error-free robots.txt files in seconds. Guide Googlebot, Bingbot, and AI web scrapers on which directories to crawl or ignore, protect private admin areas, and declare XML sitemaps.
How to Create a Robots.txt File in 3 Easy Steps
Build and deploy a search-engine-ready crawler directive in minutes:
- Choose Default Rule: Decide whether all search engines are Allowed by default (recommended) or Disallowed.
- Add Specific Restrictions & Sitemap: Disallow private folders (such as
/admin/or/checkout/) and link your XML sitemap URL. - Download & Upload: Download your
robots.txtfile and place it in the public root folder of your web server.
Standard Syntax Rules for Robots.txt
User-agent: *applies the rule to all compliant web crawlers.Disallow: /path/instructs crawlers not to visit any URL beginning with that folder path.Allow: /path/public/overrides a broader disallow rule to keep a specific subfolder accessible.Sitemap: https://yourdomain.com/sitemap.xmlprovides the direct location of your indexable URL catalog.
Recommended Robots.txt Template for Modern Websites
User-agent: *
Allow: /
Disallow: /admin/
Disallow: /api/
Disallow: /checkout/
# Block AI Scrapers (Optional)
User-agent: GPTBot
Disallow: /
# Sitemap Location
Sitemap: https://yourdomain.com/sitemap.xml