Tell crawlers
what to read.
Generate a safe robots.txt: control search engines and AI training bots, block private paths, and add your sitemap. Plus an llms.txt starter for AI assistants. Copy and save at your site root.
Save robots.txt and llms.txt at your site's root. llms.txt is an emerging standard that helps AI assistants understand your site. Ours is at stonekept.studio/llms.txt.
robots.txt is a set of instructions that well-behaved crawlers follow; it is not a security control and does not hide pages from people. Save the file at your site's root (yoursite.com/robots.txt).
Good to
know.
What should my robots.txt file contain?
For most small-business sites, very little: allow all crawlers, block any private or duplicate paths (like a cart or admin area), and point to your sitemap. Over-blocking is a common mistake that hides your site from Google. This tool generates a safe default and lets you add specific paths to block and, if you want, opt out of AI training crawlers.
Can I block AI bots like GPTBot from my site?
Yes. You can disallow AI training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and others) in robots.txt, and this tool will add those rules for you. It's a choice: blocking them keeps your content out of AI training, but may also reduce how often AI assistants cite your business. robots.txt is a request that the major, well-behaved crawlers honor.
While you’re here
Favicon Generator
Make a real favicon.ico plus apple-touch icon from a letter, emoji, or your logo, right in your browser.
Open tool301 Redirect Generator
Paste old and new URLs and get 301 redirect rules for Cloudflare, Apache, or Nginx. Keep your rankings.
Open toolPrivacy Policy Generator
Answer a few questions and get a plain-English privacy policy tailored to what your website collects.
Open toolLet's turn this into
a real plan.
Tell us what you're building and we'll send a straight, fixed proposal. No pressure, no sales fog.