Free robots.txt Generator
Upload this file as /robots.txt at the root of your domain (not in a subfolder).
Results and any AI-generated content from this tool are provided free, "as is", for general informational purposes only and may be inaccurate, incomplete, or out of date. Always verify the output yourself before relying on it — you use this tool and its results at your own risk.
By using this tool you agree to our Terms & Conditions and Privacy Policy.
Put this on your free business page
Create a BizGo digital business page — your services, gallery, WhatsApp button, and Google review link, all in one shareable page. Free to start.
What robots.txt actually does
A robots.txt file lives at the root of your domain
(https://yoursite.com/robots.txt) and tells well-behaved web
crawlers — search engines, AI training bots, SEO tools — which parts of
your site they're allowed to fetch. It is a set of instructions, not a
security wall: it works only because reputable crawlers voluntarily respect
it, so it should never be your only defense for content that genuinely
must stay private (use authentication for that instead). What it's
genuinely good for is keeping crawlers away from admin panels, checkout
flows, duplicate content, and staging environments, and pointing search
engines at your sitemap.
Picking a preset
Allow all is the right default for almost every live
public website — it lets every search engine index your content normally.
WordPress adds the standard recommended block on
/wp-admin/ while explicitly allowing
admin-ajax.php, which many WordPress plugins and themes rely
on even for logged-out visitors. Block known AI crawlers
adds a Disallow rule for each of the current major AI training/browsing
bots — GPTBot, CCBot, Google-Extended, anthropic-ai, and PerplexityBot —
useful if you don't want your content used to train AI models, though note
this list changes as new bots launch and isn't a guarantee every AI company
respects it. Staging blackout blocks every crawler from
everything — always check this is switched on for a staging or
development copy of your site, since an indexed staging site creates
duplicate-content problems for your real one.
Custom rules and common mistakes
Beyond the presets, add your own user-agent/directive/path rows for
anything specific to your site. This tool checks for the most common
robots.txt mistakes as you go: a Disallow path that's missing
its leading slash (which silently matches nothing), an empty
Allow rule (which has no effect), and a Sitemap:
line pointed at a relative path instead of a full URL — all flagged with a
clear warning before you copy or download the file.
Once your file is ready, download it and upload it as
/robots.txt at your domain's root — not inside any subfolder,
or crawlers won't find it. A BizGo digital business page is already
correctly configured for search engines out of the box — see the box above
to get started.
Frequently Asked Questions
At the root of your domain — https://yoursite.com/robots.txt. It must be at the root, not inside a subfolder, or crawlers will not find it.
No — it only asks well-behaved crawlers not to fetch certain URLs. It is not a security or access-control mechanism; anyone with the direct link can still open the page in a browser. Use authentication for content that genuinely must stay private.
It adds a Disallow rule for GPTBot, CCBot, Google-Extended, anthropic-ai, and PerplexityBot specifically. This list reflects current major AI crawlers at the time of writing — new bots launch periodically, and not every AI company is guaranteed to respect robots.txt.
If a staging or test copy of your site gets indexed by Google, it creates duplicate content that can confuse search engines and dilute your real site's rankings. The "Staging blackout" preset blocks every crawler from every URL to prevent that.
A Disallow path that does not start with "/" will not match any real URL on your site, which is a common, easy-to-miss mistake — the tool flags this, along with an empty Allow rule and a non-absolute Sitemap URL, before you copy or download the file.
