Robots.txt generator for WordPress
Tick the options you need and copy a clean robots.txt. The defaults are safe for WordPress, and the tool warns you about rules that could block your whole site or its CSS.
Runs 100% in your browser. Nothing you add is uploaded to any server.
robots.txt
How to use it
- Keep the WordPress defaults ticked unless you have a reason not to.
- Add your sitemap URL.
- Add any extra paths to block, such as cart or checkout pages.
- Copy the result into a file named robots.txt at your site root, or into the robots.txt editor in Yoast or Rank Math.
Related reading: robots.txt explained.
Want it done for you? See WordPress SEO services.
Robots.txt questions
Does robots.txt remove pages from Google?
No. It only stops crawling. A blocked page can still be indexed if other sites link to it. To remove a page, use a noindex tag and let Google crawl it.
Should I block /wp-content/ or /wp-includes/?
No. Google needs your CSS, JavaScript and images to render pages properly, and many of those files live in these folders.
Where does robots.txt go?
At the root of your domain, so it opens at example.com/robots.txt. Each subdomain needs its own file.
What does Disallow: / do?
It asks all crawlers not to crawl any page on the site. It is useful on staging sites, but on a live site it can make your pages drop out of search.
Let's build your next website
Available for freelance projects, agency white-label work and long-term maintenance. Feel free to pass this along to your team or company.