"Indexed, Though Blocked by robots.txt" and "Blocked by robots.txt": What They Mean and How to Fix Them

On this page

1. Crawling vs indexing: why this warning happens
2. The two Search Console messages compared
3. How to fix "Indexed, though blocked by robots.txt"
4. Common causes on WordPress and WooCommerce sites
5. Staging sites, hosting panels and site migrations
6. Shopify sites and robots.txt
7. When the warning is harmless
8. Testing robots.txt rules safely
9. What to do after fixing
10. Indexed though blocked: common mistakes to avoid
11. Quick check: confirm what is blocked
12. When to get help
13. FAQs

"Indexed, though blocked by robots.txt" means Google has put a URL in its index even though your robots.txt file tells it not to crawl that URL. Google found links to the page, so it knows the page exists, but it cannot read the content. The fix depends on what you want: if the page should rank, remove the robots.txt block; if it should stay out of Google, allow crawling and add a noindex tag instead.

Short answer: To fix "Indexed, though blocked by robots.txt", decide whether the URL should be in Google. If yes, remove the Disallow rule that blocks it and click Validate Fix in Search Console. If no, remove the block too, add a noindex tag and let Google recrawl: a page blocked by robots.txt cannot show Google its noindex.

This guide explains the two related Search Console messages, the common causes on WordPress, WooCommerce and Shopify sites, and the exact steps to fix and validate them.

Crawling vs indexing: why this warning happens

Two separate things happen before a page appears in Google:

• Crawling: Googlebot visits the URL and reads the content.
• Indexing: Google stores the page so it can show it in search results.

robots.txt only controls crawling. It says "do not visit these paths". It does not say "do not index". If other pages, sitemaps or external sites link to a blocked URL, Google may still index the bare URL, often showing it with no description or a note that no information is available. That is what the warning reports.

The robots.txt guide covers the file itself in more depth.

The two Search Console messages compared

Message	Where it appears	What it means	Usual action
Indexed, though blocked by robots.txt	Page indexing report, as a warning on indexed pages	URL is in the index but Google cannot crawl it	Unblock if it should rank; otherwise allow crawl plus noindex
Blocked by robots.txt	Page indexing report, under not indexed	Google respected the block and did not index the URL	Usually fine; unblock only if the page should rank

The second message is often exactly what you want. The first one needs a decision for each important URL.

How to fix "Indexed, though blocked by robots.txt"

1. Open the report. In Search Console, go to Pages, find the warning and open the list of example URLs.
2. Group the URLs. Sort them by type: real pages, product pages, search results, cart or checkout URLs, filter parameters, admin or plugin paths.
3. Decide for each group. Should it rank in Google? Yes or no.
4. If yes: find the robots.txt rule that blocks it and remove or narrow that rule. Use the URL Inspection tool to test the live URL afterwards.
5. If no: remove the robots.txt block for that path, then add a noindex meta tag or X-Robots-Tag header so Google can crawl the page, see the noindex and drop it.
6. Remove internal links to unwanted URLs where you can, and make sure they are not in your XML sitemap.
7. Validate the fix in Search Console once changes are live, and give Google time to recrawl.

The step people miss is number five. Blocking a page in robots.txt and adding noindex at the same time does not work, because Google cannot crawl the page to see the noindex.

Common causes on WordPress and WooCommerce sites

• An old robots.txt from development: a blanket Disallow rule left in place after launch, or rules copied from another site.
• Blocked wp-content or plugin folders: older advice suggested blocking these, which can stop Google loading CSS and images it needs to render pages.
• WooCommerce cart, checkout and account pages: often blocked deliberately. If they show as indexed, they are usually linked from somewhere; noindex is the cleaner control.
• Filter and sort URLs: layered navigation creates many parameter URLs. Blocking them can lead to this warning if they are widely linked. See WooCommerce product filters for cleaner approaches.
• Internal search results: URLs with the search parameter are often blocked; they are rarely worth indexing.
• SEO plugin virtual robots.txt: Yoast, Rank Math and AIOSEO can edit robots.txt from the dashboard. Check there as well as on the server. The SEO plugins comparison explains where each keeps this setting.

Staging sites, hosting panels and site migrations

Many Indian business sites are built on a staging copy first, often on a subdomain or a temporary folder on shared hosting. Developers sensibly block the staging site from search engines. The trouble starts when the site goes live and the same robots.txt, or the WordPress "Discourage search engines" setting, is carried across to the live domain. The opposite also happens: a staging subdomain is left open, Google finds it, and you end up with duplicate copies of your pages in the index.

After any launch or migration, check three things on the live domain: the robots.txt file, the WordPress reading setting, and the noindex settings in your SEO plugin. For the staging copy, password protection is a safer block than robots.txt, because it stops both crawling and indexing. Some hosting control panels and caching or security plugins can also add their own rules, so if the file you see on the server does not match what Google reports, check those too. The guide on a WordPress site not showing on Google covers the launch checks in detail.

Shopify sites and robots.txt

Shopify generates a default robots.txt that blocks paths such as cart, checkout, account and some filtered collection URLs. Seeing these listed as blocked is normal. If an important product or collection page appears, check whether a theme or app has added a custom robots.txt template with extra rules. Changes on Shopify are made through the robots.txt.liquid theme template, so edit with care and keep a copy of the original. The Shopify SEO checklist covers the wider setup.

Shopify store owners who want robots.txt, filter pages and other SEO settings checked can work with a Shopify developer on speed and SEO fixes.

When the warning is harmless

Not every listed URL needs fixing. Leave it alone when:

• The URL is a cart, checkout, login or admin path nobody searches for.
• It is a parameter URL that duplicates a real page which is indexed normally.
• The number of affected URLs is small and none of them get impressions in the Performance report.

Focus on URLs that should be bringing traffic: service pages, product pages, category pages and blog posts. If any of those are blocked, fix them first.

Testing robots.txt rules safely

A single misplaced character in robots.txt can block a whole site. Before and after any change:

• Open yourdomain.com/robots.txt in a browser and read every line.
• Check that there is no lone "Disallow: /" under "User-agent: *" unless you really mean to block everything.
• Use the robots.txt report in Search Console settings to see the version Google last fetched and any errors.
• Run URL Inspection on a few important pages and confirm crawling is allowed.
• Keep a dated copy of the old file so you can roll back.

A simple, safe WordPress robots.txt often only needs a rule for the admin area, with an exception for the admin-ajax file, and a line pointing to the sitemap.

What to do after fixing

Click Validate Fix in the report once changes are live. Validation can take days or weeks depending on how often Google crawls your site. Meanwhile:

• Request indexing for the most important pages you unblocked.
• Check your sitemap is submitted and lists only pages you want indexed. See XML sitemaps explained.
• Watch the Pages report for other issues, explained in Search Console page indexing errors.

Do not expect the counts to drop overnight. Google needs to recrawl each URL before the report updates.

Indexed though blocked: common mistakes to avoid

• Adding noindex to a page that robots.txt still blocks, so Google never sees it.
• Blocking CSS and JavaScript folders, which can affect how Google renders and understands pages.
• Using robots.txt to hide private content; anyone can read robots.txt, and blocked URLs can still be indexed. Use passwords or noindex instead.
• Panicking over cart and checkout URLs that never needed to rank.
• Editing robots.txt on the server while an SEO plugin serves a different virtual version.

Quick check: confirm what is blocked

1. Open yourdomain.com/robots.txt and read every Disallow line.
2. In Search Console, open Indexing > Pages, click the "Indexed, though blocked by robots.txt" row and note the URL pattern.
3. Run one of those URLs through URL Inspection. If the page should not be in Google, remove the robots.txt block and add a noindex tag instead; if it should be, remove the block and request indexing.

When to get help

Get help when the blocked URLs are filters, cart or search pages on a store, when your SEO plugin and a hand-written robots.txt disagree, or when many warnings appear at once. WordPress SEO services covers robots.txt, sitemaps and indexing rules. You can also send the details through the contact form.

FAQs

Is "Indexed, though blocked by robots.txt" bad for SEO?

Not always. If the URLs are cart, checkout, search or filter pages you never wanted in Google, it is mostly harmless. It matters when important pages are blocked, because Google can list them without reading their content, so they show poor snippets and rank badly.

Should I use robots.txt or noindex to keep a page out of Google?

Use noindex. robots.txt only stops crawling, so a blocked page can still be indexed from links. A noindex tag removes it, but Google must be allowed to crawl the page to see the tag.

How do I fix indexed, though blocked by robots.txt?

Decide whether each URL should rank. If yes, remove the robots.txt rule blocking it. If no, allow crawling and add a noindex tag so Google can see it and drop the page. Then click Validate Fix in Search Console.

Is "Blocked by robots.txt" an error?

Not necessarily. It means Google respected your instruction. It is only a problem if a page you want in search results is listed there.

Can I use robots.txt to remove a page from Google?

No. robots.txt stops crawling, not indexing. Use a noindex tag with crawling allowed, or the Removals tool for an urgent, temporary hide.

Why are my WooCommerce cart and checkout pages in this report?

They are often blocked in robots.txt but linked from every page, so Google knows they exist. This is usually harmless. If you want them gone, use noindex rather than a robots.txt block.

How long does validation take after the fix?

It varies with how often Google crawls your site, from days to a few weeks. Requesting indexing for key pages can help them get recrawled sooner.