Ignore it if
The URL is an admin page, cart, account area, internal search result or filter URL.
Blocked by robots.txtCheck it
“Blocked by robots.txt” in Google Search Console: what it means and how to fix it
Your robots.txt file tells Google not to crawl this URL, so Google can't read the page. That's fine for pages you don't want crawled. But robots.txt doesn't keep a page out of search: Google can still index the URL without its content if other pages link to it. To keep a page out of Google, use a noindex tag and let Google crawl it.
What Google says
This page was blocked by your site's robots.txt file, but it could still be indexed if Google finds information about it elsewhere. To keep it out of search, remove the robots.txt block and use noindex.
Paraphrased from Google's Page indexing report helpCauses
Why it happens
01A Disallow rule broader than intended, like Disallow: /blog.
02A staging robots.txt with Disallow: / shipped to production.
03A 'hide from search engines' setting in your CMS or site builder.
04Intentional rules for parameter URLs, search results or private areas.
Triage
Is it a problem?
Fix it if
A post or landing page you want in search is listed.
You're blocking pages to keep them out of search. Use noindex instead.
Fix
How to fix “Blocked by robots.txt”
Check any single URL with URL Inspection. After fixing, open the issue in the Page indexing report and click Validate fix; Google says it typically takes up to about two weeks.
- 1Find the ruleOpen yoursite.com/robots.txt and find the Disallow line that matches the URL. Rules match from the start of the path, so Disallow: /blog also blocks /blog-post-ideas.
- 2Check it in Search ConsoleSettings → robots.txt shows the file Google last fetched, when, and any errors. URL Inspection tells you whether a specific URL is blocked.
- 3Narrow or remove the ruleEdit robots.txt so it only blocks what you mean to. If the whole site is blocked after a launch, remove Disallow: /.
- 4For pages you want out of search, use noindexAllow crawling and add <meta name="robots" content="noindex">. Google has to crawl the page to see the noindex.
- 5Validate the fixIn the Page indexing report, open the issue and click Validate fix. Google says validation typically takes up to about two weeks.
On Blogizi
Blogizi writes your blog's robots.txt for you. It allows crawling and lists your sitemap. If you turn on 'Hide blog from search engines' in project settings, Blogizi adds a noindex tag to every page and keeps crawling allowed, so Google can actually see it.
Index checks for every post are part of Pro, $19/month with a 7-day trial for $0. Your AI agent can read the same statuses over MCP.
FAQ
Questions about “Blocked by robots.txt”
No. It stops crawling. Google can still index a blocked URL, without its content, if other pages link to it. Use noindex to keep pages out of search.
The URL is blocked from crawling but Google indexed it anyway from links. If you don't want it in search, unblock it and add noindex.
Google usually refetches robots.txt within about a day. The robots.txt report in Settings shows when it was last fetched.
Related statuses
Excluded by ‘noindex’ tagThe page tells Google not to index it, and Google listened.Read →Discovered - currently not indexedGoogle knows the URL exists but hasn't crawled it yet.Read →Soft 404The page looks like an error or empty page to Google but returns a success code.Read →
Post not on Google at all? Start with why isn't my blog post on Google, or browse all 9 statuses.
Last reviewed October 19, 2026