What Blocked by robots.txt Means
Search Console says some pages are blocked by robots.txt. That means your robots.txt file tells Google not to crawl them. So Google cannot read those pages, and they rarely show up in search.
Sometimes that is exactly what you want, for example for a cart or an admin area. But if the report lists your home page, your shop or your articles, something in the file is wrong.
Step 1: Find the Line That Blocks
Open your site's robots.txt file. It always lives at the root, for example https://example.com/robots.txt. In our demo, it held just two lines:
User-agent: *
Disallow: /Those two lines block the whole site for every crawler. Next, paste the file into our free Robots.txt Validator and test a few addresses. It shows which rule blocks each page, so you can see the culprit straight away.
Where this line comes from
It is almost always a leftover from a staging site, where blocking everything made sense. When the site moved to live, the file came with it.
Step 2: Fix the File
Replace the blocking line. Block only what must stay out of search, such as the cart and the checkout, and add the address of your sitemap:
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Sitemap: https://example.com/sitemap.xmlSave the file in your hosting panel or upload it by FTP. Then open the live file in the browser to confirm the new rules are there.
Run the validator again. In our demo, the home page, the shop and the about page were allowed, and the cart stayed blocked, just as planned.
robots.txt Examples
Use these as starting points, then test them with the validator. Our Robots.txt Generator can also build the file for you.
Allow everything
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlA typical WordPress site
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://example.com/sitemap_index.xmlBlock one folder for one crawler
User-agent: Googlebot-Image
Disallow: /private-photos/How Google reads the rules
Google follows the group for its own user agent, or the * group if there is none. Within a group, the most specific matching rule wins. So Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/ for that one file.
robots.txt Is Not noindex
One thing people often mix up: robots.txt stops crawling, not indexing. If other sites link to a blocked page, Google can still list its address, just without a description.
To keep a page out of Google, use a noindex meta tag instead. And let Google crawl the page, so it can see the tag. In other words, never block a page in robots.txt and add noindex at the same time.
Step 3: Ask Google to Crawl Again
Finally, open Google Search Console. In the page indexing report, open the group "Blocked by robots.txt" and click Validate fix. Google then crawls those pages again.
Validation can take a few days to a few weeks. Meanwhile, you can use the URL Inspection tool on your most important page and request indexing for it.
WordPress and Staging Sites
WordPress has a setting called "Discourage search engines from indexing this site" under Settings, Reading. On a staging site it is useful. However, if it stays on after launch, your live site tells search engines to stay away.
So after every launch or migration, check that box, the robots.txt file and the meta robots tags. A short checklist saves weeks of lost traffic.
Common Mistakes
- Keeping
Disallow: /from a staging site after launch. - Blocking CSS and JavaScript folders, so Google cannot render the page.
- Using robots.txt to hide a page that should use noindex.
- Forgetting the trailing slash, so
Disallow: /blogalso blocks /blog-tips/. - Placing robots.txt in a subfolder; it only works at the root of the host.
In short: find the blocking line, block only what must stay private, and ask Google to crawl again. For every rule in detail, see Google's robots.txt guide.
Questions and Answers
What does blocked by robots.txt mean?
Your robots.txt file tells Google not to crawl those pages, so Google cannot read them and rarely shows them in search.
How do I fix pages blocked by robots.txt?
Find the Disallow line that matches them, remove or narrow it, save the file and then click Validate fix in Search Console.
Where is my robots.txt file?
At the root of your domain, for example https://example.com/robots.txt. Subfolders do not count.
How do I test my robots.txt?
Paste it into our free Robots.txt Validator and test the addresses you care about. It shows which rule allows or blocks each one.
Does robots.txt remove a page from Google?
No. It stops crawling, not indexing. Use a noindex tag and let Google crawl the page to remove it.
Why does Google index a page blocked by robots.txt?
Other sites link to it, so Google knows the address. It lists it without a description because it cannot read the page.
What does Disallow: / do?
It blocks the whole site for the crawlers in that group. On a live site it is almost always a mistake.
What is a good robots.txt for WordPress?
Block /wp-admin/, allow /wp-admin/admin-ajax.php, and add your sitemap line. Do not block CSS or JavaScript.
Should I block CSS and JavaScript?
No. Google needs them to render the page and judge its layout and mobile view.
How long until Google recrawls after the fix?
Often a few days. Validation in Search Console can take up to a few weeks for many pages.
What is the Allow directive?
It lets a crawler reach a path inside a blocked folder. The most specific matching rule wins.
Can I block AI crawlers in robots.txt?
Yes, by naming their user agents, such as GPTBot, with Disallow: /. Well-behaved crawlers follow it.
Does robots.txt protect private files?
No. It is public and only a request. Protect private files with a password or keep them off the server.
What is the sitemap line in robots.txt?
A line such as Sitemap: https://example.com/sitemap.xml that tells crawlers where your sitemap is.
Does the WordPress "discourage search engines" setting affect robots.txt?
Yes. It adds a noindex signal and can change the virtual robots.txt. Turn it off after launch.
Is robots.txt case sensitive?
The paths are. Disallow: /Blog/ does not block /blog/, so match the case of your URLs.
What happens if there is no robots.txt?
Crawlers treat the whole site as allowed. That is fine for most small sites.
Can a bad robots.txt hurt my rankings?
Yes. If it blocks important pages, Google cannot read them, and they drop out of search over time.
How do I create a robots.txt file?
Use our free Robots.txt Generator, then upload the file to the root of your site.
Does the SEO checker read robots.txt?
Yes. Our SEO Checker shows your robots.txt and warns when it blocks the page you test.