Robots.txt Validator
Find The Line That Broke It
Paste your file into this robots.txt validator and it finds the line that broke it. Every finding appears beside the file, quoted with its line number. Then type a path and see exactly which rule decides it, and why that rule beat the others.
- FreePrice
- 20+Checks run
- 4Crawlers
- NoneUploads
Robots.txt Validator
Check the syntax, then test your paths against your own rules
Any address on the site will do. This tool keeps only the origin, because robots.txt lives at the root and nowhere else.
- Errors
- 0
- Warnings
- 0
- Groups
- 0
One per line. A full address works too, and only its path and query count. Every line goes against the crawler you picked above.
Type a path and this line says which rule wins, and why.
The verdictsReady. Paste a file, or fetch one, and the findings appear on the right.
What A Robots.txt Validator Catches
A broken robots.txt rarely announces itself. Crawlers skip the line they cannot read, then carry on as though it were never written.
So the damage shows up weeks later, in a traffic chart. Meanwhile the file still looks perfectly reasonable in your editor.
This checker reads the file the way a crawler does. Therefore a rule that matches nothing gets flagged, rather than silently obeyed.
The file ran on convention for a quarter of a century. RFC 9309, which standardised robots.txt turned that convention into a rule.
Valid Syntax Versus Correct Behaviour
A file can pass every syntax test and still ruin a launch. Disallow: / is flawless code, after all.
For that reason, the tool comes in two parts. The left half reads your lines, while the right half answers a question about one path.
Type /wp-admin/ and the verdict line names the winning rule. In short, you get the outcome rather than a guess.
Robots.txt Slips That Hide in Plain View
A space inside a path quietly truncates the rule. Everything after that space simply vanishes, so the rule matches far more than you intended.
A full web address in a Disallow line fails too. Since paths run from the root of the site, https://example.com/folder/ matches nothing.
Noindex in robots.txt stopped working in September 2019. However, plenty of old files still carry that line and its author still trusts it.
One Crawler Named in Two Groups
Write User-agent: Googlebot twice and you have made a trap. A crawler reads the first group that names it, then ignores every later one. So the second set of rules never runs, even though it sits right there in the file. The checker spots the duplicate and points at both line numbers.
Testing Your Robots.txt in Four Moves
Paste, read, test. Because the checking runs as you type, there is no button to wait on.
- 1
Paste the whole file
Copy everything from example.com/robots.txt, or open the file from your disk. Partial files give partial answers.
- 2
Read the findings
Each one quotes its line and explains the fix. Errors sit at the top, with warnings below them, then optional style notes.
- 3
Pick a crawler
Googlebot, Bingbot and GPTBot all follow different groups. So switch between them if your file names more than one.
- 4
Test a real path
Type an address such as
/cart/. The verdict line says allowed or blocked, and names the rule that decided it.
Where This Validator Goes Further
Many checkers print a green tick and stop there. Instead, this one shows its working on every finding.
Line numbers on everything
Each finding quotes the offending line beside its number. As a result you fix it without scrolling back and forth.
It explains the winner
The verdict names the exact rule and its length. Since the longest match wins, that length is the whole argument.
Nothing gets rewritten
A checker that silently fixes your file teaches you nothing. Rather than patch it, this one tells you what to change.
Dead directives flagged
Host, Clean-param and Noindex all read like real rules. However, the crawlers you care about ignore them now.
Style notes stay optional
Trailing spaces and tidiness live behind a switch. So the default view shows only what actually breaks something.
The file you paste stays inside this tab
The parser runs inside this tab, start to finish. So a pasted file and the paths you test never reach a server.
Fetch a live file
Type a domain and this server reads its robots.txt for you, because no browser may read another site. Only that address leaves.
All bots, not only Googlebot
Forty product tokens are on the list, from Googlebot-Image to ClaudeBot. Each one reads a different group, and the verdict changes with it.
Many paths at once
Paste a column of paths and every one gets a verdict, with the rule and line that decided it. The table exports as CSV.
Habits That Keep This File Honest
A robots.txt is short enough to read every time you touch it. Still, a few routines catch the rest.
- Try the URLs your business depends on. Check your cart, your admin area and one ordinary article. If the ordinary article is blocked, stop and fix it now.
- Check after every deploy. A build step can overwrite this file without telling anybody. So a ten-second paste beats a month of lost traffic.
- Merge duplicate groups. One crawler, one group. Because only the first group applies, a second one is dead weight that looks alive.
- Write paths exactly. Matching is case sensitive and space intolerant. Therefore /Cart/ and /cart/ are different, and a space ends the path early.
- Keep the file small. Google reads 500 kilobytes and stops. Since almost no honest file comes close, a huge one usually means generated junk.
- Fix errors before warnings. An error means a crawler skipped that line entirely. A warning means it worked, though probably not as you hoped.
Three Faults Worth Hunting For
These three turn up again and again in real files. Each one is invisible until somebody checks.
Rules above the first group
A Disallow line written before any User-agent line belongs to nobody. So every crawler walks straight past it.
A path without its slash
Disallow: admin/ is relative, and crawlers disagree about relative paths. Instead, always open a path with a slash.
Trusting a green tick
Valid syntax only proves the file parses. Above all, test a real path before you call the job finished.
Robots.txt Checking Questions
Why is Google ignoring my robots.txt rule?
Usually because something longer matched first. A crawler picks the longest matching path, not the first one in the file. Paste the file here, type the path, and the verdict line names the rule that actually won.
Does this fetch my live file for me?
Yes. Type a domain and this server reads its robots.txt, because no browser may read another site. Only the address travels, and no copy of the answer is kept. Pasting a file instead keeps everything in your tab.
Which crawler should I test as?
Start with Googlebot, since it drives most search traffic. Then switch to GPTBot if your file has a group for AI crawlers, because those rules are easy to get subtly wrong.
Is Crawl-delay worth keeping?
Google ignores it completely. Bing and Yandex do read it though, so keep the line if one of those is crawling too hard. Otherwise it is harmless clutter.
What does an empty Disallow line mean?
It allows everything, which is the opposite of how it reads. The line is legal and it has confused people for twenty years. In short, write Allow: / instead and save the next person some doubt.
Can I test a draft before I upload it?
Yes, and before launch is the ideal moment. Paste the version from your repository, test the paths you care about, then deploy once the verdicts look right.
Handy Companions for This Tool
Googlebot, Bingbot and the rest read these rules the same way. So each runs free, and none wants an email address.
Robots.txt Generator
Build a fresh file from presets instead of patching an old one.
Open tool →HTTP Status Code Checker
Find out what a crawler really gets back from an address.
Open tool →Hreflang Tag Generator
Point each language at its own version, without typos.
Open tool →
Paste Your File And Find Out
Findings appear as you type, and a column of paths takes one more second. A pasted file never leaves your tab, and nothing is stored either way.