Robots.txt Generator
The File Crawlers Read First
This robots.txt generator turns a few dropdown choices into a finished file. Pick a starting point, then add a rule per row. The file on the right rewrites itself as you go, so you never guess at the syntax.
- FreePrice
- 12Crawlers named
- 4Starting points
- NoneUploads
Robots.txt Generator
Build the file crawlers read first — one rule per row
Optional, though worth adding. It is the one line that helps a crawler rather than restricting it.
Your rules- Groups
- 0
- Rules
- 0
- Size
- 0 B
Ready. Pick a starting point and the file builds itself.
What a Robots.txt Generator Saves You From
Every crawler asks for one file before it asks for anything else. So that one file shapes what the rest of the visit looks like.
The syntax is small, yet it punishes typos in silence. A robots.txt generator removes the guesswork, because the crawler names come from a list rather than your memory.
One stray slash can hide a whole site. Therefore it pays to build the file from known-good parts and read it before you upload.
For twenty-five years this file had no standard at all. RFC 9309, which standardised robots.txt finally wrote one down in 2022.
The Limits of What Robots.txt Governs
Robots.txt settles a single matter: whether a crawler may request an address. It says nothing about ranking, and nothing about indexing either.
Bots check for robots.txt only in the top folder of the domain. In other words, example.com/robots.txt works, while example.com/blog/robots.txt is ignored.
Paths in the rules also care about upper and lower case. Since /Blog/ and /blog/ are two different paths, write them exactly as your site serves them.
Settling a Clash Between Two Rules
A crawler obeys one group only, the one whose user-agent matches it best. So a Googlebot group overrides the group for every crawler.
Within that group, the most specific matching path takes priority. For example, Allow: /wp-admin/admin-ajax.php beats Disallow: /wp-admin/, because it is longer and more exact.
When two rules tie on length, Allow wins. Above all, order does not decide anything here, however tidy your file looks.
Why a Disallow Rule Does Not Hide a Page
A blocked address can still show up in search results, listed without a description, if other pages link to it. A noindex tag, not robots.txt, keeps a page out of search results. Meanwhile, remember the page must stay crawlable, otherwise nobody ever reads the tag.
Building a Robots.txt File With Denwix
Four short steps, and no account anywhere in them. In fact, the first choice does most of the work for you.
- 1
Pick a starting point
Choose an open site, a WordPress site, a shop or a full block. Each one drops in the rules that suit it.
- 2
Adjust the rows
Every row holds a crawler, a verdict and a path. Add rows, delete them, or leave the preset alone.
- 3
Add your sitemap line
Enter the complete URL of your sitemap. It is the one line here that helps a crawler rather than fencing it off.
- 4
Copy or download
Press Copy file, or save it as robots.txt. After that, place it in the top folder of your domain.
What This Robots.txt Generator Gets Right
Most builders hand you a text box and wish you luck. Instead, this one narrows the choices until a typo is hard to make.
Crawler names come from a list
GoogleBot with a capital B matches nothing at all. So the names sit in a dropdown, spelled the way the crawlers spell them.
One switch for AI crawlers
Twelve training crawlers go into the file together. Therefore you do not have to hunt down each name yourself.
Comments that explain themselves
Each group can carry a short note above it. In six months that note is what stops somebody deleting the line.
Grouping done properly
Rules for the same crawler belong under one heading. So the tool collects them, rather than repeating the user-agent line.
A live size counter
Google reads the first 500 kilobytes and stops. Since real files sit far below that, the counter is simply reassurance.
Nothing leaves the tab
Your browser writes every line of this file. As a result, your paths never reach a server, ours or anyone else's.
Habits for Every Robots.txt You Write
The file is short, so the habits are short too. Still, each one prevents a specific kind of bad day.
- Read each rule before uploading robots.txt. Scan the output for a lone Disallow: / line. Because that single slash blocks the entire site, it is worth ten seconds.
- Always name the sitemap. The Sitemap line stands outside every group, so it applies to all crawlers. Therefore one line serves everybody.
- Never list private paths in the file. Anyone can open your robots.txt in a browser. Instead of listing a private folder, put it behind a password.
- Keep your CSS open to crawlers. Google builds each page in full before it assesses it. If the stylesheet is blocked, the page it sees is not the page your visitors see.
- Leave out crawl-delay, since Googlebot ignores it. Google ignores that line entirely. Bing and Yandex do read it though, so set it only when one of those is hammering you.
- Keep the file in your code repository. Keep the file in your repository beside everything else. Then a surprise change shows up in a diff instead of in your traffic.
Three Ways This File Goes Wrong
Each of these looks harmless in an editor. Yet each one has quietly cost real sites their traffic.
Shipping the staging file
A test server blocks everything, quite rightly. However, that file then travels to production with the rest of the code.
Relying on it to make a page secret
Google can list a blocked URL without a snippet. Rather than guess, add a noindex tag and let the crawler in to read it.
Trusting the file with security
Well-behaved crawlers obey it and nothing else has to. So treat it as a traffic sign, never as a lock.
Robots.txt Questions People Ask
Which folder should hold my robots.txt?
At the very root of your domain, as example.com/robots.txt. Crawlers look there and nowhere else. So a copy in a subfolder does nothing at all, however correct its contents are.
Should a small site bother with robots.txt?
No. A missing file simply means everything may be crawled, which suits plenty of small sites. Still, one is worth having, because it carries the sitemap line and gives you somewhere to add a rule later.
Does a disallow line take a page out of Google results?
Not reliably. If other sites link to that address, it can appear as a bare result with no description. Therefore use a noindex tag when you want a page gone, and leave it crawlable so the tag is read.
Do wildcards work?
Google, Bing and Yandex all understand * for any run of characters and $ for the end of an address. Some older or minor bots may ignore it. In short, wildcards are safe for the search engines that matter.
Is it wise to block GPTBot and other AI crawlers?
That choice depends on your business goals more than on code. Blocking them protects your writing from training sets, yet it may also cut your visibility in AI answers. The toggle is available in both cases.
Does the Denwix generator send my rules anywhere?
No. The page never calls a server while you work. Your browser holds the rules, builds the text and hands you the download, so nothing about your site structure is sent anywhere.
Denwix Tools to Complete the Setup
These three pair with this one naturally. So none of them asks for an account either.
Robots.txt Validator
Paste a file and find out which rule wins for any path.
Open tool →XML Sitemap Generator
Turn a list of addresses into the sitemap this file points at.
Open tool →Canonical Tag Generator
Tell Google which URL to index with a canonical tag.
Open tool →
Ready To Write Your Robots.txt?
Pick a starting point, adjust a few rows, then copy the result. It takes about a minute, it costs nothing, and your paths never leave this tab.