Why a Crawl Stops or Gets Blocked
This guide shows how to crawl a website without getting blocked. It also fixes the other problems people hit most often in the Denwix SEO Crawler. Each fix comes with a short video, so you can watch the clicks as well as read the steps.
Most problems have one of three causes. First, the server says no: a firewall answers the crawl with 403 or 429. Second, the crawler cannot find the links, because scripts build them or the scope is too narrow. Third, a limit stops the crawl, such as the Free plan's 1,000 URLs.
So start with the symptom you see. Lots of 403 rows point to a block. A crawl that ends after one page points to the address, robots.txt or the scope. A crawl that stops at exactly 1,000 URLs is the Free plan, not a bug. The sections below follow that order.
Simply put, you rarely need to reinstall anything. One or two settings under Configuration fix most cases. And when everything else fails, the app can save a debug report for our support team.
How to Crawl a Website Without Getting Blocked
A 403 Forbidden answer means the server understood the request but refused it. The 403 status often comes from a firewall, a CDN rule or a security plugin. These tools watch for many fast requests from one address. When they see them, they decide a bot is at work.
Video: fix 403 errors in a crawl
Slow the crawl down under Configuration, Speed, then change the User-Agent or allow the crawler in your firewall.
Slow the crawl down
Open Configuration → Speed. By default the crawler uses 6 threads, 3 per host, with a 120 ms delay between requests to the same host. That pace is fine for most servers. However, a strict firewall may count it as an attack.
So configure your Max threads per host setting to either 1 or 2. After that, adjust the Delay between requests per host setting upward to either 1,000 or 2,000 milliseconds. Keep Respect robots.txt crawl-delay selected since some web pages will require a delay in that area. Once you've completed those adjustments, begin crawling again and monitor the status codes.
Change the User-Agent or allow the crawler
The crawler names itself honestly as DenwixSEOCrawler/1.0. Some firewalls block every unknown bot by that name alone. In that case, open Configuration → User-Agent and pick another preset, or type your own string.
On your own site, the cleaner fix is an allow rule. Add the crawler's User-Agent or your own IP address to the firewall, CDN or security plugin. Also, a staging site may need a password. Then add the header under Configuration → HTTP Header, and the crawler sends it with every request.
429 Too Many Requests
A 429 means that the server wants you to slow down, so the solution is the same as above. If you reduce the number of threads being used and increase the amount of time between each request, then most of the 429 rows will change into 200s.
Why the Crawl Stops at 1,000 URLs
A crawl that stops at exactly 1,000 URLs is working correctly. That is the limit of the Free plan, and the app works fine. Everything else in the Free plan still works on those 1,000 URLs, such as the Issues tab, reports and sitemaps.
Video: why the crawl stops at 1,000 URLs
See every Free plan limit under Licence, then check the URL limit setting on Premium.
To see every limit side by side, open Licence → What the Free Tier Includes. The table compares Free with Premium and Agency, feature by feature.
Already on Premium?
Premium crawls with no URL limit. Still, check Configuration → Spider → Limits. The Total URL limit field can hold an older saved value, such as 1,000. Set it back to the default, 5,000,000, and the crawl runs to the end.
Premium costs 150 USDT a year for one seat, and Agency costs 399 USDT a year with unlimited seats. You can see both plans on the Denwix SEO Crawler page.
Pages Missing on a JavaScript Site
Some sites build their menus and links with JavaScript. The raw HTML then holds few links, so a text-only crawl finds only a handful of pages. Google explains the same effect in its JavaScript SEO basics.
Video: crawl pages that need JavaScript
Set Rendering to JavaScript under Configuration, Spider, Rendering. It is a Premium feature and needs Chrome or Edge.
Turn on a JavaScript website crawler mode
Open Configuration → Spider → Rendering. There are three choices: Text only, JavaScript (when the page needs it) and JavaScript (every page). Start with the middle one. It renders only the pages that seem to need scripts, so the crawl stays faster.
This is a Premium feature. It also needs Chrome or Microsoft Edge on your PC, because the crawler drives a real browser to run the scripts. As a result, it is slower than a text crawl. So use it only on sites that need it.
Content still missing?
Raise the AJAX timeout from its default of 5,000 ms when a page loads its data late. Also tick Save a screenshot of each rendered page. Then the Rendered Page tab shows what the crawler really saw.
When the Crawl Finds Only One Page
When a crawl ends after one URL, the reason is almost always simple. Check these three things in order, and the crawl should go past the first page.
Video: fix a crawl that finds only one page
Check the address, the robots.txt setting and the scope, then press Start again.
First, the address. A typo, a missing www or http in place of https can lead to a dead end or an error page. So open the address in your browser and copy it from there.
Next, robots.txt. If the file blocks the whole site, the crawler stops at the start URL. Open Configuration → robots.txt. On your own site you can untick Respect robots.txt. Do not do that on a site you are not responsible for. Our guide to pages blocked by robots.txt explains the file in detail, and the robots.txt standard (RFC 9309) has the exact rules.
Then, the scope. The box next to Start sets how far the crawl may go: one host, every subdomain, only the start folder, or external links too. If example.com redirects to www.example.com, a one-host crawl stops at that redirect. Pick a wider option, such as Subdomains.
With our Redirect Chain Checker, you are able to view all the steps involved in each of your redirects. You can then determine which host name your website ultimately resolves to.
Install Warnings and the SHA-256 Check
The free installer comes from the Denwix SEO Crawler page. It runs on 64-bit Windows 10 and Windows 11. Setup needs no admin rights, and the app checks for updates by itself.
Video: install the crawler and check the file
Get past the Windows SmartScreen warning and compare the installer's SHA-256 hash with the one on our page.
Windows may show a blue box that says Windows protected your PC. That happens because the installer is not code-signed yet. Click More info, then Run anyway.
However, you should still make sure the file is really ours. Open PowerShell in your Downloads folder and run Get-FileHash on the installer. Then compare the result with the SHA-256 hash on the product page. If the two match, nobody has changed the file.
Send a Debug Report to Support
Still cannot solve the problem? The app can gather what our support team needs, so you do not have to describe every setting by hand.
Video: save a debug report for support
Open Help, Help and Support for quick fixes, then save a debug report and email it to us.
First, open Help → Help & Support. It lists quick fixes and our contact details. Then choose Help → Save Debug Report. The file goes to your Downloads folder.
The report holds your settings and recent log lines. The app removes passwords and keys before it writes the file. Email it to admin@denwix.com with a short note: what you crawled, what you expected and what happened instead.
A Website Crawler Test Before a Big Crawl
A small website crawler test saves time on a large site. It shows a block or a missing link early, before you wait an hour for a full crawl that finds nothing.
- Check one URL first with our HTTP Status Code Checker. A 403 there means the server blocks simple requests too.
- Set the scope to the start folder and crawl one section of the site.
- Open the Response Codes tab. Look for 403, 429 and 5xx rows.
- If you see them, slow the crawl down under Configuration → Speed.
- If pages are missing, compare the page in your browser with the crawler's view, then try JavaScript rendering.
- When the sample looks clean, widen the scope and run the full crawl.
Also, crawl a busy shop or news site at quiet hours. Your crawl then competes with fewer visitors, and the server is less likely to push back.
Once the crawl works, our website crawler tools guide walks through every feature, from the Issues tab to exports and sitemaps.
Common Mistakes
- Raising threads to go faster. More threads often bring more 429s, so the crawl ends up slower.
- Turning off robots.txt on someone else's site. Only do it on a site you are responsible for.
- Rendering every page with JavaScript. Use the "when the page needs it" mode unless every page needs scripts.
- Calling the 1,000-URL stop a bug. It is the Free plan limit, and Licence shows it.
- Using a one-host scope on a site that redirects to www. The crawl stops at the redirect.
- Skipping the hash check. It takes one command and proves the installer is ours.
Questions and Answers
Why does my crawl show so many 403 errors?
A firewall, CDN rule or security plugin is refusing the requests. It usually reacts to speed. Lower the threads per host and raise the delay under Configuration, Speed, then crawl again.
How do I crawl a website without getting blocked?
Crawl slowly, identify the crawler honestly and respect robots.txt. On your own site, add an allow rule for the crawler's User-Agent or your IP address in the firewall.
What speed settings are safe for a strict server?
Try 1 or 2 threads per host and a delay of 1,000 to 2,000 ms. That is slow, but it rarely trips a firewall. Speed up again once the crawl runs cleanly.
What is the default User-Agent?
The crawler sends DenwixSEOCrawler/1.0 with a link to our product page. You can pick another preset or type your own string under Configuration, User-Agent.
What does a 429 status mean in a crawl?
It means Too Many Requests. The server is telling the crawler to slow down. So use fewer threads and a longer delay, and keep the robots.txt crawl-delay setting turned on.
Why does my crawl stop at 1,000 URLs?
That is the Free plan limit. It is not an error. Premium and Agency crawl with no URL limit.
I bought Premium, but the crawl still stops early. Why?
Check Configuration, Spider, Limits. The Total URL limit field may still hold an older value you saved. Reset the Total URL limit field to 5,000,000 and restart the crawl process.
How much does Premium cost?
Premium costs 150 USDT a year for one seat. Agency costs 399 USDT a year with unlimited seats and white-label reports.
Why are pages missing on my JavaScript site?
The links may only show up after the scripts run. A text-only crawl cannot see them. Go to Configuration, Spider, Rendering and turn on JavaScript rendering.
Is JavaScript rendering free?
No, it's a premium feature. You must install either Chrome or Microsoft Edge on your computer to access this.
Why is a JavaScript crawl slower?
The crawler opens each page in a real browser and waits for the scripts to finish. That takes much longer than reading the raw HTML. So render only the pages that need it.
Why does my crawl find only one page?
Usually the address is wrong, robots.txt blocks the site, or the scope is too narrow for a redirect. Check those three in order.
Can I ignore robots.txt?
Only on a site you run. Untick Respect robots.txt under Configuration, robots.txt. On every other site, leave it ticked.
Which scope should I pick?
Pick a single host if you have a simple website. Choose "subdomains" if your site will redirect to www or if you are using multiple subdomains. Choose the start folder to crawl a specific area of your site.
Why does Windows say it protected my PC?
The installer is not code-signed yet, therefore SmartScreen cannot determine whether it is safe. To install, you will need to select "More Info," and then choose "Run Anyway," after verifying the hash of the downloaded file.
How do I check the installer's SHA-256 hash?
Run Get-FileHash on the installer using PowerShell, then verify that the output of Get-FileHash matches the hash listed on the Denwix SEO Crawler webpage. Both hashes must be identical.
Does setup need admin rights?
No. The crawler installs for your Windows user only, so no admin prompt appears.
Does the app update itself?
It checks for updates by itself and tells you when a new version is ready.
What is in a debug report?
Your settings and recent log lines. The app removes passwords and keys before it saves the file to your Downloads folder.
Where do I send a debug report?
Email it to admin@denwix.com with a short note of what happened. That helps us find the cause quickly.
How do I run a website crawler test before a full crawl?
Check one URL first, then crawl one folder. Look for 403, 429 and 5xx rows. When the sample is clean, widen the scope.
Can the crawler use a proxy?
Yes. Configuration, Proxy sends crawl requests through an HTTP proxy. Use it when your network requires one.