What Website Crawler Tools Do
Website crawler tools visit the pages of a site the same way a search engine bot does. They follow every link, read each page and record what they find: status codes, titles, headings, images, redirects and more. Then they point out what you should fix.
This guide covers the Denwix SEO Crawler, our desktop crawler for Windows. Every section includes a short video, so you can see each feature at work. The Free plan lets you crawl up to 1,000 URLs per crawl, and most features below work on it.
The app has 33 audit tabs and 448 filters. That may seem like a lot. However, you only need a few of them to begin, and this guide shows those first. After that, it moves on to exports, sitemaps and the Premium extras.
How to Crawl a Website: Your First Crawl
Here is how to crawl a website in four steps. Type your website address in the bar at the top. Then pick the scope: one host, every subdomain, only the start folder, or external links too.
Video: run your first crawl
Type the address, pick a scope, press Start, then read the tabs, filters and the bottom pane.
Now press Start. Pages appear in the grid as they are found, and the status bar shows the progress of the crawl. So you can look around before the crawl is complete.
Read the results
Click a tab on the left, such as Page titles. Then use the filter list above the grid, for example to show only missing or duplicate titles.
Look at one URL in detail
Finally, click any row. The bottom pane then shows everything about that URL: its links in and out, its headers, images and more.
If your crawl ends too soon or gets stuck, our crawler troubleshooting guide has a fix for every common cause.
Find Any URL with Search
On a big site, scrolling is slow. So use the search box above the grid to filter the rows, whichever tab you are on.
Video: search and filter the grid
Jump to search with the slash key, match words, search one column and hide rows with a minus sign.
Press / anywhere to jump to the box. Type a few words, and you get the rows that contain all of them. Put a column name in front to search only that column, like title:pricing. Numbers work too, such as words:>500.
A minus sign makes rows disappear, so -draft removes every row with that word. And if you cannot remember a trick, press ?. It lists every keyboard shortcut in the app.
Right-Click Actions
Right-click any row to see every option for that URL. That saves you a trip through the menus.
Video: right-click a row or a column
Copy, open, re-crawl, check or exclude a URL, and sort, hide or copy a column.
You can copy the URL or the whole row, open the page in your browser, or fetch it again with Re-Spider. Re-Spider is handy after a fix: you check one page without running the whole crawl again.
The same menu can check a URL's index status, validate it, or exclude it from future crawls. Also, right-click a column header to sort, hide or copy that column.
Include and Exclude URLs
Sometimes you only want part of a site. Or you want to skip pages that add nothing, such as tag archives. In both cases, use Include and Exclude.
Video: crawl only the URLs you need
Add one regular expression per line under Exclude or Include. The crawler skips a URL that matches both.
Open Configuration → Include / Exclude. Add a single regular expression each time on a separate line. The example below shows how to skip tag pages using an Exclude line:
/tag/Include works the other way round: only matching URLs get crawled. So /blog/ under Include keeps the crawl inside the blog. If a URL matches both lists, Exclude wins and the crawler skips it.
The Issues Tab
An audit really begins in the Issues tab. The crawler works out the issues when the crawl ends, so complete a crawl first.
Video: read the Issues tab
See errors, warnings and notices with their URL counts, then open an issue for its meaning and fix.
The tab lists errors, warnings and notices, with the number of URLs each one affects. Click an issue to see what it means, how to fix it and every URL it affects. The line at the top gives you a starting point, so you fix the biggest problems first.
For a single page, you can also run our free SEO Checker in the browser. The crawler does the same kind of checks, but for the whole site at once.
Visualisations
Visualizations can turn your crawl into pictures of your site. It is very easy to see structural issues, even for clients who have no experience using spreadsheets.
Video: see your site as a graph
Explore the Crawl Tree Graph and the Force-Directed diagram, colour the nodes and save the picture.
The Crawl Tree Graph illustrates how the crawler got to every page. You can use the scroll feature to zoom in on the graph and use the dragging action to navigate through it. Additionally, the Force-Directed Diagram displays the identical links as a network which will help you identify those pages that sit deep in the site and have few links.
You can colour the nodes by status or depth. Then save the picture as SVG, PNG or HTML for a report.
Export Data and Reports
Crawl finished? Next, get the data out to share it or work on it in a spreadsheet.
Video: export data and reports
Export a tab as CSV, use Bulk Export sets, or make a PDF, HTML, JSON or CSV report.
Open any tab, then press Export to save that tab as a CSV file. Bulk Export has ready sets for links, images, issues and more, plus the current tab or filter. The Free plan includes over 30 core bulk exports, and Premium unlocks all of them.
Finally, the Reports menu makes a PDF audit, an HTML report, or a JSON or CSV export. Often, a PDF is the easiest thing to send to a client.
Build and Audit XML Sitemaps
After a crawl, the crawler can build your XML sitemap from the pages it found.
Video: build and check sitemaps
Make an XML or image sitemap from the crawl, or audit the sitemap your site already has.
Open Sitemaps → XML Sitemap. You get the file, and the app writes nothing to your server. So you upload it yourself, where you want it. Image Sitemap does the same for images.
Already have a sitemap? Then Audit Published Sitemap checks it against the crawl. For a quick sitemap without the app, our online XML Sitemap Generator works in the browser.
Saved Crawls and Crawl Files
The app saves every crawl on your PC, so you can open it again any time. Nothing goes to a cloud account.
Video: open, rename and move crawls
Manage stored crawls under File, Crawls, and move a whole crawl to another PC as one file.
Open File → Crawls to open, rename, export or delete a stored crawl. Also, Export Crawl File saves a whole crawl as one file. Open it on another computer with Import Crawl File, for example to hand a crawl to a colleague.
AI and Google Integrations
Premium can provide you with AI and Google data for your crawls. Your own keys allow you to pay the provider directly, giving you complete control over costs.
Video: connect AI and Google keys
Add an AI provider key and your PageSpeed Insights and Google service-account keys.
Open Configuration → AI Integrations. Pick OpenAI, Anthropic, Gemini or any compatible service, and paste your key. Then, under API Access, add a PageSpeed Insights key and a Google service-account key for Search Console and GA4.
Your keys stay on this computer. They go only to the service you chose, and never to us.
White-Label Reports
Agencies can put their own brand on the audit reports, so clients see the agency, not us.
Video: brand your reports
Add your agency name, logo, colour and contact line under Reports, White-label Report Settings.
Open Reports → White-label Report Settings. Add your agency name, logo, colour and contact line. The PDF and HTML reports then carry no Denwix branding. White-label reports are part of the Agency plan.
Free, Premium and Agency
The Free plan crawls up to 1,000 URLs, with no limit on the number of crawls. Premium costs 150 USDT a year and removes the URL limit. It also adds JavaScript rendering, all bulk exports and the AI and Google integrations. Agency costs 399 USDT a year, with unlimited seats and white-label reports.
Video: upgrade to Premium or Agency
Pick a plan in the app, pay in USDT or with Binance Pay, and the app activates itself.
To upgrade, click Free — Upgrade at the top right, and pick Premium or Agency. Then click Continue to payment, and the order page opens in your browser. Pay in USDT or with Binance Pay, paste your transaction ID and come back. The app activates itself, usually within minutes.
Already have a key? Open Licence → Enter Licence Key and paste it.
Choosing Between SEO Crawler Tools
SEO crawler tools come in two kinds. Cloud crawlers run on someone else's servers and keep your data there. Desktop crawlers, like ours, run on your own PC. As a result, your crawl data stays with you, and big crawls do not cost credits.
So ask a few questions before you choose. How many URLs does your largest site have? Do you need JavaScript rendering? Will clients see the reports? And do you need it on Windows, Mac or both? The Denwix SEO Crawler runs on 64-bit Windows 10 and 11 only.
The simplest test is a real one. Install the free version, crawl your own site and read the Issues tab. If the fixes it suggests make sense to you, the tool fits.
Questions and Answers
What do website crawler tools do?
They visit each page of a site, follow its links and record status codes, titles, headings, images and redirects. Then they list the problems they find.
How do I crawl a website with the Denwix SEO Crawler?
Type the address, pick a scope and press Start. Pages appear in the grid as the crawl runs.
Which scope should I choose for my first crawl?
One host suits most sites. Choose Subdomains if the site redirects to www or uses other subdomains.
Can I look at results before the crawl ends?
Yes. The grid fills as the crawl is in progress, so you can open both tabs and rows as the crawl continues. After the crawl completes, the Issues tab will fill with data.
How do I search one column only?
Type the column name, a colon and the word, like title:pricing. A minus sign in front of a word hides rows that contain it.
What keyboard shortcut opens the search box?
Press the slash key anywhere. The question mark key lists every shortcut.
What does Re-Spider do?
It fetches one URL again. Use it after you fix a page, so you can check the fix without running the whole crawl again.
How do I skip tag pages or other sections?
Add a regular expression such as /tag/ under Configuration, Include / Exclude, Exclude. One pattern per line.
What happens when a URL matches Include and Exclude?
Exclude wins, and the crawler skips the URL.
Why is the Issues tab empty?
The crawler works out issues when a crawl ends. Let the crawl finish, or stop it, and the tab fills.
Can I save a visualisation?
Yes, you can save the Crawl Tree Graph or Force-Directed diagram as an SVG, PNG, or HTML file.
Which file formats can I export?
Tabs export as CSV. The Reports menu makes PDF and HTML reports and JSON or CSV exports.
Does the sitemap tool change my server?
No. You get the XML file, and the app writes nothing to your server. You upload it yourself.
Where does the app keep my crawls?
On your PC. Open them again under File, Crawls, or move one to another computer with Export Crawl File and Import Crawl File.
Which AI services can I connect?
OpenAI, Anthropic, Gemini or any compatible service. You paste your own key, and it is a Premium feature.
Are my API keys sent to Denwix?
No. Keys stay on your computer and go only to the service you chose.
Can I connect Search Console and GA4?
Yes, on Premium. Add a Google service-account key under Configuration, AI Integrations, API Access.
Which plan includes white-label reports?
The Agency plan, at 399 USDT a year with unlimited seats.
How do I pay for Premium?
Click the plan pill at the top right, pick a plan and pay in USDT or with Binance Pay. The app activates itself, usually within minutes.
Is there a limit on the number of crawls?
No. All of the plans (even the Free one) have an endless number of crawls. However, there is a limit on the maximum number of URLs per crawl on the Free plan, which is 1,000.
Does the crawler run on Mac?
Not yet. It runs on 64-bit Windows 10 and Windows 11.
How are desktop and cloud SEO crawler tools different?
A desktop crawler runs on your PC and keeps the data there. A cloud crawler runs on remote servers and often charges per crawl or per page.