Bot Info - WizardsTool

What is WizardsTool Bot?

WizardsTool Broken-links Spider is a web crawler operated by wizardstool.com. It discovers and analyzes links on websites to detect broken links, redirects, and other link-related issues. The data collected is used to generate reports that help website owners maintain healthy link structures.

User-Agent String

The bot identifies itself with the following User-Agent header:

WizardsTool Broken-links Spider/1.0

You can identify requests from our bot in your server access logs by looking for this User-Agent string:

203.0.113.50 - - [20/Jul/2024:03:41:29 +0000] "GET /page HTTP/1.1" 200 7330 "-" "WizardsTool Broken-links Spider/1.0"

How WizardsTool Bot Crawls

The bot starts with a URL provided by a WizardsTool user and follows hyperlinks it discovers on the pages. During the crawl, it collects information about each URL including HTTP status codes, content types, redirects, and link attributes.

Concurrent Requests

3

Max URLs per Crawl

Varies by plan

Request Timeout

15 seconds

Respects robots.txt

Yes

Rate Limiting

WizardsTool Bot respects HTTP 429 (Too Many Requests) responses. When your server returns a 429 status:

The bot honors the Retry-After header if provided
It retries up to 3 times with exponential backoff
Maximum retry delay is 5 minutes
If rate limiting persists, the bot will stop crawling your site

How to Block WizardsTool Bot

WizardsTool Bot fully respects robots.txt rules. To prevent the bot from crawling your website, add the following to your robots.txt file:

User-agent: WizardsTool
Disallow: /

To block crawling of specific directories only:

User-agent: WizardsTool
Disallow: /private/
Disallow: /admin/

Changes to your robots.txt file are respected on the next crawl. The bot reads robots.txt before crawling any pages on your site.

What Data Is Collected

The bot collects the following information for each URL it visits:

HTTP status code (200, 301, 404, etc.)
Content type (MIME type)
Redirect destinations
Link attributes (rel, target, anchor text)
URL schema (HTTP/HTTPS)
Subdomain information

The bot does not store page content, submit forms, or interact with JavaScript. All collected data is used solely to generate link analysis reports for the user who initiated the crawl.

Frequently Asked Questions

Why is WizardsTool Bot crawling my website?

A WizardsTool user has submitted your website URL for a broken link analysis. The bot crawls the site to discover all links and check their status.

The bot is requesting pages that don't exist. Is this a problem?

No. The bot follows all links found on your pages. If a link on your site points to a non-existent page, the bot will request it and record the 404 status — that is exactly the kind of broken link the tool is designed to detect.

Does the bot follow nofollow links?

Yes. The bot follows nofollow links to provide a complete picture of all links on your website, including those that search engines may ignore.

I blocked the bot via robots.txt, but it's still crawling. Why?

The bot reads robots.txt at the start of each crawl session. If a crawl was already in progress when you updated the file, the changes will take effect on the next crawl.

Contact

If you have questions or concerns about WizardsTool Bot, please reach out at [email protected].