About Our Bot
Information about the Kant web crawler
What is Kant?
Kant is a web crawler operated by kant.au for website analysis and monitoring services. It helps our users understand the technical health, SEO performance, and content structure of their websites.
User-Agent: Kant/1.0 (+https://kant.au/bot)
What We Do
Our crawler may access your website to:
- Analyze page content, titles, meta descriptions, and headings
- Check for broken links and redirect chains
- Evaluate page load performance
- Identify SEO issues and opportunities
- Search for specific text or patterns (Site Search tool)
Our Crawl Behavior
We are committed to being a good web citizen. Our crawler:
- Respects robots.txt - We honor all Disallow and Allow directives
- Respects Crawl-delay - We follow crawl-delay directives to limit request rates
- Rate limited - We limit requests to avoid impacting your site's performance
- Identifies itself - We always send our User-Agent string so you can identify us
- Only fetches HTML - We primarily request HTML pages, not images or other assets
How to Block Our Bot
If you don't want our crawler to access your website, add the following to your robots.txt file:
User-agent: Kant Disallow: /
To block only specific sections of your site:
User-agent: Kant Disallow: /private/ Disallow: /admin/
To request a slower crawl rate (in seconds):
User-agent: Kant Crawl-delay: 5
Data Usage
Data collected by our crawler is used to:
- Provide site analysis reports to kant.au users who have added your domain
- Power our Site Search tool for finding content on websites
- Generate SEO recommendations and technical audits
We only access publicly available information. We do not attempt to access password-protected areas or bypass any access controls.
Contact Us
If you have questions or concerns about our crawler, please contact us at [email protected]