Cairrot helps businesses understand how they appear in AI search engines such as ChatGPT, Perplexity, Gemini and Claude. To do that, our crawler occasionally visits public websites and checks whether they’re easy for AI systems to find and understand.

What our crawler does

– Reads publicly available pages only: your homepage, robots.txt, sitemap.xml, llms.txt, and pages linked from your homepage, such as your privacy policy, terms and about page.

– Sends a small number of ordinary GET and HEAD requests, at most once every few days.

– Always identifies itself with a User-Agent that contains Cairrot.

What our crawler never does

– Test for security vulnerabilities or send exploit payloads

– Try to log in, submit forms or access private areas

– Probe administrative, configuration or other sensitive files

– Hide its identity or pretend to be another crawler

How to recognize it

Our requests use one of these User-Agents:

User-Agent	Purpose
CairrotReadinessBot/1.0 (+https://cairrot.ai/bot)	AI Readiness checks
Mozilla/5.0 (compatible; CairrotBot/1.0)	Reading homepage and page content for insights
Mozilla/5.0 (compatible; Cairrot/1.0; +https://cairrot.ai)	Fetching your site’s title and description

How to block our crawler

Option 1: robots.txt (recommended)

Add these lines to the `robots.txt` file at the root of your site:

User-agent: CairrotReadinessBot
Disallow: /

User-agent: CairrotBot
Disallow: /

Option 2: Block by User-Agent at your firewall or server

Block any request whose User-Agent contains `Cairrot` (case-insensitive).

Cloudflare

(Security → WAF → Custom rules):
(lower(http.user_agent) contains "cairrot")
Action: **Block**

Nginx

if ($http_user_agent ~* "cairrot") {
    return 403;
}

Apache (.htaccess)

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} cairrot [NC]
RewriteRule .* - [F,L]

Request a permanent opt-out

Prefer that we never visit your site? Please email support@cairrot.co and we’ll add your domain to our permanent exclusion list, so no Cairrot crawler will visit it, whatever your robots.txt says. We handle requests within 3 business days and confirm by email.

Questions or Concerns

If you believe our crawler has behaved unexpectedly, contact support@cairrot.com. Please include your domain and, if you have them, the times and IP addresses of the requests.