Bot documentation

AgentReadyWebsiteBot

AgentReadyWebsiteBot checks whether a website is readable by AI agents. It reads a small number of public files, such as the homepage, robots.txt, and the .well-known agent-readiness files, and reports what it found. It does not collect content for model training, and it does not resell what it reads.

How to identify it

User-Agent
AgentReadyWebsiteBot/1.0 (+https://agentreadywebsite.io/)
Verification
Web Bot Auth, the IETF HTTP Message Signatures method (RFC 9421). Requests carry Signature, Signature-Input, and Signature-Agent headers.
Signature-Agent
"https://agentreadywebsite.io"
Public key directory
/.well-known/http-message-signatures-directory
Operator
AgentReadyWebsite. We run this bot ourselves; no third party operates it on our behalf.
Contact
[email protected]

Because identity is proven cryptographically, the User-Agent string alone is not the thing to trust. Anyone can copy a User-Agent; only we can produce a valid signature against the key published above.

When it visits

Every visit traces back to a person who asked for it. This bot does not crawl the open web looking for sites.

  • Pre-Check. Someone enters a domain on our free Pre-Check to get an Agent Readiness Score for it. The visit happens once, while they wait for the result.
  • Package generation. After a customer purchases, we detect the site's CMS and whether it sits behind a CDN, so the install instructions we send match the platform they actually use.
  • Domain Watch. For customers who bought a package, we periodically confirm the files they purchased are still hosted and still correct, for twelve months.

What it requests, and how much

A handful of requests per visit, against files a site already serves publicly to any visitor: the homepage, robots.txt, sitemap.xml, llms.txt, and the .well-known agent-readiness paths. Most are HEAD requests that check whether a file exists.

It does not log in, submit forms, follow paywalls, or request anything behind authentication. It does not attempt to enumerate a site's pages. For customers who purchase a package, the deeper page read is performed by Firecrawl on their own infrastructure, not by this bot.

How to allow it

The most durable way is to allow verified bots and signed agents at your CDN or WAF, which recognises the signature rather than a string that anyone could copy. If you prefer to match on the User-Agent, use the substring AgentReadyWebsiteBot.

If your site is blocking us and you want it unblocked, or you want us to stop visiting entirely, email [email protected] and we will act on it.

Back to AgentReadyWebsite