ai-crawler-registry: machine-readable registry of AI crawlers with measured block rates

Key points

1. Each entry carries robots.txt tokens, user-agent strings, crawler purpose, and compliance status, with block-rate data measured across more than 4,194 domains.
2. The dataset separates training crawlers from answer-engine and retrieval crawlers run by the same operator, including OpenAI, Anthropic, and Google, and records which agents state that they disregard robots.txt.
3. Licensed CC BY 4.0 with a live API.