Desvela AI Crawler Block Test — free
Free test of whether a domain's CDN blocks AI crawlers, versus its robots.txt.
About
What a domain's CDN actually serves each AI crawler, as opposed to what its robots.txt declares. crawler_block_test(domain) sends the real user-agent of GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and CCBot at the homepage and compares each response against a browser's: a site can allow GPTBot in robots.txt and still have its CDN answer it a 403, and robots.txt cannot show that. Verdict per crawler: ok, blocked, throttled (a 429 means too fast, not not-you), degraded (a 200 with a fraction of the bytes), or error. When the site refuses the control request too, measurable is false and no verdict is claimed. Also over plain HTTP at GET https://registry.desvela.dev/crawler-block?domain=. Free, no token.
At a glance
- Category
- Marketing and SEO AI visibility and GEO
- Made for
- Business teams
- Pricing
- Free
Category, summary and these details are StealthStack's reading of the publisher's own listing. Check the publisher's page before you rely on them.
Capabilities
-
test whether a domain's CDN blocks or degrades AI crawlers -
compare a crawler's response against a browser's on the same URL -
distinguish declared robots.txt policy from observed behaviour
Provenance
- Discovered
- Crawled directly
- Identifier
- urn:air:desvela.ai:registry:crawler-block
- Catalog host
- desvela.ai · crawled directly
- Anchor check
- Passed
- Last crawled
- seen 5h ago
Structural trust only: the URN authority (desvela.ai) matches the catalog host that published this entry. This is NOT a cryptographic proof of ownership — it means the listing is self-consistent with the domain it claims. How trust works →