Agent Crawl Readiness MCP Server
io.github.sadri-dridi/agent-crawl-readiness
What this kind of tool is for
- Let your agent search the web or a document set and cite what it found.
- Pull structured facts out of pages instead of copy-pasting by hand.
- Track changes on a topic over days without re-reading everything.
How to compare servers like this
- Source coverage:
- Check which engines, sites, or document types the tool can actually reach.
- Citation behavior:
- Prefer tools that return links you can open, not just summarized claims.
- Terms of service:
- Some sources forbid automated access; make sure the tool stays inside them.
- Maintenance signal:
- Scrapers break silently when pages change; recent commits are the best proxy.
This page carries your copy-paste config: pick your MCP client in the install section on this page and use the copy button.
What is the Agent Crawl Readiness MCP server?
Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.
How to install Agent Crawl Readiness
Copy-paste configuration for popular MCP clients.
{
"mcpServers": {
"agent-crawl-readiness": {
"url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
}
}
}Related MCP servers

Agent Discovery Index
Where AI agents discover products, and how to be listed there with measured results.

Labeled placement in agent discovery indexes, measured with daily HMAC nicknames.

Agent Facing Catalog
Machine catalog for products that want agent clients: surfaces, hops, discovery paths.

Agent First Listing
Agent-first product listing: JSON catalog, agent card, MCP door, hops, measured landings.

Agent Hops For Your Site
Hops are 302s an agent followed to your site. Count them, attribute them, grow them.

Agent Lead Generation
Leads from agents: hops attributed with a per-hop utm_id from discovery to trial to invoice.