PluginBench
MCP Server
Active

Agent Crawl Readiness MCP Server

io.github.sadri-dridi/agent-crawl-readiness

RepositoryRegistry
How to choose

What this kind of tool is for

  • Let your agent search the web or a document set and cite what it found.
  • Pull structured facts out of pages instead of copy-pasting by hand.
  • Track changes on a topic over days without re-reading everything.

How to compare servers like this

Source coverage:
Check which engines, sites, or document types the tool can actually reach.
Citation behavior:
Prefer tools that return links you can open, not just summarized claims.
Terms of service:
Some sources forbid automated access; make sure the tool stays inside them.
Maintenance signal:
Scrapers break silently when pages change; recent commits are the best proxy.

This page carries your copy-paste config: pick your MCP client in the install section on this page and use the copy button.

What is the Agent Crawl Readiness MCP server?

Bounded crawl-readiness check for agent access: robots.txt, sitemap.xml, llms.txt, homepage.

How to install Agent Crawl Readiness

Copy-paste configuration for popular MCP clients.

transport: http
Config generated by PluginBench — verify against the source before use.
~/.cursor/mcp.json
JSON config
{
  "mcpServers": {
    "agent-crawl-readiness": {
      "url": "https://agent-observatory-sensor.nolimit-observatory.workers.dev/s/agent-crawl-readiness/mcp"
    }
  }
}

Related MCP servers

Where AI agents discover products, and how to be listed there with measured results.

0
View repository →

Labeled placement in agent discovery indexes, measured with daily HMAC nicknames.

0
View repository →

Machine catalog for products that want agent clients: surfaces, hops, discovery paths.

0
View repository →

Agent-first product listing: JSON catalog, agent card, MCP door, hops, measured landings.

0
View repository →

Hops are 302s an agent followed to your site. Count them, attribute them, grow them.

0
View repository →

Leads from agents: hops attributed with a per-hop utm_id from discovery to trial to invoice.

0
View repository →