PluginBench
MCP Server
Maintained
AGPL-3.0

TheCrawler MCP Server

io.github.manchittlab/thecrawler

RepositoryRegistry
How to choose

What this kind of tool is for

  • Let your agent search the web or a document set and cite what it found.
  • Pull structured facts out of pages instead of copy-pasting by hand.
  • Track changes on a topic over days without re-reading everything.

How to compare servers like this

Source coverage:
Check which engines, sites, or document types the tool can actually reach.
Citation behavior:
Prefer tools that return links you can open, not just summarized claims.
Terms of service:
Some sources forbid automated access; make sure the tool stays inside them.
Maintenance signal:
Scrapers break silently when pages change; recent commits are the best proxy.

This page carries your copy-paste config: pick your MCP client in the install section on this page and use the copy button.

What is the TheCrawler MCP server?

Universal web scraper with LLM-ready markdown, RAG chunking, PDF/DOCX support.

How to install TheCrawler

Copy-paste configuration for popular MCP clients.

transport: stdio
Config generated by PluginBench — verify against the source before use.
~/Library/Application Support/Claude/claude_desktop_config.json
JSON config
{
  "mcpServers": {
    "thecrawler": {
      "command": "npx",
      "args": [
        "-y",
        "thecrawler"
      ]
    }
  }
}

Related MCP servers

REReturnCheck logo

Can this product actually be returned? The merchant's exact clause, its URL and the date.

0
JavaScript
View repository →

AI brand monitor: what ChatGPT and Claude say about your brand, press tracking, executive briefs.

View repository →

NHC storm tracks, forecast advisories, watches/warnings, and HURDAT2 data

6
Python
MIT
View repository →

NWS surface wind observations (ASOS/AWOS) via Weather.gov and Iowa Environmental Mesonet APIs

6
Python
MIT
View repository →
OPOpenLMNP logo

OpenLMNP

Active

Accounting MCP for French LMNP furnished rentals: income, expenses, depreciation, tax return & FEC

3
PHP
AGPL-3.0
View repository →

Create, edit, review, and explicitly publish Live or Snapshot Markdown Documents in mdedit.ai.

View repository →