MCP Server

Active

MIT

Crawlberg MCP Server

io.github.xberg-io/crawlberg

Scrape and crawl websites into clean Markdown or JSON with a single command or API call.

What is the Crawlberg MCP server?

Crawlberg is a web scraping and crawling engine that converts websites into structured Markdown, metadata, and links. It handles JavaScript-heavy pages, respects robots.txt, detects bot filters, and provides an MCP server for AI agents plus language bindings for Rust, Python, Node.js, Go, Java, C#, Ruby, PHP, Elixir, Dart, Swift, Zig, Kotlin, and WebAssembly.

Crawlberg turns raw HTML into clean, structured data. Point it at a URL and get back Markdown with metadata (titles, links, images, JSON-LD, Open Graph), either from a single page or an entire crawled site. It handles the hard parts: JavaScript rendering via headless browser, bot-filter detection, rate limiting, SSRF protection, and robots.txt compliance. Use it from code, an AI agent, REST API, or CLI.

How to install Crawlberg

Copy-paste configuration for popular MCP clients.

transport: stdio

Config generated by PluginBench — verify against the source before use.

~/Library/Application Support/Claude/claude_desktop_config.json

{
  "mcpServers": {
    "crawlberg": {
      "command": "docker",
      "args": [
        "run",
        "-i",
        "--rm",
        "ghcr.io/xberg-io/crawlberg:1.8.0",
        "mcp"
      ]
    }
  }
}

Tools & capabilities

Tools this server exposes to the agent.

crawl — Crawl a website with configurable depth, page limits, and concurrency; supports depth-first, breadth-first, or best-first traversal.
scrape — Scrape a single URL or batch of URLs and extract structured data: text, metadata, links, images, assets, JSON-LD, Open Graph, hreflang, favicons, headings, and response headers.
markdown_conversion — Convert HTML to clean Markdown with citations, document structure, and fit-content mode.
url_filtering — Filter URLs by include/exclude patterns, BM25 relevance scoring, and sitemap discovery.
browser_rendering — Optional headless browser rendering for JavaScript-heavy single-page applications with WAF detection and bypass.
authentication — Support for HTTP Basic, Bearer token, and custom-header authentication with cookie jar management.
rate_limiting — Per-domain request throttling and concurrent crawl control.

Use cases

Extract structured content from a website and convert it to Markdown for documentation or knowledge bases.
Crawl an entire site to build a searchable index or feed data into a vector database for RAG.
Scrape product listings, pricing, or metadata from e-commerce sites for competitive analysis.
Fetch and parse JSON-LD, Open Graph, and microdata from multiple pages for SEO or metadata extraction.
Monitor website changes by crawling and comparing page content over time.

Crawlberg MCP server FAQ

What is Crawlberg?

Crawlberg is a web scraping and crawling library that converts websites into clean Markdown and structured metadata. It handles JavaScript rendering, bot detection, rate limiting, and SSRF protection out of the box.

Is Crawlberg free?

Yes, Crawlberg is open-source under the MIT License. There are also commercial offerings (Xberg Pro and Xberg Enterprise) for managed extras like proxy pools, authenticated sessions, and scheduling.

How do I use Crawlberg with Claude or Cursor?

Install the Crawlberg plugin from the marketplace in Claude Code, Cursor, or other AI coding assistants. It ships the MCP server and agent skills for site crawling and HTML-to-Markdown conversion.

What languages does Crawlberg support?

Crawlberg has bindings for Rust, Python, Node.js, Go, Java, C#, Ruby, PHP, Elixir, Dart, Swift, Zig, Kotlin (Android), and WebAssembly, plus a CLI and REST API.

Does Crawlberg require authentication?

No authentication is required to use Crawlberg. It supports optional HTTP Basic, Bearer token, and custom-header auth for crawling protected resources.

How does Crawlberg handle JavaScript-heavy sites?

Crawlberg falls back to a real headless browser for JavaScript-heavy single-page applications and includes WAF detection and bot-evasion strategies.

README (reference)

Source of truth, from the repository.

Crawlberg

Turn any website into clean, structured data. Point Crawlberg at a URL and get back Markdown, metadata, and links — from a single page or a whole site — in the language you already use.

What and Why?

You need data that lives on the web, and raw HTML is not it. Crawlberg does the crawling, scraping, and cleanup end-to-end: it fetches pages, follows links, converts each one to Markdown, and hands you structured metadata (titles, links, images, social-card and JSON-LD data) — so you skip the parsing and go straight to the content.

It runs from a single Rust core with identical results across 14 language bindings, and it handles the awkward parts for you: JavaScript-heavy pages fall back to a real headless browser, bot filters are detected and worked around, robots and sitemaps are respected, requests are throttled per domain, and requests to private or internal addresses are refused by default. Drive it from your code, an AI agent, a REST service, or the CLI.

Every part of the pipeline is a trait you can swap — the crawl frontier, rate limiter, storage, event stream, and content filters — so you can plug in your own behavior. Managed extras like proxy pools, tuned bot-evasion, authenticated sessions, scheduling, and billing live in xberg-enterprise.

Features

Feature	Description
Structured extraction	Text, metadata, links, images, assets, JSON-LD, Open Graph, hreflang, favicons, headings, response headers
Markdown conversion	Clean Markdown with citations, document structure, and fit-content mode
Concurrent crawling	Depth-first, breadth-first, or best-first traversal with configurable depth, page limits, and concurrency
14 language bindings	Rust, Python, Node.js, Ruby, Go, Java, Kotlin (Android), C#, PHP, Elixir, Dart, Swift, Zig, and WebAssembly
Smart filtering	BM25 relevance scoring, URL include/exclude patterns, robots.txt compliance, sitemap discovery
Browser rendering	Optional headless browser for JavaScript-heavy SPAs with WAF detection and bypass
Batch & streaming	Scrape or crawl hundreds of URLs concurrently; real-time crawl events via async streams
SSRF-safe by default	Refuses loopback, private, link-local, and cloud-metadata addresses; opt out via env var or `CrawlConfig`
Auth & rate limiting	HTTP Basic, Bearer, and custom-header auth with cookie jars; per-domain request throttling
MCP server & REST API	Model Context Protocol integration for AI agents plus an HTTP server with OpenAPI spec

Supported Platforms

Precompiled binaries for Linux (x86_64/aarch64), macOS (ARM64), and Windows (x64) across every binding. See the platform support reference for the full matrix.

⭐ Star this repo to show your support — it helps others discover Crawlberg.

Quick Start

Language Packages

<details open> <summary>Python</summary>

pip install crawlberg

See Python README for full documentation.

</details> <details> <summary>Node.js</summary>

npm install @xberg-io/crawlberg

See Node.js README for full documentation.

</details> <details> <summary>Rust</summary>

cargo add crawlberg

See Rust README for full documentation.

</details> <details> <summary>Go</summary>

go get github.com/xberg-io/crawlberg/packages/go

See Go README for full documentation.

</details> <details> <summary>Java</summary>

Available on Maven Central as io.xberg.crawlberg:crawlberg. See Java README for the dependency snippet and current version.

</details> <details> <summary>C#</summary>

dotnet add package XbergIo.Crawlberg

See C# README for full documentation.

</details> <details> <summary>Ruby</summary>

gem install crawlberg

See Ruby README for full documentation.

</details> <details> <summary>PHP</summary>

composer require xberg-io/crawlberg

See PHP README for full documentation.

</details> <details> <summary>Elixir</summary>

Add {:crawlberg, "~> 0.3"} to your mix.exs dependencies. See Elixir README for full documentation.

</details> <details> <summary>Dart / Flutter</summary>

dart pub add crawlberg

See Dart README for full documentation.

</details> <details> <summary>Kotlin (Android)</summary>

Available on Maven Central as io.xberg.crawlberg.android:crawlberg-android. See Kotlin README for the dependency snippet and current version.

</details> <details> <summary>Swift</summary>

Add via Swift Package Manager. See Swift README for full documentation.

</details> <details> <summary>Zig</summary>

See Zig README for installation and usage.

</details> <details> <summary>WebAssembly</summary>

npm install @xberg-io/crawlberg-wasm

See WebAssembly README for full documentation.

</details> <details> <summary>C/C++ (FFI)</summary>

C header + shared library from GitHub Releases. See FFI crate for full documentation.

</details> <details> <summary>CLI</summary>

cargo install crawlberg-cli

# Or install the prebuilt binary via cargo-binstall:
cargo binstall crawlberg-cli

brew install xberg-io/tap/crawlberg

See CLI README for full documentation.

</details>

AI Coding Assistants

Install the Crawlberg plugin from xberg-io/crawlberg. It ships the Crawlberg agent skills (site crawling, HTML→Markdown scraping, headless-Chrome fallback) plus the crawlberg MCP server, and works with every major coding agent — expand your harness below.

<details open> <summary>Claude Code</summary>

/plugin marketplace add xberg-io/crawlberg
/plugin install crawlberg@crawlberg

</details> <details> <summary>Codex CLI</summary>

/plugins add https://github.com/xberg-io/crawlberg

Then search for crawlberg and select Install Plugin.

</details> <details> <summary>Cursor</summary>

Settings → Plugins → Add from URL → https://github.com/xberg-io/crawlberg, then select crawlberg.

</details> <details> <summary>Gemini CLI</summary>

gemini extensions install https://github.com/xberg-io/crawlberg

</details> <details> <summary>Factory Droid</summary>

droid plugin marketplace add https://github.com/xberg-io/crawlberg
droid plugin install crawlberg@crawlberg

</details> <details> <summary>GitHub Copilot CLI</summary>

copilot plugin marketplace add https://github.com/xberg-io/crawlberg
copilot plugin install crawlberg@crawlberg

</details> <details> <summary>opencode</summary>

Add the package to opencode.json:

{
  "$schema": "https://opencode.ai/config.json",
  "plugin": ["@xberg-io/opencode-crawlberg"]
}

</details>

Documentation

Full guides, per-language API references, the substrate/operational model, antibot strategy, and observability live at docs.crawlberg.xberg.io.

Contributing

Contributions are welcome! See our Contributing Guide.

Part of Xberg.io

Xberg — the open-source content-intelligence engine: text, tables, and metadata from 101 formats (115 file extensions), with OCR, transcription, and code intelligence. MIT.
Xberg Pro — a complete self-hosted content-intelligence backend in a single container. Commercial.
Xberg Enterprise — the distributed, governed content-intelligence platform, scaled on Kubernetes with team governance and support. Commercial.
crawlberg — web crawling and scraping with HTML→Markdown and headless-Chrome fallback.
html-to-markdown — fast, lossless HTML→Markdown engine.
liter-llm — universal LLM API client with native bindings for 14 languages and 165 providers.
tree-sitter-language-pack — tree-sitter grammars and code-intelligence primitives.
alef — the polyglot binding generator that produces every per-language binding across the 5 polyglot repos.

License

MIT License

Related MCP servers

TypeScript

MIT

View repository →