fastCRW MCP Server
io.github.us/fastcrw
Open-source Rust web crawler and search engine for AI agents—scrape, crawl, map, and search with markdown or JSON output.
What this kind of tool is for
- Let your agent search the web or a document set and cite what it found.
- Pull structured facts out of pages instead of copy-pasting by hand.
- Track changes on a topic over days without re-reading everything.
How to compare servers like this
- Source coverage:
- Check which engines, sites, or document types the tool can actually reach.
- Citation behavior:
- Prefer tools that return links you can open, not just summarized claims.
- Terms of service:
- Some sources forbid automated access; make sure the tool stays inside them.
- Maintenance signal:
- Scrapers break silently when pages change; recent commits are the best proxy.
This page carries your copy-paste config: pick your MCP client in the install section on this page and use the copy button.
What is the fastCRW MCP server?
fastCRW is an open-source, self-hostable Rust crawler and search engine designed for AI agents. It converts URLs into clean markdown or structured JSON and supports scraping, crawling, mapping, searching, and extracting data from web pages. Available as a managed API with 1000 free credits or as a local binary.
fastCRW turns URLs into clean markdown or JSON output with a single engine for search, scrape, map, crawl, and extract operations. Run it locally as a Rust binary or use the managed API. It's optimized for AI agents and benchmarks show it recovers more accurate data than competitors while using minimal RAM (~14 MB).
How to install fastCRW
Copy-paste configuration for popular MCP clients.
{
"mcpServers": {
"fastcrw": {
"url": "https://fastcrw.com/mcp"
}
}
}Tools & capabilities
Tools this server exposes to the agent.
Scrape— Convert one URL to markdown, HTML, links, screenshots, or schema JSONCrawl— Follow a bounded site crawl and collect its pagesMap— Discover URLs without scraping every pageSearch— Search the web and optionally scrape selected resultsExtract— Produce structured fields from one or many URLs
Use cases
- Extract structured data from websites for AI training or analysis
- Search the web and automatically scrape relevant results for context
- Crawl entire websites to build knowledge bases or site maps
- Convert web content to markdown for LLM processing
- Monitor and extract data from multiple URLs in batch operations
fastCRW MCP server FAQ
fastCRW is an open-source web crawler and search engine written in Rust. It converts URLs into clean markdown or JSON and supports scraping, crawling, mapping, searching, and extracting data. It's designed for AI agents and available as both a managed API and self-hosted binary.
Yes. The managed API offers 1000 free credits with no credit card required (sign up with GitHub). You can also run it locally as a self-hosted Rust binary at no cost.
Run `npx -y crw-mcp@latest install` to install the MCP server and CRW skill in your detected AI tools. Alternatively, run `crw setup` which includes this step. Manual setup instructions are available in the docs.
For the managed API, yes—get a free API key at fastcrw.com/register. For local/self-hosted mode, no API key is needed; just install and run the Rust binary.
fastCRW outputs markdown, HTML, JSON (including schema), links, and screenshots. You can specify which formats you need when scraping.
Yes. fastCRW is open-source (AGPL-3.0) and can be self-hosted. You control renderers, search, authentication, proxies, and capacity. Installation and self-hosting guides are in the documentation.
README (reference)
Source of truth, from the repository.
One-command install
curl -fsSL https://fastcrw.com/install | sh
Then point it at the Cloud:
crw setup
1000 free credits, no credit card. Managed proxies, JS rendering and search, with nothing to run or keep up to date. Get my free key →
The same command configures a fully local install if you would rather run it yourself. macOS and Linux, Intel and ARM. More install options →
What it does
| Operation | Outcome |
|---|---|
| Scrape | One URL to markdown, HTML, links, screenshots, or schema JSON |
| Crawl | Follow a bounded site crawl and collect its pages |
| Map | Discover URLs without scraping every page |
| Search | Search the web and optionally scrape selected results |
| Extract | Produce structured fields from one or many URLs |
Why fastCRW
On Firecrawl's own public 1,000-URL dataset, fastCRW recovered more truth than Crawl4AI and Firecrawl, matched the fastest median latency, and idled at ~14 MB RAM.
<p align="center"> <a href="BENCHMARKS.md"> <img src=".github/benchmarks/bench-radar.svg" alt="fastCRW compared with Crawl4AI and Firecrawl on truth-recall, unique recoveries, median latency, download size, and recall depth" width="100%"> </a> </p> <p align="center"><sub><a href="BENCHMARKS.md">Methodology, full numbers, and how to reproduce it</a></sub></p>Choose how you use it
CLI
crw https://example.com # scrape, works right after install
crw search "rust async runtime" # search, after `crw setup`
Python SDK
Using Cloud? Get an API key, then export it once:
export CRW_API_KEY="crw_live_..."
pip install crw
from crw import CrwClient
client = CrwClient()
page = client.scrape("https://example.com", formats=["markdown"])
print(page["markdown"])
<details>
<summary><strong>Node.js</strong></summary>
npm install crw-sdk
import { CrwClient } from "crw-sdk";
const client = new CrwClient();
const page = await client.scrape("https://example.com", {
formats: ["markdown"],
});
console.log(page.markdown);
</details>
Local mode and more SDK examples → · REST API →
MCP for AI agents
npx -y crw-mcp@latest install
Installs the CRW skill and MCP server in your detected AI tools. crw setup can
also do this step, so either path is enough.
Manual setup →
Choose where it runs
| Managed API | Local / self-hosted | |
|---|---|---|
| Best for | Zero infrastructure and managed scaling | Data control, private networks, or custom infrastructure |
| Start | Create an API key, then crw setup | Install and run crw <URL> |
| Operations | Managed proxies, billing, and hosted capabilities | You choose renderers, search, auth, proxies, and capacity |
Capabilities and response shapes can differ by deployment:
/v1/capabilities · response shapes
Learn more
Contributing
The workspace requires Rust 1.85 or newer:
git clone https://github.com/us/crw
cd crw
make check-fast
Engine and MCP server: AGPL-3.0. Python and TypeScript SDKs: MIT. Embedding license: hello@fastcrw.com.
Star History
<a href="https://www.star-history.com/?repos=us%2Fcrw&type=date&legend=top-left"> <picture> <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=us/crw&type=date&theme=dark&legend=top-left&sealed_token=Pe6pRWL7lqTM-St9eo-Cmpk5kYNyuyun0krw9eVZQFIrm3g_R2h46IW6wfNalPXquMsWSNCgKqiar1YVo9MGy2IZmN5Lz6rjZcjBCw6bCcRHORKORFRi9A" /> <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=us/crw&type=date&legend=top-left&sealed_token=Pe6pRWL7lqTM-St9eo-Cmpk5kYNyuyun0krw9eVZQFIrm3g_R2h46IW6wfNalPXquMsWSNCgKqiar1YVo9MGy2IZmN5Lz6rjZcjBCw6bCcRHORKORFRi9A" /> <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=us/crw&type=date&legend=top-left&sealed_token=Pe6pRWL7lqTM-St9eo-Cmpk5kYNyuyun0krw9eVZQFIrm3g_R2h46IW6wfNalPXquMsWSNCgKqiar1YVo9MGy2IZmN5Lz6rjZcjBCw6bCcRHORKORFRi9A" /> </picture> </a> <details> <summary>Contributors</summary> <!-- contributors:start --> <p align="center"> <a href="https://github.com/us" title="us"><img src="https://github.com/us.png?size=96" width="48" height="48" alt="us"/></a> <a href="https://github.com/santhreal" title="santhreal"><img src="https://github.com/santhreal.png?size=96" width="48" height="48" alt="santhreal"/></a> <a href="https://github.com/AsheTheWings" title="AsheTheWings"><img src="https://github.com/AsheTheWings.png?size=96" width="48" height="48" alt="AsheTheWings"/></a> <a href="https://github.com/adambenhassen" title="adambenhassen"><img src="https://github.com/adambenhassen.png?size=96" width="48" height="48" alt="adambenhassen"/></a> <a href="https://github.com/paoloantinori" title="paoloantinori"><img src="https://github.com/paoloantinori.png?size=96" width="48" height="48" alt="paoloantinori"/></a> <a href="https://github.com/mj520" title="mj520"><img src="https://github.com/mj520.png?size=96" width="48" height="48" alt="mj520"/></a> </p> <!-- contributors:end --> </details><sub>Please respect website policies. Crawl and map follow robots.txt by default.</sub>
Related MCP servers

CRW Web Scraper
Turn URLs into clean markdown or JSON with one engine for search, scrape, map, crawl, and extract.

Statsnet
Background check any company in the world: registration, executives, courts and finances.

Joint Genesis Research
Physician reviewed research on joint pain, osteoarthritis, and relief options from Dr. Mark Weis
Video analytics for TikTok, Instagram, and YouTube. Track, analyze, and discover content.
View repository →Verify emails and domains for routing, disposable providers, role accounts, and SMTP risk.
View repository →Create, edit, organize, publish, and configure JustBlogged blogs from MCP clients.
View repository →
