ai-agent

MCP Crawl4ai RAG

Name: MCP Crawl4ai RAG
Author: coleam00

Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants

2,240 stars572 forksPythonUpdated 7/25/2025100% free · open source

What it does

Turns any public webpage into a searchable RAG knowledge base that you can query with natural‑language questions.

When to use it

•You need up‑to‑date information from a competitor’s site without manually copying text.
•Your AI assistant must answer questions about a product’s documentation that lives on a public URL.
•You want to quickly prototype a chatbot that pulls facts from a handful of web pages.

Ready-to-paste prompt

python -m mcp_crawl4ai_rag.run --url https://example.com/roadmap --question "When is the next major release planned?"

Heads up: The library requires a valid OpenAI API key and Python 3.9+. If you forget to set `OPENAI_API_KEY` (or pass it to `Crawl4AIRag`) the `ask` call fails with an authentication error.

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🔑 Needs an API key

Works after you add credentials — the setup agent will ask you for them.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/mcp-crawl4ai-rag && curl -fsSL https://workflowstacks.com/api/skills/mcp-crawl4ai-rag/claude-skill -o ~/.claude/skills/mcp-crawl4ai-rag/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How MCP Crawl4ai RAG works

Codeflow

Free to inspect

MCP Crawl4ai RAG is a medium Python project (~7.5k lines across 10 code files). Setup is light: installs like a normal app — add your API keys. Reading the code is optional. Last commit 14 months ago, MIT license, has a test suite.

Size

Medium codebase

~7.5k lines · 10 code files · ~1 h to skim

Setup

One-command install

Installs like a normal app — add your API keys. Reading the code is optional.

Runs on

Python · Docker

Needs API keys (.env)

Python 98%PLpgSQL 2%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
pyproject.toml
Dependencies and the commands it exposes
3
.env.example
The API keys and settings you must provide
4
src/crawl4ai_mcp.py
Inside src/ — the main logic begins here

What's in each folder

src/Core code — the actual logic2 files

knowledge_graphs/Folder7 files

READMEHas testsThin docsNo CIDocker readyMIT licenseLast update 14 mo ago

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🔑 Needs an API key

Works after you add credentials — the setup agent will ask you for them.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/mcp-crawl4ai-rag && curl -fsSL https://workflowstacks.com/api/skills/mcp-crawl4ai-rag/claude-skill -o ~/.claude/skills/mcp-crawl4ai-rag/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

Quick Actions

Details

Creator

coleam00

Language

Python