ai-agent

mcp-crawl4ai-rag

Web Crawling and RAG Capabilities for AI Agents and AI Coding Assistants
2,269 stars+3★ this week573 forksPythonUpdated 7/25/2025100% free · open source
What it does

Turns any public webpage into a searchable RAG knowledge base that you can query with natural‑language questions.

When to use it
  • •You need up‑to‑date information from a competitor’s site without manually copying text.
  • •Your AI assistant must answer questions about a product’s documentation that lives on a public URL.
  • •You want to quickly prototype a chatbot that pulls facts from a handful of web pages.
Ready-to-paste prompt
python -m mcp_crawl4ai_rag.run --url https://example.com/roadmap --question "When is the next major release planned?"
Heads up: The library requires a valid OpenAI API key and Python 3.9+. If you forget to set `OPENAI_API_KEY` (or pass it to `Crawl4AIRag`) the `ask` call fails with an authentication error.
Saves to your device
Use with Claude
New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🔑 Needs an API key

Works after you add credentials — the setup agent will ask you for them.

Try it instantly — no install
Claude Code
mkdir -p ~/.claude/skills/mcp-crawl4ai-rag && curl -fsSL https://workflowstacks.com/api/skills/mcp-crawl4ai-rag/claude-skill -o ~/.claude/skills/mcp-crawl4ai-rag/SKILL.md
Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)
claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

Quick Actions
Details
Creator
coleam00
Language
Python
Category
ai-agent
Published
5/3/2025

Are you the creator of this tool? and earn 85% of every sale.

Show it off in your README:

Featured on WorkflowStacks
[![Featured on WorkflowStacks](https://workflowstacks.com/api/badge/mcp-crawl4ai-rag.svg)](https://workflowstacks.com/skills/mcp-crawl4ai-rag?utm_source=github&utm_medium=badge)
🔥 Hot this week, in your inbox
Skills like this, every Monday.

The five fastest-growing open-source AI skills, ranked by GitHub star growth. One email a week, unsubscribe anytime.