ai-agent

crawl4ai-rag-system

Name: crawl4ai-rag-system
Author: pdichone

51 stars31 forksPythonUpdated 1/20/2025100% free · open source

What it does

Turns any web page into a searchable knowledge source by crawling it with Crawl4AI and feeding the extracted text into a Retrieval‑Augmented Generation (RAG) pipeline you can query via a simple API.

When to use it

•You need up‑to‑date information from a specific website (e.g., competitor blog) without building a full scraper.
•Your product needs a “ask the web page” feature for customers (e.g., “what does this terms‑of‑service say?”).
•You want a quick prototype that combines web‑crawling, vector storage, and LLM answering in one Dockerized service.

Ready-to-paste prompt

curl -X POST http://localhost:8000/query -H "Content-Type: application/json" -d '{"url":"https://github.com/pdichone/crawl4ai-rag-system","question":"Summarize the deployment steps and required environment variables."}'

Heads up: The service will fail to start if `OPENAI_API_KEY` (or the configured LLM provider key) is missing or invalid; the container logs will show a 401 error. Set the key in `.env` before running `docker compose up`.

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🔑 Needs an API key

Works after you add credentials — the setup agent will ask you for them.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/crawl4ai-rag-system && curl -fsSL https://workflowstacks.com/api/skills/crawl4ai-rag-system/claude-skill -o ~/.claude/skills/crawl4ai-rag-system/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How crawl4ai-rag-system works

Codeflow

Free to inspect

Crawl4ai-rag-system is a small Python project (~2.3k lines across 15 code files). Setup is light: install Python, run one command. Last commit 21 months ago, no license file, no tests found.

Size

Small codebase

~2.3k lines · 15 code files · 19 min skim

Setup

Light setup

Install Python, run one command.

Runs on

Python

No API keys detected

Python 98%PLpgSQL 2%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
requirements.txt
Dependencies and the commands it exposes
3
examples/fast_crawl.py
A worked example — copy this to get going

What's in each folder

examples/Examples you can copy3 files

rag-crawl/Folder6 files

gen-rag-crawl/Folder5 files

.vscode/Editor / agent settings1 files

READMENo tests foundDocumentedNo CIExamples includedNo licenseLast update 21 mo ago