workflowstacks
devtools

crawl4ai

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
85,093 stars+463★ this week8,824 forksPythonInstall verified Sep 14Updated 10/5/2026100% free · open source
What it does

Crawl4AI lets you fetch a web page and have an LLM extract the clean, structured content you need with a single Python call or CLI command.

When to use it
  • •You need to pull product details, FAQs, or pricing tables from competitor sites for market research.
  • •You want to feed fresh web content into a generative AI prompt without manual copy‑pasting.
  • •You must automatically summarize news articles or blog posts for a daily briefing email.
Ready-to-paste prompt
crawl4ai -u https://stripe.com/pricing -m gpt-4o --output json > stripe_pricing.json
Heads up: Crawl4AI requires an LLM API key (e.g., OPENAI_API_KEY) and will fail with a 401 error if the env var is missing or the key lacks the needed model access.
Saves to your device
Use with Claude
New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🛠️ Technical setup✓ Install verified 9/14/2026

Expect 20–40 minutes in a terminal — or let your AI agent drive it.

Try it instantly — no install
Claude Code
mkdir -p ~/.claude/skills/crawl4ai && curl -fsSL https://workflowstacks.com/api/skills/crawl4ai/claude-skill -o ~/.claude/skills/crawl4ai/SKILL.md
Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)
claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How crawl4ai works
Codeflow
Free to inspect

Crawl4ai is a very large Python project (~109k lines across 282 code files, plus 242 test files). It is a full software project: use it through its install path rather than reading it end to end. Last commit this month, Apache-2.0 license, has a test suite.

Size
Very large codebase
~109k lines · 282 code files · days to read — use, don't read
Setup
Developer setup
A real software project. Use it via its install path; don't expect to read it all.
Runs on
Python · Docker
No API keys detected
Python 99%JavaScript 1%
Where to start reading
  1. 1
    README.md
    Start here — what it does and how to install it
  2. 2
    crawl4ai/cli.py
    Where the program starts running
  3. 3
    pyproject.toml
    Dependencies and the commands it exposes
  4. 4
    crawl4ai/__init__.py
    Inside crawl4ai/ — the main logic begins here
What's in each folder
crawl4ai/Core code — the actual logic83 files
prompts/Prompts, skills & agent definitions1 files
docs/Documentation496 files
deploy/Deployment / infrastructure68 files
sbom/Folder2 files
scripts/Helper scripts2 files
tests/Tests — proof it works218 files
.github/CI / automation (GitHub Actions)14 files
READMEHas testsDocumentedCI checksDocker readyApache-2.0 licenseUpdated this month
Quick Actions
Details
Creator
unclecode
Language
Python
Category
devtools
Published
5/9/2024

Are you the creator of this tool? and earn 85% of every sale.

Show it off in your README:

Featured on WorkflowStacks
[![Featured on WorkflowStacks](https://workflowstacks.com/api/badge/crawl4ai.svg)](https://workflowstacks.com/skills/crawl4ai?utm_source=github&utm_medium=badge)
🔥 Hot this week, in your inbox
Skills like this, every Monday.

The five fastest-growing open-source AI skills, ranked by GitHub star growth. One email a week, unsubscribe anytime.

Related skills

More devtools tools founders pair with this one.

devtools★ 190,001
firecrawl – Scalable Web Scraping and Search API for Founders
Use firecrawl to quickly search, scrape, and interact with web data at any scale. Ideal for founders building AI agents, data pipelines, or content tools. 190k+ GitHub stars.
devtools★ 117,385
browser-use: Automate Web Tasks
Founders use browser-use to automate online tasks with ease, ideal for startups needing AI-powered browser automation.
devtools★ 97,371
Playwright: Unified Cross‑Browser Testing for Faster Releases
Accelerate QA with Playwright’s single API that covers Chromium, Firefox and WebKit—perfect for founders building reliable web apps. 97k+ GitHub stars.
devtools★ 2,819
Content: Automate Security
Get security automation content for compliance and security. For founders, with 2.8k+ GitHub stars
devtools★ 1,904
WebRPA: Automate Tasks
Get no-code automation for web scraping and testing. For founders, with 1.8k+ GitHub stars.
devtools★ 5,650
Secure Code with Nettacker
Get automated penetration testing with Nettacker, a free tool for startup founders, with 5.3k+ GitHub stars.