workflowstacks
devtools

Crawl4ai

Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
78,834 stars8,170 forksPythonUpdated 8/20/2026100% free ยท open source

About this skill

๐Ÿš€๐Ÿค– Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
How Crawl4ai works
Codeflow
Free to inspect

Crawl4ai is a very large Python project (~109k lines across 282 code files, plus 242 test files). It is a full software project: use it through its install path rather than reading it end to end. Last commit this month, Apache-2.0 license, has a test suite.

Size
Very large codebase
~109k lines ยท 282 code files ยท days to read โ€” use, don't read
Setup
Developer setup
A real software project. Use it via its install path; don't expect to read it all.
Runs on
Python ยท Docker
No API keys detected
Python 99%JavaScript 1%
Where to start reading
  1. 1
    README.md
    Start here โ€” what it does and how to install it
  2. 2
    crawl4ai/cli.py
    Where the program starts running
  3. 3
    pyproject.toml
    Dependencies and the commands it exposes
  4. 4
    crawl4ai/__init__.py
    Inside crawl4ai/ โ€” the main logic begins here
What's in each folder
crawl4ai/Core code โ€” the actual logic83 files
prompts/Prompts, skills & agent definitions1 files
docs/Documentation496 files
deploy/Deployment / infrastructure68 files
sbom/Folder2 files
scripts/Helper scripts2 files
tests/Tests โ€” proof it works218 files
.github/CI / automation (GitHub Actions)14 files
READMEHas testsDocumentedCI checksDocker readyApache-2.0 licenseUpdated this month
Quick Actions
Details
Creator
unclecode
Language
Python
Category
devtools
Published
5/9/2024

Are you the creator of this tool? Claim your listing โ†’ and earn 85% of every sale.

Related skills

More devtools tools founders pair with this one.

devtoolsโ˜… 171,506
Scale web data with firecrawl
Get web data at scale with firecrawl, for founders. 132k+ GitHub stars.
devtoolsโ˜… 107,682
browser-use: Automate Web Tasks
Founders use browser-use to automate online tasks with ease, ideal for startups needing AI-powered browser automation.
devtoolsโ˜… 95,478
Puppeteer: Automate Browsers
Get browser automation with Puppeteer. For founders who need reliable testing and automation tools.
devtoolsโ˜… 94,927
Playwright: Simplify Web Testing
Get easy cross-browser testing with Playwright, ideal for startup founders and developers, with 90k+ GitHub stars
devtoolsโ˜… 70,437
Streamline IT with Ansible
Deploy and maintain apps with ease using Ansible, a simple automation platform, ideal for founders and dev teams, with 69k+ GitHub stars.
devtoolsโ˜… 64,009
Scrapy: Fast Web Scraping
Extract web data with Scrapy, a Python framework for founders. 62k+ GitHub stars.