devtools

Crawlee-python: Reliable Web Scraping

Name: Crawlee-python: Reliable Web Scraping
Author: apify

Extract data with Crawlee-python, a library for startup founders, powered by 9.3k+ GitHub stars, for web scraping and automation.

From the repo · “Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.”

9,615 stars+23★ this week812 forksPythonGuide quality 8/10 · reviewed Jul 3Updated 10/21/2026100% free · open source

What it does

Crawlee-python is a web scraping and browser automation library for Python that allows startup founders to extract data from websites and download files in various formats.

When to use it

•When you need to extract data from a website that doesn't provide an API
•When you want to automate interactions with a website, such as filling out forms or clicking buttons
•When you need to download files from a website, such as HTML, PDF, JPG, or PNG files

Ready-to-paste prompt

Extract all links from a webpage using the `fetch` function and Parsel: `from crawlee import Actor; from parselp import Selector; actor = Actor(); @actor.task def scrape_page(url); async def scrape_page(url): html = await fetch(url); selector = Selector(html); links = selector.css('a::attr(href)').get(); yield links`

Heads up: Make sure you have the correct version of Python installed, as Crawlee-python requires Python 3.7 or later, and also be aware that some features, like Playwright, require additional dependencies to be installed

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

✅ Light setup

Installs with a command or two; your AI agent can do it for you.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/crawlee-python && curl -fsSL https://workflowstacks.com/api/skills/crawlee-python/claude-skill -o ~/.claude/skills/crawlee-python/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How Crawlee-python: Reliable Web Scraping works

Codeflow

Free to inspect

Crawlee-python: Reliable Web Scraping is a large Python project (~56k lines across 828 code files, plus 109 test files). Setup is light: installs like a normal app. Reading the code is optional. Last commit this month, Apache-2.0 license, has a test suite.

Size

Large codebase

~56k lines · 828 code files · ~8 h to skim

Setup

One-command install

Installs like a normal app. Reading the code is optional.

Runs on

Python

No API keys detected

Python 78%MDX 17%JavaScript 3%CSS 2%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
AGENTS.md
The instructions the AI actually follows
3
pyproject.toml
Dependencies and the commands it exposes
4
src/crawlee/__init__.py
Inside src/ — the main logic begins here

What's in each folder

src/Core code — the actual logic233 files

docs/Documentation297 files

website/Project website638 files

scripts/Helper scripts2 files

tests/Tests — proof it works111 files

.github/CI / automation (GitHub Actions)12 files

READMEHas testsDocumentedCI checksApache-2.0 licenseUpdated this month