mcp-server

Scrapling

Name: Scrapling
Author: D4Vinci

️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

76,176 stars7,617 forksPythonUpdated 8/23/2026100% free · open source

What it does

Scrapling lets you turn a single URL or a whole site into structured data with a few CLI commands, handling pagination, JS rendering, and data extraction automatically.

Install / run

pip install git+https://github.com/D4Vinci/Scrapling.git

When to use it

•You need to pull product listings, pricing or reviews from a competitor’s site without writing a custom scraper.
•You have to harvest public data from a paginated directory (e.g., startup databases, event listings) on a schedule.
•You want a quick proof‑of‑concept that can handle JavaScript‑heavy pages and then scale to a full crawl.

Quick start

11. Clone the repo to get the example configs: `git clone https://github.com/D4Vinci/Scrapling.git && cd Scrapling`
22. Install the bundled Python requirements: `pip install -r requirements.txt`
33. Install the headless browser Playwright needs (run once): `playwright install chromium`
44. Test the built‑in example – it crawls a demo e‑commerce site and writes JSON: `python -m scrapling run examples/ecommerce.yaml --output demo_products.json`
55. Create your own config (e.g., `my_job.yaml`) with `start_urls`, `selectors`, and optional `pagination` rules, then run: `python -m scrapling run my_job.yaml --output my_data.json`

Ready-to-paste prompt

python -m scrapling run my_job.yaml --output startups.json --concurrency 5 --max-pages 200

Heads up: Scrapling relies on Playwright's Chromium; if Chromium isn’t installed or your environment blocks headless browsers (e.g., on some corporate servers), the crawl will fail – run `playwright install chromium` and ensure the process can launch a headless browser.

Saves to your device

Topics

ai-scraping

automation

crawler

crawling

crawling-python

data

data-extraction

mcp

mcp-server

playwright

python

scraping

selectors

stealth

web-scraper

web-scraping

web-scraping-python

webscraping

xpath

How Scrapling works

Codeflow

Free to inspect

Scrapling is a large Python project (~17k lines across 64 code files, plus 62 test files). Setup is light: runs with Docker. Last commit this month, BSD-3-Clause license, has a test suite.

Size

Large codebase

~17k lines · 64 code files · ~2 h to skim

Setup

Light setup

Runs with Docker.

Runs on

Python · Docker

No API keys detected

Python 100%

Where to start reading