ai-agent

MediaCrawler

Name: MediaCrawler
Author: NanmiCoder

小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频｜评论爬虫、微博帖子｜评论爬虫、百度贴吧帖子｜百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

63,580 stars12,375 forksPythonUpdated 8/14/2026100% free · open source

What it does

Crawls posts and their comments from Chinese social platforms (Xiaohongshu, Douyin, Kuaishou, Bilibili, Weibo, Baidu Tieba, Zhihu) into structured JSON files.

When to use it

•You need to analyze user sentiment on a competitor’s post across multiple Chinese platforms.
•You want to build a dataset of Q&A comments from Zhihu for training a language model.
•You are monitoring brand mentions in real‑time by pulling new comments from TikTok‑style videos.

Ready-to-paste prompt

python run.py --platform douyin --type video --url https://www.douyin.com/video/7145678901234567890 --output ./douyin_comments.json

Heads up: The tool relies on Selenium + ChromeDriver; you must install ChromeDriver that matches your local Chrome version and ensure it is in your system PATH, otherwise the crawler will abort with a driver‑not‑found error.

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🛠️ Technical setup

Expect 20–40 minutes in a terminal — or let your AI agent drive it.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/mediacrawler && curl -fsSL https://workflowstacks.com/api/skills/mediacrawler/claude-skill -o ~/.claude/skills/mediacrawler/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How MediaCrawler works

Codeflow

Free to inspect

MediaCrawler is a large Python project (~33k lines across 200 code files, plus 24 test files). It is a full software project: use it through its install path rather than reading it end to end. Last commit this month, Other license, has a test suite.

Size

Large codebase

~33k lines · 200 code files · ~5 h to skim

Setup

Developer setup

A real software project. Use it via its install path; don't expect to read it all.

Runs on

Python

Needs API keys (.env)

Python 96%TypeScript 4%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
main.py
Where the program starts running
3
package.json
Dependencies and the commands it exposes
4
.env.example
The API keys and settings you must provide

What's in each folder

api/Backend / API10 files

docs/Documentation59 files

webui/Folder66 files

media_platform/Folder65 files

store/Folder20 files

tools/Helper scripts14 files

config/Configuration10 files

proxy/Folder9 files

READMEHas testsDocumentedCI checksOther licenseUpdated this month