analytics

data-juicer: Simplify Data Processing

Name: data-juicer: Simplify Data Processing
Author: datajuicer

Founders get easy data processing for foundation models with data-juicer, a Python tool with 6.8k+ GitHub stars, ideal for data-driven startups.

7,128 stars+18★ this week432 forksPythonGuide quality 9/10 · reviewed Aug 4Updated 10/9/2026100% free · open source

What it does

Data-juicer is a tool for processing data for and with foundation models, allowing users to efficiently handle and transform data for machine learning tasks

When to use it

•When you need to preprocess large datasets for training foundation models
•When you want to integrate data processing with existing foundation model pipelines
•When you require flexible and customizable data transformation for specific machine learning tasks

Ready-to-paste prompt

python data_juicer.py --input_path data/input.csv --output_path data/output.csv --model_name 'my_foundation_model'

Heads up: Make sure you have the correct version of Python installed, as data-juicer is compatible with Python 3.8 and later, and also ensure you have the necessary dependencies installed, including pandas and numpy

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

✅ Light setup

Installs with a command or two; your AI agent can do it for you.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/data-juicer && curl -fsSL https://workflowstacks.com/api/skills/data-juicer/claude-skill -o ~/.claude/skills/data-juicer/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How data-juicer: Simplify Data Processing works

Codeflow

Free to inspect

Data-juicer: Simplify Data Processing is a very large Python project (~128k lines across 511 code files, plus 358 test files). Setup is light: installs like a normal app. Reading the code is optional. Last commit this month, Apache-2.0 license, has a test suite.

Size

Very large codebase

~128k lines · 511 code files · days to read — use, don't read

Setup

One-command install

Installs like a normal app. Reading the code is optional.

Runs on

Python · Docker

No API keys detected

Python 100%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
app.py
Where the program starts running
3
pyproject.toml
Dependencies and the commands it exposes
4
data_juicer/__init__.py
Inside data_juicer/ — the main logic begins here
5
demos/README.md
A worked example — copy this to get going

What's in each folder

data_juicer/Core code — the actual logic376 files

docs/Documentation304 files

demos/Examples you can copy155 files

tools/Helper scripts110 files

scripts/Helper scripts4 files

tests/Tests — proof it works419 files

.github/CI / automation (GitHub Actions)16 files

.pre-commit-hooks/Tool / agent settings2 files

READMEHas testsDocumentedCI checksDocker readyExamples includedApache-2.0 licenseUpdated this month

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

✅ Light setup

Installs with a command or two; your AI agent can do it for you.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/data-juicer && curl -fsSL https://workflowstacks.com/api/skills/data-juicer/claude-skill -o ~/.claude/skills/data-juicer/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

Quick Actions

Details

Creator

datajuicer

Language

Python