analytics

data-juicer: Simplify Data Processing

Name: data-juicer: Simplify Data Processing
Author: datajuicer

Founders get easy data processing for foundation models with data-juicer, a Python tool with 6.8k+ GitHub stars, ideal for data-driven startups.

6,794 stars400 forksPythonHealth Score 9/10Updated 7/28/2026100% free · open source

What it does

Data-juicer is a tool for processing data for and with foundation models, allowing users to efficiently handle and transform data for machine learning tasks

Install / run

pip install data-juicer

When to use it

•When you need to preprocess large datasets for training foundation models
•When you want to integrate data processing with existing foundation model pipelines
•When you require flexible and customizable data transformation for specific machine learning tasks

Quick start

1 Clone the data-juicer repository from GitHub using the command `git clone https://github.com/datajuicer/data-juicer`
2Navigate to the cloned repository using `cd data-juicer`
3Install the required packages using `pip install -r requirements.txt`
4Run the example script using `python examples/example.py` to test the installation
5Modify the `config.json` file to customize the data processing pipeline for your specific use case

Ready-to-paste prompt

python data_juicer.py --input_path data/input.csv --output_path data/output.csv --model_name 'my_foundation_model'

Heads up: Make sure you have the correct version of Python installed, as data-juicer is compatible with Python 3.8 and later, and also ensure you have the necessary dependencies installed, including pandas and numpy

Saves to your device