local-ai

Auto Round

Name: Auto Round
Author: intel

A SOTA quantization toolkit for high-accuracy low-bit LLM inference|简洁且高效的量化工具包

1,607 stars175 forksPythonUpdated 9/8/2026100% free · open source

What it does

Auto‑Round quantizes large language models to low‑bit (e.g., 4‑bit) weights while keeping accuracy comparable to full‑precision inference.

When to use it

•You need to run a GPT‑style model on a single GPU or CPU with limited VRAM.
•You want to cut inference cost without re‑training the model.
•You are deploying a pre‑trained LLM in a production service and need sub‑8‑bit speed‑up.

Ready-to-paste prompt

python -m auto_round.quantize --model_name_or_path meta-llama/Llama-2-7b-hf --output_dir ./llama2-7b-w4a8 --bits 4 --group_size 128 --scale_dtype fp16

Heads up: Auto‑Round requires PyTorch ≥ 2.0 compiled with the same CUDA version you plan to run on; mismatched CUDA/PT versions cause runtime crashes, so install `torch` via the official wheel (e.g., `pip install torch==2.2.0+cu121 -f https://download.pytorch.org/whl/torch_stable.html`).

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

✅ Light setup

Installs with a command or two; your AI agent can do it for you.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/auto-round && curl -fsSL https://workflowstacks.com/api/skills/auto-round/claude-skill -o ~/.claude/skills/auto-round/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How Auto Round works

Codeflow

Free to inspect

Auto Round is a very large Python project (~227k lines across 522 code files, plus 329 test files). Setup is light: installs like a normal app. Reading the code is optional. Last commit this month, Apache-2.0 license, has a test suite.

Size

Very large codebase

~227k lines · 522 code files · days to read — use, don't read

Setup

One-command install

Installs like a normal app. Reading the code is optional.

Runs on

Python

No API keys detected

Python 74%C++ 25%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
AGENTS.md
The instructions the AI actually follows
3
auto_round/__main__.py
Where the program starts running
4
pyproject.toml
Dependencies and the commands it exposes
5
auto_round/__init__.py
Inside auto_round/ — the main logic begins here

What's in each folder

auto_round/Core code — the actual logic340 files

docs/Documentation29 files

auto_round_extension/Folder217 files

test/Tests — proof it works313 files

.azure-pipelines/Tool / agent settings48 files

.github/CI / automation (GitHub Actions)12 files

.claude/Editor / agent settings8 files

.codex/Tool / agent settings1 files

READMEHas testsDocumentedCI checksApache-2.0 licenseUpdated this month