local-ai

auto-round

Name: auto-round
Author: intel

A SOTA quantization toolkit for high-accuracy low-bit LLM inference|简洁且高效的量化工具包

1,687 stars+42★ this week194 forksPythonUpdated 10/23/2026100% free · open source

What it does

Auto‑Round quantizes large language models to low‑bit (e.g., 4‑bit) weights while keeping accuracy comparable to full‑precision inference.

When to use it

•You need to run a GPT‑style model on a single GPU or CPU with limited VRAM.
•You want to cut inference cost without re‑training the model.
•You are deploying a pre‑trained LLM in a production service and need sub‑8‑bit speed‑up.

Ready-to-paste prompt

python -m auto_round.quantize --model_name_or_path meta-llama/Llama-2-7b-hf --output_dir ./llama2-7b-w4a8 --bits 4 --group_size 128 --scale_dtype fp16

Heads up: Auto‑Round requires PyTorch ≥ 2.0 compiled with the same CUDA version you plan to run on; mismatched CUDA/PT versions cause runtime crashes, so install `torch` via the official wheel (e.g., `pip install torch==2.2.0+cu121 -f https://download.pytorch.org/whl/torch_stable.html`).

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

✅ Light setup

Installs with a command or two; your AI agent can do it for you.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/auto-round && curl -fsSL https://workflowstacks.com/api/skills/auto-round/claude-skill -o ~/.claude/skills/auto-round/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

✅ Light setup

Installs with a command or two; your AI agent can do it for you.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/auto-round && curl -fsSL https://workflowstacks.com/api/skills/auto-round/claude-skill -o ~/.claude/skills/auto-round/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

Quick Actions

Details

Creator

intel

Language

Python