ai-agent

evalscope: Optimize LLMs

Evaluate large models efficiently with evalscope, a customizable framework.
intermediate30 minutes💵 Free + LLM API costs
3,331 stars471 forksPythonGuide quality 9/10Updated 8/30/2026100% free · open source
What it is

Evaluates large language models efficiently to check their performance.

Real use case example

"A founder building a product with a large language model wants to compare its performance with another model. They use evalscope to create a custom evaluation framework and run it on both models, getting results in under an hour."

Best insideCursorClaude DesktopCodex CLIAny AI Client
When to use it
  • When you need to compare the performance of different large models on your dataset
  • When you want to optimize your model's performance on a specific task or metric
  • When you need to evaluate the effectiveness of your model in a production environment
Ready-to-paste prompt
Run an evaluation task using `evalscope run --config-path=configs/llm_benchmark.yml --model-name=mymodel --dataset=mydataset`
Heads up: Make sure you have the necessary dependencies installed, including PyTorch and transformers, and that your model and dataset are compatible with the evalscope framework
Saves to your device
Use with Claude
New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

✅ Light setup

Installs with a command or two; your AI agent can do it for you.

Try it instantly — no install
Claude Code
mkdir -p ~/.claude/skills/evalscope && curl -fsSL https://workflowstacks.com/api/skills/evalscope/claude-skill -o ~/.claude/skills/evalscope/SKILL.md
Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)
claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

What's inside — free to inspect

No purchase needed

Read the entire source before you build — unlike paid marketplaces that hide it behind a buy button.

13
top-level files
9
folders
104.5M
source size
Apache-2.0
license
Key files
.pre-commit-config.yaml
AGENTS.md
README_zh.md
README.md
File tree
.github/
custom_eval/
docs/
evalscope/
examples/
requirements/
scripts/
skills/
tests/
.gitignore
.pre-commit-config.yaml
AGENTS.md
CONTRIBUTING.md
DESIGN.md
LICENSE
Makefile
MANIFEST.in
pyproject.toml
README_zh.md
README.md
setup.cfg
setup.py
Quick Actions
Details
Creator
modelscope
Language
Python
Category
ai-agent
Published
12/7/2023

Are you the creator of this tool? Claim your listing → and earn 85% of every sale.