claude-skill

VLMEvalKit

Name: VLMEvalKit
Author: open-compass

Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks

4,382 stars762 forksPythonUpdated 9/10/2026100% free · open source

What it does

Runs standardized benchmarks to measure the accuracy of vision‑language models (LMMs) across dozens of tasks.

When to use it

•You need to compare several LMM candidates before picking one for a product feature.
•You want to track how a new model version improves on existing vision‑language benchmarks.
•You are building a pitch deck and need concrete numbers (e.g., VQA, image captioning scores) to back claims.

Ready-to-paste prompt

python run.py --model_type llava --model_path ~/models/llava-1.5-7b/ --bench_name mmbench --split val --output_dir ./myrun

Heads up: VLMEvalKit requires a matching PyTorch + CUDA build; if you install the default `torch` wheel on a CPU‑only machine the benchmark will crash. Install the correct CUDA wheel (e.g., `pip install torch==2.2.0+cu121 -f https://download.pytorch.org/whl/torch_stable.html`) before running any evaluation.

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

⚡ Runs out of the box

A prompt/skill package — nothing to install beyond adding it to your AI tool.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/251c180d-d63b-4c7a-b8e3-0f6f179cc766 && curl -fsSL https://workflowstacks.com/api/skills/251c180d-d63b-4c7a-b8e3-0f6f179cc766/claude-skill -o ~/.claude/skills/251c180d-d63b-4c7a-b8e3-0f6f179cc766/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How VLMEvalKit works

Codeflow

Free to inspect

VLMEvalKit is a very large Python project (~197k lines across 746 code files, plus 3 test files). It is a full software project: use it through its install path rather than reading it end to end. Last commit this month, Apache-2.0 license, has a test suite.

Size

Very large codebase

~197k lines · 746 code files · days to read — use, don't read

Setup

Developer setup

A real software project. Use it via its install path; don't expect to read it all.

Runs on

Python

No API keys detected

Python 100%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
run.py
Where the program starts running
3
requirements.txt
Dependencies and the commands it exposes

What's in each folder

vlmeval/api/Backend / API45 files

docs/Documentation36 files

vlmeval/dataset/Data files516 files

vlmeval/vlm/Folder195 files

scripts/Helper scripts17 files

vlmeval/smp/Folder6 files

vlmeval/utils/Helper scripts4 files

requirements/Folder1 files

READMEHas testsDocumentedCI checksApache-2.0 licenseUpdated this month