local-ai

Run LLMs with llama.cpp

Name: Run LLMs with llama.cpp
Author: ggml-org

Get fast LLM inference with llama.cpp, for founders using C++.

intermediate⏱ 1-2 hours💵 Free (self-hosted)

127,573 stars22,928 forksC++Guide quality 9/10Updated 9/9/2026100% free · open source

What it is

Run big language models directly from C++ code.

Real use case example

"A founder building a high-stakes AI-powered trading platform uses llama.cpp to integrate their proprietary natural language processing models, allowing their platform to quickly and accurately analyze investor intent from email correspondence."

Best insideClaude DesktopClaude CodeGemini Code Assist

When to use it

•When you need to deploy a language model in a resource-constrained environment
•When you want to integrate a language model into a C/C++ application
•When you need to achieve high-performance inference for large language models

Ready-to-paste prompt

cout << LlamaModel::generate("Tell me a story about a character who", 100) << endl;

Heads up: Ensure that you have the necessary dependencies installed, including a C++ compiler and the CMake build system, and that your system meets the minimum requirements for the pre-trained models, including at least 4GB of RAM

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🛠️ Technical setup

Expect 20–40 minutes in a terminal — or let your AI agent drive it.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/llama-cpp && curl -fsSL https://workflowstacks.com/api/skills/llama-cpp/claude-skill -o ~/.claude/skills/llama-cpp/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How Run LLMs with llama.cpp works

Codeflow

Free to inspect

Run LLMs with llama.cpp is a very large C++ project (~834k lines across 2095 code files). It is a full software project: use it through its install path rather than reading it end to end. Last commit this month, MIT license, has a test suite.

Size

Very large codebase

~834k lines · 2095 code files · days to read — use, don't read

Setup

Developer setup

A real software project. Use it via its install path; don't expect to read it all.

Runs on

Python

Needs API keys (.env)

C++ 56%C 16%Python 7%Cuda 5%TypeScript 4%

What happens, step by step

Run AI models locally

You give

User query

1Install Tool

Download and install llama.cpp

README.md

2Run Model

Use cli to run a model

3Launch Server

Start OpenAI-compatible API server

4Use API

Send requests to the server

You get

AI response

Where to start reading

1
README.md
Start here — what it does and how to install it
2
AGENTS.md
The instructions the AI actually follows
3
pyproject.toml
Dependencies and the commands it exposes
4
src/llama-adapter.cpp
Inside src/ — the main logic begins here
5
examples/CMakeLists.txt
A worked example — copy this to get going

What's in each folder

ggml/Folder1272 files

tools/Helper scripts999 files

examples/Examples you can copy226 files

src/Core code — the actual logic210 files

models/Data models & types119 files

conversion/Folder87 files

tests/Tests — proof it works77 files

.github/CI / automation (GitHub Actions)71 files

READMEHas testsDocumentedCI checksExamples includedMIT licenseUpdated this month

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🛠️ Technical setup

Expect 20–40 minutes in a terminal — or let your AI agent drive it.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/llama-cpp && curl -fsSL https://workflowstacks.com/api/skills/llama-cpp/claude-skill -o ~/.claude/skills/llama-cpp/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

Quick Actions

Details

Creator

ggml-org

Language

C++

Related skills

More local-ai tools founders pair with this one.

local-ai★ 91,165

Stirling-PDF: Edit PDFs Anywhere

Edit PDFs on any device with Stirling-PDF, a tool for startup founders

local-ai★ 31,165

Run AI Locally with llmfit

Find compatible AI models for your hardware with llmfit, a tool.

local-ai★ 29,040

koreader: Multi-format eBook Reader

Get a versatile eBook reader for various devices.

local-ai★ 16,018

MNN: Fast On-Device AI

Get high-performance Edge AI with MNN, battle-tested by Alibaba, for startup founders

local-ai★ 10,300

Run AI Locally with runanywhere-sdks

Run AI models on-device with this production-ready toolkit, ideal for startup founders

local-ai★ 9,622

amlogic-s9xxx-armbian: Run Armbian on Devices

Get Armbian on Amlogic, Allwinner, Rockchip devices with amlogic-s9xxx-armbian, for founders needing local AI solutions