ai-agent

Lucebox: Fast Local AI

Name: Lucebox: Fast Local AI
Author: Luce-Org

Get fast LLM inference on consumer hardware with lucebox, ideal for local AI startup founders, backed by 2.7k+ GitHub stars

2,697 stars253 forksC++Health Score 9/10Updated 7/30/2026100% free · open source

What it does

Lucebox is a fast LLM speculative inference server that allows you to run large language models on consumer-grade hardware for AI-related tasks.

Install / run

git clone https://github.com/Luce-Org/lucebox.git && cd lucebox

When to use it

•When you need to deploy AI models on consumer hardware with limited resources
•When you want to test and prototype AI applications quickly without relying on cloud services
•When you require a self-hosted solution for AI inference with a small footprint

Quick start

1Build the project using the command 'cmake . && make' in the terminal
2Create a configuration file 'config.json' to specify the model and settings
3Run the server with './lucebox --config config.json' to start the inference service
4Use a tool like 'curl' to test the API endpoints, for example 'curl -X POST -H 'Content-Type: application/json' -d '{"prompt": "Hello World"}' http://localhost:8080/infer'
5Monitor the server's performance and adjust the configuration as needed

Ready-to-paste prompt

curl -X POST -H 'Content-Type: application/json' -d '{"prompt": "Generate a short story about a character who discovers a hidden world"}' http://localhost:8080/infer

Heads up: Ensure you have a compatible C++ compiler (such as GCC 10 or higher) and CMake installed on your system before attempting to build and run Lucebox

Saves to your device

Topics

cuda

cuda-kernels

dflash

kernel

llama-cpp

local-ai

luce

megakernel

pflash

poolside

qwen

rocm

rtx3090

spark

speculative-decoding

speculative-prefill

strix-halo

How Lucebox: Fast Local AI works

Codeflow

Free to inspect

Lucebox: Fast Local AI is a very large C++ project (~360k lines across 827 code files, plus 120 test files). It is a full software project: use it through its install path rather than reading it end to end. Last commit this month, Apache-2.0 license, has a test suite.

Size

Very large codebase

~360k lines · 827 code files · days to read — use, don't read

Setup

Developer setup

A real software project. Use it via its install path; don't expect to read it all.

Runs on

Python · Docker

No API keys detected

C++ 60%C 14%Cuda 12%Python 7%Metal 2%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
pyproject.toml
Dependencies and the commands it exposes
3
server/src/cuda_cross_device_copy.cpp
Inside server/src/ — the main logic begins here

What's in each folder

server/src/Core code — the actual logic304 files

server/docs/Documentation14 files

docs/Documentation6 files

server/examples/Examples you can copy1 files

server/deps/Folder1098 files

optimizations/Folder60 files

harness/Folder55 files

server/scripts/Helper scripts42 files