ai-agent

openinfer: Rust‑powered GPU LLM inference without PyTorch

Name: openinfer: Rust‑powered GPU LLM inference without PyTorch
Author: openinfer-project

Run Qwen3‑Kimi‑K2 models fast on CUDA using pure Rust. Ideal for founders who need low‑latency, OpenAI‑compatible LLM serving on GPUs.

From the repo · “Pure Rust + CUDA LLM inference engine — no PyTorch, OpenAI-compatible, serves Qwen3 to Kimi-K2”

724 stars+4★ this week120 forksRustGuide quality 9/10 · reviewed Oct 10Updated 10/20/2026100% free · open source

What it does

OpenInfer provides fast and compatible large language model (LLM) inference for startup founders, serving models from Qwen3 to Kimi-K2 using a Rust-based engine

When to use it

•When you need to deploy LLM models without relying on PyTorch
•When you require OpenAI-compatible model serving for your startup
•When you prefer a Rust-based solution for performance and compatibility

Ready-to-paste prompt

curl -X POST -H 'Content-Type: application/json' -d '{"prompt": "Write a short story about a character who discovers a hidden world."}' http://localhost:8000/infer

Heads up: Make sure you have Rust and CUDA installed on your system, as OpenInfer relies on these dependencies to function correctly

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🛠️ Technical setup

Expect 20–40 minutes in a terminal — or let your AI agent drive it.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/openinfer && curl -fsSL https://workflowstacks.com/api/skills/openinfer/claude-skill -o ~/.claude/skills/openinfer/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How openinfer: Rust‑powered GPU LLM inference without PyTorch works

Codeflow

Free to inspect

Openinfer: Rust‑powered GPU LLM inference without PyTorch is a very large Rust project (~246k lines across 588 code files, plus 111 test files). It is a full software project: use it through its install path rather than reading it end to end. Last commit this month, Apache-2.0 license, has a test suite.

Size

Very large codebase

~246k lines · 588 code files · days to read — use, don't read

Setup

Developer setup

A real software project. Use it via its install path; don't expect to read it all.

Runs on

Rust

No API keys detected

Rust 78%Python 11%Cuda 9%C++ 1%Shell 1%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
AGENTS.md
The instructions the AI actually follows
3
Cargo.toml
Dependencies and the commands it exposes

What's in each folder

docs/Documentation161 files

pegainfer-kernels/Folder249 files

pegainfer-glm52/Folder90 files

pegainfer-qwen3/Folder79 files

kvbm/Folder64 files

pegainfer-qwen35/Folder46 files

pegainfer-k3/Folder40 files

pegainfer-kimi-k2/Folder36 files

READMEHas testsDocumentedCI checksApache-2.0 licenseUpdated this month

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🛠️ Technical setup

Expect 20–40 minutes in a terminal — or let your AI agent drive it.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/openinfer && curl -fsSL https://workflowstacks.com/api/skills/openinfer/claude-skill -o ~/.claude/skills/openinfer/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

Quick Actions

Details

Creator

openinfer-project

Language

Rust

Related skills

More ai-agent tools founders pair with this one.

ai-agent★ 303,229

superpowers

An agentic skills framework & software development methodology that works.

ai-agent★ 277,148

Track GitHub Stars with 996.ICU

ai-agent★ 253,013

Linux – 251k+ GitHub stars for building custom kernel solutions

Get the complete Linux kernel source in C, perfect for founders needing deep system control or tailored OS components.

ai-agent★ 1,275

GPTQModel: Faster AI

Compress AI models with GPTQModel, a toolkit for startup founders, with 1.2k+ GitHub stars.

ai-agent★ 12,509

Github-Ranking

:star:Github Ranking:star: Github stars and forks ranking list. Github Top100 stars list of different languages. Automatically update daily. | Github仓库排名，每日自动更新

ai-agent★ 6,860

Mooncake: Deploy Kimi LLM at Scale with High‑Performance C++

Mooncake lets startup founders serve Moonshot AI’s Kimi LLM quickly and efficiently, leveraging RDMA, KV‑cache, and GPU acceleration. Ideal for AI‑heavy products.