paddler: Scale LLMs
From the repo ยท โOpen-source LLM/VLM load balancer and serving platform for self-hosting LLMs (and VLMs) at scale ๐๐ฆ Alternative to projects like llm-d, Docker Model Runner, etc but with less moving parts and simple deployments built around ggml ecosystem. Runs on CPU and GPU.โ
You can self-host Large Language Models (LLMs) with paddler to create highly customized applications.
"A solo developer is creating a chatbot for their startup's customer support. They use paddler to host their LLM, set up load balancing, and serve queries from their users. They configure paddler to pull product data from Amazon to answer customers' product-related questions. With paddler, their chatbot can now provide more accurate and helpful responses."
- โขWhen you need to deploy and manage multiple LLMs or VLMs in a self-hosted environment
- โขWhen you require a simple and lightweight load balancer for your AI models
- โขWhen you want to use a ggml ecosystem-based solution for deploying and serving AI models
curl -X POST 'http://localhost:8080/infer' -H 'Content-Type: application/json' -d '{"model": "my_llm", "prompt": "Write a story about a character who learns a new skill"}'Skip the builder โ one click puts this in Claude, Cursor, Antigravity and more.
Installs with a command or two; your AI agent can do it for you.
mkdir -p ~/.claude/skills/paddler && curl -fsSL https://workflowstacks.com/api/skills/paddler/claude-skill -o ~/.claude/skills/paddler/SKILL.mdOpens the app with this repo with the prompt ready to go โ no copy-paste needed.
Paddler: Scale LLMs is a large Rust project (~46k lines across 902 code files, plus 487 test files). Setup is light: runs with Docker. Last commit this month, Apache-2.0 license, has a test suite.
- 1README.mdStart here โ what it does and how to install it
- 2CLAUDE.mdThe instructions the AI actually follows
- 3Cargo.tomlDependencies and the commands it exposes
- 4example/README.mdA worked example โ copy this to get going
Skip the builder โ one click puts this in Claude, Cursor, Antigravity and more.
Installs with a command or two; your AI agent can do it for you.
mkdir -p ~/.claude/skills/paddler && curl -fsSL https://workflowstacks.com/api/skills/paddler/claude-skill -o ~/.claude/skills/paddler/SKILL.mdOpens the app with this repo with the prompt ready to go โ no copy-paste needed.
Are you the creator of this tool? and earn 85% of every sale.
Show it off in your README:
[](https://workflowstacks.com/skills/paddler?utm_source=github&utm_medium=badge)The five fastest-growing open-source AI skills, ranked by GitHub star growth. One email a week, unsubscribe anytime.
Do this with a working workflow
Pages we wrote and tested that put a skill like this to work โ importable n8n templates, outcome guides and Claude Desktop configs.
AI Support Agent (Chat)
A chat agent that answers customer questions from YOUR business info โ with conversation memory. It never invents policies; anything it doesn't know, it hands to a human.
Outcome guideAutomate replies to your customer reviews
The workflow reads your reviews and drafts a personalized reply for each: warm thanks for happy customers, careful and de-escalating for unhappy ones. Nothing posts automatically โ you approve every draft.
MCP configGitHub MCP config
Search repos, read code, open issues and PRs from Claude.
Related skills
More ai-agent tools founders pair with this one.