ai-agent

paddler: Scale LLMs

Name: paddler: Scale LLMs
Author: intentee

Founders get a simple load balancer for self-hosting large language models with paddler, used by the community with 1.6k+ GitHub stars

intermediate⏱ 1-2 hours💵 Free (self-hosted)

1,639 stars91 forksRustQuality 9/10Updated 7/19/2026100% free · open source

What it is

You can self-host Large Language Models (LLMs) with paddler to create highly customized applications.

What you can make with it

Automations like an e-commerce bot that answers customer queries with detailed product information from Amazon.

How it helps

paddler's load balancing and serving platform simplifies running LLMs at scale without having to manage complex deployments.

Real use case example

"A solo developer is creating a chatbot for their startup's customer support. They use paddler to host their LLM, set up load balancing, and serve queries from their users. They configure paddler to pull product data from Amazon to answer customers' product-related questions. With paddler, their chatbot can now provide more accurate and helpful responses."

If you're new

Beginners can start using paddler to build and test simple LLM-powered applications without in-depth knowledge of AI or infrastructure management.

If you're senior

Senior engineers and developers will appreciate paddler's scalability, flexibility, and ease of integration with existing systems.

Common confusion cleared up

paddler is a load balancing platform, not a standalone LLM deployment solution like some other tools.

Best inside these AI tools

Self-hosted

Pairs with

Amazon product APIStripe webhookNotion databaseGemini

Why we list it on WorkflowStacks: We offer paddler as a completely free and open-source solution to save costs compared to paid alternatives.

What it does

Paddler is an open-source load balancer and serving platform for self-hosting large language models (LLMs) and vector databases (VLMs) at scale, allowing founders to deploy and manage AI models efficiently.

Install / run

cargo build --release

When to use it

•When you need to deploy and manage multiple LLMs or VLMs in a self-hosted environment
•When you require a simple and lightweight load balancer for your AI models
•When you want to use a ggml ecosystem-based solution for deploying and serving AI models

Quick start

1Clone the paddler repository from GitHub using the command 'git clone https://github.com/intentee/paddler.git'
2Build the paddler binary using the command 'cargo build --release'
3Create a configuration file (e.g., 'config.toml') to define your LLM or VLM settings
4Run paddler using the command './target/release/paddler --config config.toml'
5Use the paddler API to interact with your deployed models, such as sending inference requests

Ready-to-paste prompt

curl -X POST 'http://localhost:8080/infer' -H 'Content-Type: application/json' -d '{"model": "my_llm", "prompt": "Write a story about a character who learns a new skill"}'

Heads up: Make sure you have Rust and Cargo installed on your system, as paddler is built using Rust and requires these tools to build and run

Saves to your device