Deploy LLM inference services on Kubernetes with rbg, a workload for dev teams, with 229 GitHub stars
intermediateโฑ 30 minutes๐ต Free + LLM API costs
254 stars66 forksGoQuality 9/10Updated 7/6/2026100% free ยท open source
What it is
Helps you create and deploy AI-powered services on a Kubernetes cluster.
What you can make with it
Automations like deploying AI models to process customer inquiries on a website, and responding with answers to those inquiries.
How it helps
It streamlines deploying and managing Large Language Model (LLM) services on Kubernetes, making it easier to build and scale complex AI-powered applications.
Real use case example
"A founder of a startup, Sarah, wants to quickly build a FAQ page where customers can ask questions and receive answers from her product knowledge base. She uses rbg to deploy a Kubernetes cluster, which makes it easy to deploy her AI model and scale it as needed."
If you're new
When you're new to working with AI models and Kubernetes, but want to explore building simple AI-powered applications.
If you're senior
When you need to deploy LLM services on a Kubernetes cluster as part of a larger cloud-native application, and want to leverage the performance and scalability of Kubernetes.
Common confusion cleared up
It's not a separate service that provides AI answers, but rather a tool to help you deploy and manage AI services you've built on your own.
Pairs with
Stripe webhookNotion database
Why we list it on WorkflowStacks: Because it provides a free and open-source way to deploy LLM services on Kubernetes, which is often the infrastructure of choice for cloud-native applications.
What it does
Rbg allows you to deploy Large Language Model (LLM) inference services on a Kubernetes cluster, making it easier to manage and scale your AI workloads.
Install / run
git clone https://github.com/sgl-project/rbg.git
When to use it
โขWhen you need to deploy LLMs in a production environment with high scalability and reliability requirements
โขWhen your development team requires a simplified workflow for deploying and managing AI services on Kubernetes
โขWhen you want to leverage the benefits of Kubernetes for your AI workloads, such as automated rolling updates and self-healing
Quick start
1Run 'make deploy' to deploy the rbg workload and its dependencies to your Kubernetes cluster
2Create a new LLM inference service by defining a YAML configuration file, as seen in the examples provided in the rbg repository
3Apply the YAML configuration file using 'kubectl apply -f your_config.yaml' to create the LLM inference service
4Verify the deployment status using 'kubectl get deployments' and 'kubectl get pods'
5Test the LLM inference service by sending requests to the exposed API endpoint
Heads up: Ensure you have a Kubernetes cluster with the necessary dependencies installed, such as the Kubernetes API server and the rbg custom resource definitions, before attempting to deploy the rbg workload
Saves to your device
Topics
k8s
llm
pd-disagg
sglang
What's inside โ free to inspect
No purchase needed
Read the entire source before you build โ unlike paid marketplaces that hide it behind a buy button.