ai-agent

LMDeploy: Streamline LLM Deployment

Name: LMDeploy: Streamline LLM Deployment
Author: InternLM

Get efficient LLM compression and serving with LMDeploy, a toolkit for startup founders, backed by 8.0k+ GitHub stars.

7,979 stars716 forksPythonHealth Score 9/10Updated 7/27/2026100% free · open source

What it does

LMDeploy is a toolkit for compressing, deploying, and serving large language models (LLMs) to reduce their size and improve performance

Install / run

pip install lmdeploy

When to use it

•When you need to deploy an LLM in a resource-constrained environment
•When you want to reduce the latency of your LLM-based application
•When you need to serve multiple LLMs and want to optimize storage and memory usage

Quick start

1Clone the LMDeploy repository using `git clone https://github.com/InternLM/lmdeploy.git`
2Navigate to the repository directory using `cd lmdeploy`
3Install the required dependencies using `pip install -r requirements.txt`
4Use the `lmdeploy compress` command to compress a pre-trained LLM, for example `lmdeploy compress --model-name bert-base-uncased --output-path compressed_model`
5Deploy the compressed model using `lmdeploy serve --model-path compressed_model --port 8000`

Ready-to-paste prompt

lmdeploy compress --model-name bert-base-uncased --output-path compressed_model --compression-ratio 0.5

Heads up: Make sure you have the necessary dependencies installed, including PyTorch and Transformers, and that your Python version is compatible with the requirements specified in the README, currently Python 3.8 or later

Saves to your device

Topics

codellama

cuda-kernels

deepspeed

fastertransformer

internlm

llama

llama2

llama3

llm

llm-inference

turbomind

How LMDeploy: Streamline LLM Deployment works

Codeflow

Free to inspect

LMDeploy: Streamline LLM Deployment is a very large Python project (~250k lines across 1154 code files, plus 172 test files). Setup is light: installs like a normal app. Reading the code is optional. Last commit this month, Apache-2.0 license, has a test suite.

Size

Very large codebase

~250k lines · 1154 code files · days to read — use, don't read

Setup

One-command install

Installs like a normal app. Reading the code is optional.

Runs on

Python · Docker

No API keys detected