ai-agent

TensorRT-LLM: Accelerate LLM Inference on NVIDIA GPUs

Name: TensorRT-LLM: Accelerate LLM Inference on NVIDIA GPUs
Author: NVIDIA

Boost your AI product’s response speed with TensorRT-LLM’s Python API and high‑performance runtimes, ideal for founders deploying large language models. 15k+ GitHub stars.

From the repo · “TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.”

intermediate⏱ 30 minutes💵 Free + LLM API costs

14,906 stars+92★ this week2,847 forksPythonGuide quality 9/10 · reviewed Oct 5Updated 10/21/2026100% free · open source

What it is

Use Python to define and run Large Language Models.

Real use case example

"A founder uses TensorRT LLM to create a customer support chatbot. They write a Python script to define the model, train it on customer data, and deploy it on their GPU. The chatbot answers common questions, freeing up human support staff to focus on complex issues."

Best insideClaude DesktopCodex CLIContinue

When to use it

•When you need to deploy LLMs in production with high performance and low latency
•When you want to optimize your LLM inference workflow on NVIDIA GPUs
•When you need a Python-friendly interface to define and run LLMs

Ready-to-paste prompt

python -c 'from tensorrt_llm import LLM; model = LLM("bert-base-uncased", "https://example.com/model.pth"); print(model.infer("What is the meaning of life?"))'

Heads up: You need to have an NVIDIA GPU and the CUDA Toolkit installed to use TensorRT-LLM, as well as a compatible version of Python (currently Python 3.8 or 3.9)

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

✅ Light setup

Installs with a command or two; your AI agent can do it for you.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/tensorrt-llm && curl -fsSL https://workflowstacks.com/api/skills/tensorrt-llm/claude-skill -o ~/.claude/skills/tensorrt-llm/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How TensorRT-LLM: Accelerate LLM Inference on NVIDIA GPUs works

Codeflow

Free to inspect

TensorRT-LLM: Accelerate LLM Inference on NVIDIA GPUs is a very large Python project (~1.3M lines across 2933 code files, plus 1471 test files). Setup is light: installs like a normal app. Reading the code is optional. Last commit this month, Other license, has a test suite.

Size

Very large codebase

~1.3M lines · 2933 code files · days to read — use, don't read

Setup

One-command install

Installs like a normal app. Reading the code is optional.

Runs on

Python

No API keys detected

Python 63%C++ 29%Cuda 7%Groovy 1%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
AGENTS.md
The instructions the AI actually follows
3
pyproject.toml
Dependencies and the commands it exposes
4
tensorrt_llm/__init__.py
Inside tensorrt_llm/ — the main logic begins here
5
examples/README.md
A worked example — copy this to get going

What's in each folder

tensorrt_llm/Core code — the actual logic1446 files

examples/Examples you can copy683 files

docs/Documentation415 files

cpp/Folder10742 files

agent-flow/Folder133 files

scripts/Helper scripts127 files

jenkins/Folder90 files

triton_kernels/Folder57 files

READMEHas testsDocumentedCI checksExamples includedOther licenseUpdated this month