ai-agent

TensorRT-LLM: Fast LLM Inference

Name: TensorRT-LLM: Fast LLM Inference
Author: NVIDIA

Get efficient Large Language Model inference with TensorRT-LLM, a Python API.

intermediate⏱ 30 minutes💵 Free + LLM API costs

14,546 stars2,721 forksPythonGuide quality 8/10Updated 9/4/2026100% free · open source

What it is

Use Python to define and run Large Language Models.

Real use case example

"A founder uses TensorRT LLM to create a customer support chatbot. They write a Python script to define the model, train it on customer data, and deploy it on their GPU. The chatbot answers common questions, freeing up human support staff to focus on complex issues."

Best insideClaude DesktopCodex CLIContinue

When to use it

•When you need to deploy LLMs in production with high performance and low latency
•When you want to optimize your LLM inference workflow on NVIDIA GPUs
•When you need a Python-friendly interface to define and run LLMs

Ready-to-paste prompt

python -c 'from tensorrt_llm import LLM; model = LLM("bert-base-uncased", "https://example.com/model.pth"); print(model.infer("What is the meaning of life?"))'

Heads up: You need to have an NVIDIA GPU and the CUDA Toolkit installed to use TensorRT-LLM, as well as a compatible version of Python (currently Python 3.8 or 3.9)

Saves to your device

Use with Claude

New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

✅ Light setup

Installs with a command or two; your AI agent can do it for you.

Try it instantly — no install

Claude Code

mkdir -p ~/.claude/skills/tensorrt-llm && curl -fsSL https://workflowstacks.com/api/skills/tensorrt-llm/claude-skill -o ~/.claude/skills/tensorrt-llm/SKILL.md

Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)

claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How TensorRT-LLM: Fast LLM Inference works

Codeflow

Free to inspect

TensorRT-LLM: Fast LLM Inference is a very large Python project (~1.3M lines across 2933 code files, plus 1471 test files). Setup is light: installs like a normal app. Reading the code is optional. Last commit this month, Other license, has a test suite.

Size

Very large codebase

~1.3M lines · 2933 code files · days to read — use, don't read

Setup

One-command install

Installs like a normal app. Reading the code is optional.

Runs on

Python

No API keys detected

Python 63%C++ 29%Cuda 7%Groovy 1%

Where to start reading

1
README.md
Start here — what it does and how to install it
2
AGENTS.md
The instructions the AI actually follows
3
pyproject.toml
Dependencies and the commands it exposes
4
tensorrt_llm/__init__.py
Inside tensorrt_llm/ — the main logic begins here
5
examples/README.md
A worked example — copy this to get going

What's in each folder

tensorrt_llm/Core code — the actual logic1446 files

examples/Examples you can copy683 files

docs/Documentation415 files

cpp/Folder10742 files

agent-flow/Folder133 files

scripts/Helper scripts127 files

jenkins/Folder90 files

triton_kernels/Folder57 files

READMEHas testsDocumentedCI checksExamples includedOther licenseUpdated this month