ai-agent

Run Large LLMs with RTX 6000 Pro

Name: Run Large LLMs with RTX 6000 Pro
Author: local-inference-lab

Run large LLMs on PCIe GPUs with RTX 6000 Pro, a tool for founders working with AI agents, with 657 GitHub stars.

657 stars42 forksPythonQuality 9/10Updated 7/19/2026100% free · open source

What it does

RTX 6000 Pro allows founders to run large language models like Qwen3.5-397B, Kimi-K2.5, and GLM-5 on PCIe GPUs without NVLink, enabling local inference for AI agents

Install / run

git clone https://github.com/local-inference-lab/rtx6kpro.git

When to use it

•You need to deploy large language models on local hardware for data privacy or latency reasons
•Your AI workflow requires running models like Qwen3.5-397B or Kimi-K2.5 on commodity GPUs
•You want to test and fine-tune large language models without relying on cloud services

Quick start

1Clone the repository and navigate to the project directory: cd rtx6kpro
2Update the config file to point to your model and GPU: nano config.py
3Run the example script for Qwen3.5-397B: python run_qwen.py
4Monitor performance and adjust settings as needed using the provided benchmarks and tools
5Refer to the RTX 6000 Pro Wiki for troubleshooting and optimization tips

Ready-to-paste prompt

python run_qwen.py --model qwen3.5-397b --gpu 0 --batch-size 32

Heads up: Ensure you have a compatible NVIDIA driver installed (version 470 or later) and that your system meets the minimum hardware requirements outlined in the RTX 6000 Pro Wiki

Saves to your device

What's inside — free to inspect

No purchase needed

Read the entire source before you build — unlike paid marketplaces that hide it behind a buy button.

top-level files

folders

151.2M

repo size

—

license

Key files

INDEX.md