workflowstacks
local-ai

jitllm

Java Inference Tornado toolkit: High performance LLM inference & serving for the JVM
289 stars40 forksJavaUpdated 10/20/2026100% free · open source
What it does

Runs LLM inference directly on the JVM with GPU acceleration via TornadoVM, letting Java apps serve language models locally.

When to use it
  • •You need low‑latency LLM responses inside a Java microservice without calling an external API.
  • •Your team is already building on the JVM and wants to keep the entire stack in Java.
  • •You want to experiment with on‑prem GPU‑accelerated inference without Docker‑orchestrated services.
Ready-to-paste prompt
curl -X POST http://localhost:8080/generate -H "Content-Type: application/json" -d '{"prompt":"Write a 150‑word intro for a SaaS landing page about AI‑driven customer support.","temperature":0.7,"max_tokens":120}'
Heads up: JITLLM requires a JDK that supports TornadoVM (Java 11 or newer) **and** the TornadoVM runtime installed on the host; without the TornadoVM native libraries the server will fall back to CPU and be dramatically slower.
Saves to your device
Use with Claude
New

Skip the builder — one click puts this in Claude, Cursor, Antigravity and more.

🛠️ Technical setup

Expect 20–40 minutes in a terminal — or let your AI agent drive it.

Try it instantly — no install
Claude Code
mkdir -p ~/.claude/skills/jitllm && curl -fsSL https://workflowstacks.com/api/skills/jitllm/claude-skill -o ~/.claude/skills/jitllm/SKILL.md
Open in another AI app

Opens the app with this repo with the prompt ready to go — no copy-paste needed.

Connect the whole catalog (MCP)
claude mcp add --transport http workflowstacks https://workflowstacks.com/api/mcp

Adds a WorkflowStacks connector to Claude Code: search and load any skill here by chatting.

How jitllm works
Codeflow
Free to inspect

Jitllm is a very large Java project (~176k lines across 693 code files, plus 409 test files). It is a full software project: use it through its install path rather than reading it end to end. Last commit this month, MIT license, has a test suite.

Size
Very large codebase
~176k lines · 693 code files · days to read — use, don't read
Setup
Developer setup
A real software project. Use it via its install path; don't expect to read it all.
Runs on
Java/JVM
No API keys detected
Java 98%Python 1%Shell 1%
Where to start reading
  1. 1
    README.md
    Start here — what it does and how to install it
  2. 2
    src/test/resources/chat-templates/render_expected.py
    Inside src/ — the main logic begins here
What's in each folder
src/Core code — the actual logic1157 files
docs/Documentation21 files
scripts/Helper scripts26 files
.claude/Editor / agent settings23 files
.github/CI / automation (GitHub Actions)14 files
.mvn/Tool / agent settings2 files
READMEHas testsDocumentedCI checksMIT licenseUpdated this month
Quick Actions
Details
Creator
beehive-lab
Language
Java
Category
local-ai
Published
5/10/2025

Are you the creator of this tool? and earn 85% of every sale.

Show it off in your README:

Featured on WorkflowStacks
[![Featured on WorkflowStacks](https://workflowstacks.com/api/badge/jitllm.svg)](https://workflowstacks.com/skills/jitllm?utm_source=github&utm_medium=badge)
🔥 Hot this week, in your inbox
Skills like this, every Monday.

The five fastest-growing open-source AI skills, ranked by GitHub star growth. One email a week, unsubscribe anytime.

Related skills

More local-ai tools founders pair with this one.

local-ai★ 131,925
Run Powerful LLMs Locally with llama.cpp – Fast C++ Inference
Empower your product with on‑device LLM inference using llama.cpp’s efficient C++ engine. Ideal for founders who need privacy, speed, and no cloud costs. 130k+ GitHub stars.
local-ai★ 94,670
Stirling-PDF: Edit PDFs Anywhere
Edit PDFs on any device with Stirling-PDF, a tool for startup founders, backed by 80k+ GitHub stars
local-ai★ 38,207
Run AI Locally with llmfit
Find compatible AI models for your hardware with llmfit, a tool for founders, with 29k+ GitHub stars.
local-ai★ 282
ggrun: Fast Multi‑GPU LLM Server for Startup AI Apps
Deploy GGUF models on llama.cpp with auto‑tuned flags, tensor‑split, and MoE placement. Ideal for founders needing a self‑hosted, OpenAI‑compatible inference server on multi‑GPU rigs.
local-ai★ 1,236
crit: Streamline Your AI Agent Feedback Loop
crit lets founders quickly iterate with AI agents via a simple CLI, turning feedback into actionable code improvements. Ideal for Go‑based dev teams. 1.2k+ GitHub stars
local-ai★ 1,605
Run LLMs on Any Device Fast with nobodywho
nobodywho lets founders run on‑device LLM inference efficiently on any hardware—no cloud fees, full control. Backed by 1.6k+ GitHub stars.