sonar: Fast LLM Inference
Get fast large-scale LLM inference with sonar, a tool for founders and machine learning teams, backed by 1.8k+ GitHub stars.
1,798 stars205 forksC++Quality 9/10Updated 7/16/2026100% free ยท open source Sonar is a large-scale LLM inference engine that allows founders and machine learning teams to run fast and efficient language model inference
git clone https://github.com/dphnAI/sonar.git
- โขWhen you need to deploy a language model for production use cases
- โขWhen you want to speed up your language model inference pipeline
- โขWhen you require a scalable solution for large-scale language model deployments
- 1Build the sonar engine with the command 'mkdir build && cd build && cmake .. && make -j'
- 2Prepare your language model by converting it to the sonar format using the 'convert_model.py' script
- 3Configure the sonar engine by editing the 'config.json' file to specify your model and inference settings
- 4Run the sonar engine with the command './sonar -c config.json -m your_model.sonar'
- 5Test the sonar engine with a sample input using the 'test.py' script
Ready-to-paste prompt ./sonar -c config.json -m your_model.sonar -i 'What is the meaning of life?'
Heads up: Sonar requires a compatible C++ compiler (e.g. GCC 10 or later) and a CPU with AVX-512 support for optimal performance
Saves to your device
Topics
api-rest
cuda
inference-engine
inferentia
intel
lora
machine-learning
rocm
speculative-decoding
tpu
What's inside โ free to inspect
No purchase needed
Read the entire source before you build โ unlike paid marketplaces that hide it behind a buy button.