Colibrì - an open-source, lightweight, local inference engine with tiered storage for inference. - AiBoss

What is Colibrì?

Colibrì is an open-source, lightweight local inference engine that can run the flagship GLM-5.2 MoE model with 744B parameters on a consumer-grade computer with 25GB of RAM. The model uses a hierarchical storage architecture to keep resident parameters in memory and store massive routing expert parameters on NVMe disks and load them on demand, breaking through the traditional limitation that the model must be fully loaded into high-speed memory.

The main functions of Colibrì

Tiered storage inferenceThe expert parameters are distributed across three levels of storage: GPU memory, system memory, and NVMe hard drive, and dynamically loaded on demand, thus overcoming memory capacity limitations.
Local deployment of ultra-large-scale modelsThe flagship GLM-5.2 MoE model, supporting 744B parameters, runs on a consumer-grade PC with 25GB of memory.
int4 quantization compressionThe resident parameters, after being quantized by int4, are approximately 9.9GB of resident memory; 370GB of routing expert parameters are stored on the local hard drive.
Pure C streaming CPU inferenceA lightweight engine with zero external dependencies, which achieves dynamic scheduling of expert parameters through streaming loading, eliminating the need to load the entire model into high-speed memory.
Web Visual Control PanelBuilt-in Chat interface for real-time display of inference performance metrics such as TTFT, throughput, and queue status.
Expert Routing Brain ViewThe Brain page uses a 76-layer × 256 expert matrix to present routing heatmaps and storage hierarchies; hovering the mouse over an expert's topic preferences will display their preferences.

Colibrì's technical principles

MoE sparse activation utilizationGLM-5.2 has a total parameter count of 744B, with approximately 40B activated per token. Colibrì uses this sparsity feature to keep fixed parameters resident in memory and load expert parameters that change with the route only as needed.
Tiered storage architectureThe model is split into approximately 9.9GB of resident parameters and approximately 370GB of routing experts, which reside in memory and NVMe hard drives respectively, forming a three-level storage pool of video memory/memory/disk.
Expert Streaming21,504 routing experts dynamically stream from disk during inference based on router selection.
LRU and Learning CacheEach layer is configured with an independent LRU cache to retain hot experts. At the same time, by learning the cache to record cross-session usage history, high-frequency experts are prioritized to be fixed in memory, reducing the number of disk I/O operations.
Router-Lookahead prefetchBased on the attention state of the current layer, the required expert for the next layer is predicted. This is achieved by asynchronously prefetching the expert's information during computation in the current layer using a dedicated I/O thread. WILLNEED System calls accelerate reading.

How to use Colibrì

Environmental preparationRequires a Linux or WSL2 system, an x86-64 CPU that supports AVX2, a GCC compiler, at least 16GB of memory, and a local NVMe SSD with approximately 400GB of available space.
Get the engine source code:implement git clone https://github.com/JustVugg/colibri Clone repository, enter colibri/c Run after directory ./setup.sh Complete one-click compilation to generate a pure C executable file with zero dependencies.
Download pre-conversion modelFrom the Hugging Face repository jlnsrk/GLM-5.2-colibri-int4 Download the int4 quantized model container and save it to your local NVMe hard drive.
Start local dialogue: Execute after setting the model path COLI_MODEL=/nvme/glm52_i4 ./coli chatThe initial loading time is approximately 30 seconds, and the resident memory size is approximately 9.9GB. It can interact with GLM-5.2 on the terminal.
Start API service:implement COLI_MODEL=/nvme/glm52_i4 ./coli serve --host 0.0.0.0 --port 8000 Start the OpenAI-compatible HTTP service, which can be directly accessed by any client that supports the OpenAI protocol.
Windows native compilationWindows 11 users can install GCC via MinGW-w64 or MSYS2. c/ Directory Execution make glm.exe Compile the native executable file.

Colibrì's core advantages

Consumer-grade hardware running flagship modelsBreaking away from the traditional limitation that GLM-5.2 with 744B parameters must be deployed on a server cluster, it can run locally with only 25GB of memory and an NVMe hard drive.
Pure C, zero dependencies, extremely lightweightThe entire engine is written in pure C language, does not depend on any external libraries or runtime, has a very small compilation output, and starts up quickly.
Tiered storage intelligent schedulingIt features a unique three-level expert cache system (video memory/RAM/hard disk), combining LRU and learning cache to keep high-frequency experts resident in memory and low-frequency experts streamed from disk on demand.
Router-Lookahead prefetch accelerationIt predicts the experts needed for the next layer based on the current layer state with an accuracy of over 70%. It also performs prefetching during computation intervals using asynchronous I/O threads, significantly reducing disk read latency.
int4 quantization and handwritten kernelGlobal int4 quantization compresses the 744B model to approximately 380GB of disk space. Combined with the AVX2 handwritten integer dot product kernel, it enables efficient inference without BLAS dependency.
MLA KV-Cache Extreme CompressionKV-Cache uses only 576 values per token, which is about 1/57 of the traditional solution, significantly reducing the memory footprint of long contexts and supporting persistent recovery.

Colibrì's project address

GitHub repositoryhttps://github.com/JustVugg/colibri

Comparison of Colibrì's similar competing products

Dimension	Colibrì	KTransformers
Core positioning	A massive MoE hierarchical storage inference engine with zero dependencies and pure C architecture.	A CPU-GPU heterogeneous MoE inference and fine-tuning framework developed by Tsinghua University.
Maximum model support	744B GLM-5.2 (can run with 25GB RAM + NVMe)	671B DeepSeek-R1 / 1TB + Kimi-K2.5 (requires large memory + optional GPU)
Expert scheduling strategy	Three-tier streaming of video memory/main memory/disk + Router-Lookahead prefetching	CPU-GPU frequency-aware placement + dynamic reallocation + three-layer prefix cache
Hardware threshold	A pure CPU is sufficient; 25GB RAM + NVMe is the minimum requirement.	Recommended configuration: large memory workstation + consumer-grade GPU; CPU must be AVX2/AMX.
Dependence on volume	Pure C, zero dependencies, extremely lightweight	Python + PyTorch + CUDA, relatively heavy
Quantitative support	Global int4 + handwritten AVX2 kernel	FP8 / BF16 / INT8 / INT4 / GGUF multi-level quantization
Visualization	Expert route heatmap at the cerebral cortex level	Basic monitoring panel and logs
Fine-tuning capability	Reasoning only	Native support for LoRA SFT fine-tuning, integrated with LLaMA-Factory

Application scenarios of Colibrì

Personal Local Deployment Flagship Large ModelDevelopers without a server budget can experience the full inference capabilities of the 744B parameter GLM-5.2 on their home computers.
Offline inference for edge devicesRun ultra-large MoE models on memory-constrained industrial PCs or edge boxes to meet the intelligent decision-making needs in network-free environments.
Rapid preliminary study of model performanceIt allows you to verify the actual performance of ultra-large-scale models on specific business tasks without having to apply for expensive cloud resources.
Privacy-sensitive data processingIn scenarios such as healthcare, finance, and government affairs, data remains entirely local throughout the process, meeting stringent data privacy compliance requirements.
MoE Mechanism Teaching and ResearchLow-cost research expert routing, hierarchical caching, and quantitative inference technologies facilitate classroom demonstrations and academic paper experiments in universities.