TurboQuant - Google's vector quantization algorithm

What is TurboQuant?

TurboQuant, a vector quantization algorithm from Google Research, can compress large model key-value caches from 32-bit to 3-bit, achieving a 6x reduction in memory usage, an 8x increase in inference speed, and zero loss of accuracy. TurboQuant transforms vectors to a coordinate system following a Beta distribution through random rotation, combined with 1-bit QJL residual correction, eliminating the need for calibration constants and model fine-tuning, making it plug-and-play. TurboQuant has been validated to support long-context tasks for models such as Gemma and Mistral, providing a key breakthrough for edge device deployment and reducing cloud inference costs.

TurboQuant's main functions

• Extreme compressionIt compresses the 32-bit floating-point KV cache to 3-bit, reducing memory usage by more than 6 times, while also supporting extreme compression modes with a minimum of 1-bit.
• Accelerate reasoningThrough highly vectorized quantization computation, attention computation speed is increased by 8 times on H100 GPU, significantly reducing inference latency.
• Precision maintenanceIn long-context benchmark tests such as LongBench and Needle in a Haystack, the compressed model scores exactly the same as the original model, achieving true zero precision loss.
• Plug and playIt employs a data-agnostic online quantization strategy, eliminating the need for model retraining, fine-tuning, or calibration for specific datasets, thus having a low deployment threshold.
• Dual-mode quantizationIt provides an MSE optimization mode to minimize reconstruction error and an inner product optimization mode to provide unbiased attention score estimation, meeting the needs of different application scenarios.
• Applicable to multiple scenariosIt is suitable for large-scale model KV cache compression to support ultra-long contexts and nearest neighbor search of vector databases, and outperforms traditional methods in both recall and indexing speed.

TurboQuant's technical principles

• Random rotation dimensionality reductionBy applying a random rotation matrix to the input vector, the high-dimensional vector is transformed from the Cartesian coordinate system to a space where each coordinate follows a Beta distribution, making the different coordinates nearly independent. This allows for the independent application of optimal scalar quantization to each coordinate, eliminating the need to store data-dependent calibration constants.
• Optimal scalar quantizationBased on the statistical properties of the Beta distribution, the Lloyd-Max algorithm is used to solve the continuous one-dimensional k-means problem. The optimal quantization codebook is pre-calculated for each coordinate to achieve near-optimal MSE distortion rate.
• Two-stage residual correctionThe MSE optimal quantizer is applied for primary compression, and then a 1-bit Quantized Johnson-Lindenstrauss transform is applied to the residual vector for correction, eliminating the bias in the inner product estimation and achieving unbiased and low-distortion attention computation.
• Information Theory Optimal GuaranteeBy proving that TurboQuant's distortion rate differs from Shannon's theoretical lower bound by only about 2.7 times the constant factor, and is closer to the optimal value at low bit depths, the algorithm's limit performance is theoretically verified.

Key information and usage requirements for TurboQuant

• PublisherThis paper, a joint effort between Google Research and Google DeepMind, was published at ICLR 2026.
• Key IndicatorsKV Cache is compressed to 3-bit, reducing memory usage by 6 times, increasing inference speed by 8 times, and achieving zero loss of accuracy.
• Technology combinationIt consists of two stages: PolarQuant (random rotation + Beta distribution quantization) and QJL (1-bit residual correction).
• Theoretical guaranteeThe distortion rate is no more than 2.7 times the lower bound of information theory, and only 1.45 times the lower bound for 1-bit.
• Validation ModelLarge open-source models such as Gemma and Mistral passed five long-context benchmark tests, including LongBench and Needle in a Haystack.
• Community RealizationMultiple third-party implementations, such as PyTorch, MLX, and C/CUDA, have emerged.
• No training requiredIt does not require retraining or fine-tuning the model and can be directly applied to the pre-trained model.
• No calibration requiredIt employs a data-independent online quantization strategy, eliminating the need for offline calibration or preprocessing for specific datasets.
• Hardware supportIt requires vector-enabled AI accelerators (such as GPUs) to achieve optimal performance, and the algorithm itself is not tied to specific hardware.

TurboQuant's core advantages

• Extreme compression ratioCompressing the 32-bit KV Cache to 3-bit reduces memory usage by more than 6 times, and supports minimum 1-bit extreme compression, significantly alleviating the memory bottleneck in long context scenarios.
• Zero precision lossIn five long-context benchmark tests, the compressed model scored exactly the same as the original model, achieving true lossless compression, rather than near-lossless compression.
• Reasoning speedup is significantThe highly vectorized algorithm design improves the speed of attention calculation by 8 times, effectively reducing inference latency and increasing throughput.
• Plug and play deploymentNo model retraining, fine-tuning, or data calibration is required; it is ready to use out of the box, significantly reducing the barriers to engineering implementation and deployment costs.
• Theoretical optimal performanceThe distortion rate differs from Shannon's lower bound by only about 2.7 times the constant factor, and the difference is even smaller at low bit depths, approaching the theoretical limit.

How to use TurboQuant

The official open-source code has not yet been released. You can follow the official Google Research repository or the arXiv paper page for the latest open-source information.

TurboQuant's project address

• Project official websitehttps://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/
• arXiv technical paper: https://arxiv.org/pdf/2504.19874

Comparison of TurboQuant's similar products

Application scenarios of TurboQuant

• Long Context LLM ServiceCompressing the KV Cache by 6 times enables cloud APIs to support millions of token contexts, significantly reducing computing costs and improving concurrency capabilities.
• Consumer-grade graphics card deploymentIt supports enabling consumer-grade GPUs with 32GB of video memory to smoothly run long-context tasks with models of 7B or higher, breaking the video memory bottleneck of locally deployed systems.
• Edge device inferenceIt provides compression solutions for memory-constrained scenarios such as mobile phones and IoT devices, enabling large model capabilities to be deployed to edge devices.
• Vector database retrievalIt replaces traditional Product Quantization, enabling semantic search with higher recall and lower indexing latency in RAG systems.