AB
AB
AiBoss

All Categories

project

Gemma 3 QAT - Google's latest open-source model, a quantized version of Gemma 3.

Gemma 3 QAT (Quantization-Aware Training) is Google's latest open-source model, a quantization-optimized version of Gemma 3. Through quantization-aware training technology, Gemma 3 QAT significantly reduces memory usage...

What is Gemma 3 QAT?

Gemma 3 QAT (Quantization-Aware Training) is Google's latest generation of open-source models, a quantization-optimized version of Gemma 3. Through quantization-aware training technology, Gemma 3 QAT maintains high-quality performance while significantly reducing memory requirements. The Gemma 3 27B's VRAM requirement has been reduced from 54GB to 14.1GB, enabling it to run natively on consumer-grade GPUs such as the NVIDIA RTX 3090. The Gemma 3 12B's VRAM requirement has been reduced from 24GB to 6.6GB, allowing it to run efficiently on laptops with NVIDIA RTX 4060 GPUs. This enables more users to experience powerful AI capabilities on ordinary hardware.

Main functions of Gemma 3 QAT

  • Significantly reduce video memory requirementsBy using quantization-aware training technology, Gemma 3 QAT significantly reduces the model's GPU memory usage.
    • Gemma 3 27BThe video memory requirement has been reduced from 54GB (BF16) to 14.1GB (int4), enabling it to run on consumer-grade GPUs such as the NVIDIA RTX 3090 (24GB VRAM).
    • Gemma 3 12BThe video memory requirement has been reduced from 24GB (BF16) to 6.6GB (int4), allowing it to run efficiently on a laptop's NVIDIA RTX 4060 (8GB VRAM).
    • Smaller versions (4B, 1B)It can even run on resource-constrained devices such as mobile phones.
  • Maintain high performanceGemma 3 QAT maintains performance comparable to the native BF16 model: On Chatbot Arena Elo scores, the quantized version of Gemma 3 QAT remains competitive, performing no worse than top language models. Using QAT technology, Google reduced perplexity by 54% in approximately 5000 training steps, ensuring the model maintains high accuracy after quantization.
  • Multimodal capabilitiesIt supports image input and text generation, making it suitable for tasks such as visual question answering (VQA) and document analysis.
  • Long context supportIt features a context window with 128,000 tokens and is optimized through a hybrid attention mechanism (local sliding window attention and global attention), which reduces the memory footprint of the key-value cache.
  • Hardware supportGemma 3 QAT can run on a variety of consumer hardware, including desktop GPUs, laptop GPUs, and edge devices.
  • Framework supportIt supports mainstream inference frameworks such as Ollama, LM Studio, llama.cpp, and MLX, allowing users to easily deploy it on different platforms.

Technical Principles of Gemma 3 QAT

  • Pseudo-quantization operationDuring the forward propagation of training, the model simulates low-precision calculations through "pseudo-quantization" nodes, rounding the weights and activation values to the quantized values.
  • High-precision backpropagationDuring backpropagation, the model still uses high-precision floating-point numbers to calculate gradients, ensuring the accuracy of weight updates.
  • Combining training and quantificationIn this way, the model learns how to maintain performance in low-precision environments during the training phase, and the accuracy loss is minimal after actual quantization.
  • KV caching optimizationBy using sparse caching and dynamic compression techniques, the memory footprint in long-context tasks can be further reduced.
  • Hardware accelerationSupports SIMD instruction set optimizations, such as AVX512 and NEON, resulting in a 3x speedup for inference.

Project address for Gemma 3 QAT

Application scenarios of Gemma 3 QAT

  • Visual Question Answering (VQA)Gemma 3 QAT performs well in multimodal tasks, and the quantized version performs close to FP16 on tasks such as DocVQA.
  • Document AnalysisIt supports long context windows (128K tokens), making it suitable for tasks that require processing large amounts of text, such as document analysis.
  • Long text generationThrough KV caching optimization and Grouped Query Attention (GQA), Gemma 3 QAT reduces memory usage by 40% and improves inference speed by 1.8x in a 128K context window.
  • Long sequence reasoningSuitable for tasks that require processing long sequences, such as long document analysis and complex language model reasoning.
  • Edge device deploymentThe 1B version (529MB) of Gemma 3 QAT can run offline on Android or Web with a latency as low as 10ms, making it suitable for privacy-sensitive scenarios (such as medical and financial).
Back