Quantized models

Quantization is a model optimization technique: it converts high-precision parameters (FP32, FP16) into lower-precision integers (INT8, INT4) to boost efficiency.

Quantization is a critical deployment process, mapping a model's weights and activations from high-precision floating-point formats (FP32 or FP16) to low-bit integers, typically INT8 or INT4. This compression directly addresses the resource demands of large models (LLMs), significantly cutting the memory footprint—often by 75%—and accelerating inference speeds by up to 40% on compatible hardware (e.g., NVIDIA TensorRT). The core benefit is enabling efficient, low-latency deployment on resource-constrained environments: think mobile devices, edge computing, and consumer GPUs. The trade-off is a minimal, managed loss in model accuracy (quantization error) for massive gains in operational efficiency.

https://llm-stats.com/blog/model-quantization-across-providers/