Model serving

Model serving is the work of running a trained model in production so it can answer real requests quickly and affordably. The hardware, usually GPUs, is expensive and does its best work on many inputs at once, so servers batch requests together: bigger batches mean more answers per second, but each request may wait a little longer to be grouped, so you’re always trading throughput against latency. For large language models, memory is often the bottleneck, both for the weights and for the cache of past tokens each conversation keeps. Quantization stores weights with fewer bits, say 8 or 4 instead of 16, shrinking memory and speeding things up with usually a small loss in quality.