SLMHow to choose a small language model and the hardware to run it
A clear guide to model size, quantisation, GPU memory and workload testing before buying a local AI machine.
Choose a small language model by testing your actual task, then size hardware around its memory and response-time needs. Parameter count alone does not tell you whether the model will answer accurately or how many people it can serve.
Small is a design choice, not a quality score
A compact model can be useful for narrow tasks such as classifying requests, extracting fields or drafting short summaries. Broader reasoning may require a different model or a more constrained workflow. Microsoft lists Phi-4-mini-instruct as a 3.8-billion-parameter model, illustrating that useful candidates exist well below the largest model sizes.
Separate model weights from working memory
As a rough arithmetic estimate, four-bit weights for an 8B model occupy about 4 GB before runtime overhead. A 32B model needs about 16 GB for those weights alone. Context caches, concurrent requests, embedding models and other processes add memory. Quantisation reduces storage and memory but can change answer quality; evaluate the exact compressed version.
Watch the practical frontier
Smaller models, lower-precision inference and local speech pipelines expand the tasks a workstation can attempt. The useful advance is a workload running well on available hardware, not a benchmark headline. Compare the exact runtime, prompt length, model version and output quality. Published Qwen benchmarks show why software and input settings belong beside any speed figure.
Create a short acceptance test
Collect representative examples and define what a good answer must contain. Include an unsupported question, a long document and the languages your team uses. Measure factual accuracy, source quality, time to first output, full completion time and behaviour under simultaneous requests. Keep a larger-model baseline when deciding whether the compact option is sufficient.
Buy for the workload you can demonstrate
Core proposes 32 GB GPU memory; Advanced proposes 96 GB ECC; Custom starts from 192 GB total across GPUs. These are proposed configurations, with final parts confirmed in a quote. Memory across multiple GPUs is not automatically one shared pool. Check the model licence, runtime support, power, cooling and maintenance plan before committing.
Is more GPU memory always faster?
No. More memory may let a workload fit, but speed also depends on GPU bandwidth, compute, runtime, model format, context length and concurrency.
Sources & further reading
Reviewed 4 November 2026. These are practical deployment guides, not measured customer results.