Private AI

Local LLM Hardware Requirements: What Do You Actually Need?

How model size, quantization, VRAM, RAM and concurrency interact, with workload profiles you can update as hardware and models change.

Hardware conversations around local models go wrong in a familiar way. Someone posts a GPU name. Someone else posts a parameter count. A third person asks whether it will “run Llama”. None of those are the requirement.

The requirement is a workload: who uses it, how often, what goes in, what must come out, and how long they will wait. This article is a way to derive CPU, GPU, RAM, VRAM and disk from that, without a shopping list that will be stale by the next model release. For the business decision around owning any of this, start with local LLMs for business. For buying the work rather than the parts, see private AI systems.

The variables that matter

Five numbers dominate:

1. Parameters — how large the model is (7B, 14B, 32B, 70B, …).
2. Quantization — how many bits you store per weight (roughly: more bits, more quality, more memory).
3. Context length — how much text sits in the prompt plus the reply so far.
4. Concurrency — how many of those prompts are in flight at once.
5. Auxiliary models — speech-to-text, embeddings, rerankers, all of which want memory too.

If you remember nothing else: VRAM (or RAM, on CPU) has to hold the weights plus the KV cache for every active sequence. Everything else is comfort.

Parameters

Parameter count is a label, not a quality certificate. A 70B model that does not fit, and is therefore quantized into mush or swapped to disk, can lose to a 14B model that sits comfortably in VRAM and actually runs.

For business text tools, think in bands, not trophies:

• Small (roughly 7B–9B class): drafting help, extraction, short summaries. Fine for a workstation. Weak at hard reasoning and long, messy instructions.
• Mid (roughly 14B–32B class): the usual sweet spot for internal assistants with retrieval. Needs a serious GPU or patience on CPU.
• Large (70B class and up): closer to hosted quality on some tasks, expensive in VRAM, slower, harder to serve to a team.

Mixture-of-experts models change the arithmetic (active vs total parameters). Read the model’s own memory notes; do not assume dense-model rules.

Quantization

Quantization compresses weights. For GGUF-style quants, people talk in Q4, Q5, Q8 and so on. For GPU serving, you will also see FP16, 8-bit, and 4-bit recipes.

A usable mental model, not a promise:

• 4-bit-class weights: common production compromise for chat and RAG.
• 8-bit / higher: when you have measured a quality drop you cannot live with.
• FP16: when you have the VRAM and you are serving as a benchmark, not as a default.

Always test the quant on your prompts. JSON extraction and careful legal summarisation are where aggressive quants show up as missing fields and invented clauses.

Keep a table in the runbook: model id, quant, checksum, VRAM at load, VRAM at max context, tokens/s on a fixed prompt. That table is the artefact. A forum post about a GPU SKU is not.

VRAM

VRAM is the scarce resource for GPU inference.

At load time you need room for weights. At run time you need room for KV cache, which grows with context length and with batch size. Two users with 32k context are not “the same as one user”. They can be the difference between fits and does not fit.

When it does not fit, engines spill to RAM or they error. Spilling looks like the machine is working and the user is waiting. Erroring is kinder.

Do not plan VRAM from a marketing screenshot of a model “running” at 2k context. Plan from the longest prompt you will allow in the UI, times the number of concurrent interactive jobs, plus a margin for fragmentation and for a second model if you are sloppy about unloading.

RAM

System RAM still matters.

CPU inference holds weights in RAM. Some GPU setups also need substantial host RAM for loading, for the application, for the database, and for embeddings. A box with a large GPU and 16 GB of RAM is a poorly specified server, not a clever saving.

For CPU-only:

• small quantized models can be a personal tool;
• mid-size models need tens of gigabytes and will not feel interactive under load;
• the application, OS, and any index have to fit too.

If the machine is also a workstation, subtract what the browser and the IDE already eat. Those are not theoretical.

CPU inference

CPU inference is legitimate for:

• a single user;
• batch jobs that can run overnight;
• extraction that is not latency-sensitive;
• proving the prompts before a GPU exists.

It is the wrong default for a team chat UI. Threads, AVX, and llama.cpp will only get you so far. Measure tokens per second on a representative prompt. If you need more than a few tokens per second for interactive use, you are in GPU territory or you should shrink the model.

GPU inference

A GPU earns its keep when people are waiting, when context is long, or when speech-to-text shares the box.

You do not need the newest card in a review video. You need enough VRAM for the model band you chose, a driver stack your engineer will not hate, and a machine that can cool it. Datacentre cards and workstation cards differ in power, noise, and how you buy them. Pick for the room they will live in.

Sharing one GPU between Whisper and a 32B chat model is a queueing problem. Either schedule them, or use two devices, or accept that meetings will pause the assistant.

Context length

Context is not free. A 128k window is a marketing number. Filling it is slow and memory-heavy.

For RAG, prefer retrieving the right 2–8k tokens over stuffing a binder into the prompt. For transcription summaries, summarise in sections rather than loading a four-hour transcript in one shot.

Cap context in the application. Users will paste a repository. The hardware should not have to be sized for that habit.

Concurrency

One user is a laptop. A team is a server.

Interactive concurrency is usually small (a handful of simultaneous chats) but bursty at the start of the day. Batch concurrency is how you utilise the GPU: several transcription jobs, or re-embedding, in a queue.

Plan:

• max interactive sequences;
• max batch jobs;
• whether batch yields to interactive.

That is capacity planning. It is also why on-premise LLM deployment spends time on queues rather than on model brands.

Transcription requirements

Speech-to-text is a second workload. Faster-whisper-class pipelines like a GPU. CPU is possible for a trickle of files.

Size it by hours of audio per day and real-time factor on your recordings, not on clean studio speech. See local AI transcription for the pipeline. Hardware-wise, decide whether STT has its own GPU time or lives in the leftover slots of the chat GPU.

VRAM for STT is usually smaller than for a mid-size LLM, but it is not zero, and loading both models at once is how “it worked in the demo” becomes “CUDA out of memory” on Monday.

Embeddings

Embedding models are small compared with generators. They still want RAM or a slice of GPU, and they want disk for the index.

A document-search system that embeds on CPU while the GPU chats is often a good split. Re-embedding a large corpus is a batch job: schedule it, do not surprise the chat users with a full-scan at 10:00.

Index size on disk is a function of chunk count, not of how clever the model is. Budget for growth.

Storage

Weights + alternatives + STT + embeddings + documents + indexes + logs (the allowed kind) add up.

Use fast local disk for the active weights and the index. Use slower storage for cold recordings and old PDFs. Do not put the only copy of the index on the GPU host’s scratch disk without a backup.

Single-user vs multi-user

Single-user workstation. A recent CPU, 32 GB RAM as a comfortable floor for small-to-mid quantized models, optional consumer GPU if you want speed or a mid-size model. This is how you learn whether the workflow is real.

Small team (a handful of concurrent users). A dedicated machine. GPU VRAM sized for the mid-band generator at your max context, plus STT if you transcribe. SSO in front. Not someone’s laptop that goes home at 18:00.

Wider org. Either several services (chat vs batch), or a rented GPU profile you can enlarge, or you admit that a public API is the scaler and local is for the sensitive subset.

Example workload profiles

These are profiles, not quotes and not a SKU list. Update the “typical class” column when you change models; keep the workload column stable.

If your workload is not in the table, write a new row. Do not round it up to E because E sounds serious.

What not to do

• Do not hard-code a GPU model name into an architecture decision that has to live for years. Cards change. The VRAM-and-workload table does not.
• Do not buy the largest card “for the future” if you have no operator and no workflow. It will sit at idle and become a punchline.
• Do not size from a YouTube thumbnail of tokens per second.
• Do not forget the application server, the database, and the people. The GPU is one line in the bill of materials.

If you want the hardware derived from a real corpus and a real concurrency number, that is part of the audit on the local LLM setup for business page. The output should be a profile like the table above, plus a recommendation to rent, buy, or stay on an API — not a link to a shopping cart.