Private AI

Local LLM vs Cloud AI: Which One Should Your Business Use?

A side-by-side of local models and cloud APIs: what you pay, what you give up, and a decision tree that does not assume local is the answer.

The local-versus-cloud argument is usually a proxy fight. One side has a privacy policy. The other side has a demo that looks like a consumer chatbot. Neither is a decision procedure.

This comparison is for people who have to pick an architecture for a real workflow: transcription, search, drafting, extraction, or an internal assistant. It does not conclude that a local LLM for business is automatically better. It concludes that you should be able to say what leaves the building, what it costs, and who operates it.

If you already know you need a mixed design, that is the normal outcome of private AI deployment work. What follows is the table and the tree.

What is being compared

Local LLM: weights on a machine you operate (workstation or on-premise server). Prompts stay there unless you send them elsewhere.

Private cloud AI: the same software shape, on a VM or GPU instance in an account you control. Not a public chatbot. The datacentre is still someone else’s building.

Public cloud AI API: you send text (and maybe files) to a vendor’s model endpoint. You pay per token or per minute. You get frontier quality and someone else’s operations.

People say “local vs ChatGPT” when they mean “do we send this text to a vendor”. Be exact. A private GPU in a cloud account is not ChatGPT. A local 8B model is not “ChatGPT minus the bill”.

Comparison table

No column wins on all rows. If a vendor slide has a column of ticks and a column of crosses, it is a slide.

Model quality

Public APIs currently win on general reasoning, messy instructions, and “make this nicer”. That gap narrows on narrow tasks with good context: summarise this transcript, fill this schema, answer from these four passages.

Local quality is a function of model size, quantisation, and the prompt. A well-retrieved document plus a 32B-class local model often beats a frontier model that was given no documents and a vague question. A frontier model with the same documents will often still win. Measure on your files, not on a public leaderboard.

If the product requirement is “indistinguishable from the best hosted chat”, local inference is the wrong tool. If the requirement is “good enough, reviewable, and the text stays here”, local is in the race.

Privacy and data control

Local is the strongest story for prompts that must not go to a SaaS AI vendor. It is not a cryptographic guarantee. Disks fail, admins copy files, logs capture bodies.

Public APIs can be acceptable under a contract, a DPA, retention settings, and a classification that says this class of text may travel. Many companies already send worse things to email and ticketing vendors. Hypocrisy is not a strategy, but neither is pretending an API call is “just like sending a letter”.

Private cloud sits in the middle: you avoid multi-tenant AI products; you still trust a datacentre operator and your own IAM.

Write the path. “Here is what runs locally. Here is what leaves. Here is the hardware requirement.” That sentence is worth more than a privacy slogan.

Cost

Do the sum on a unit of work.

Public API. Easy to start. You pay per token, per audio minute, or per embedding. Painful when speech-to-text or embeddings run all day, or when a chat UI is left open on a 100k-token corpus. Caps and routing help. They do not make the cost graph flat.

Local. No per-token bill. Completions on hardware you control do not appear on a vendor usage invoice. You still have capex or a workstation you already bought, then electricity, maybe a GPU, and engineering. Cheap at high utilisation. Expensive as a monument.

Private cloud GPU. The honest number for many European teams: no purchase order for a card, hourly burn while the instance is up, and the same engineering as local. Turn it off at night if the workflow allows.

A local system is not cheaper because it is local. It is cheaper when the API would have been running anyway, at volume, for a long time, and you can keep the GPU busy. Otherwise you bought a very stylish space heater.

Worked hardware ranges belong in local LLM hardware requirements; they go stale. The cost structure above does not.

Maintenance

APIs: you maintain prompts, keys, and the application. The model host is not yours.

Local and private cloud: you maintain the application and the inference stack. Weights, drivers, CUDA or Metal dramas, disk, backups, “why is VRAM full”. If you do not have a person for that, the API is not overpriced. It is the operations you were trying not to hire.

Scalability

APIs scale until your account or the vendor’s region has a bad day. You will not provision the fleet.

Local scales by procurement. That is slow, which is fine if load is stable. It is fatal if you promised “the whole company” on one workstation.

Private cloud scales faster than a purchase order and slower than a fully managed API. Concurrency is still your problem; see on-premise LLM deployment for queues and batching, which apply on a rented GPU too.

Latency and offline

Local can be the lowest latency in the building: no egress, no vendor queue. It can also be the highest, if one GPU is doing transcription and chat.

Offline is the one dimension where public APIs cannot compete. If the site cannot call out, you run locally or you do not run.

Customization and time to production

Local: you can pin a model forever, air-gap it, and put a JSON schema in front of it. You cannot fine-tune a frontier closed model you do not have.

API: you ship a prototype this afternoon. You accept that the model name on the other end will change, get more expensive, or grow a new policy.

Speed to production is not the same as speed to prototype. A local system that has to pass IT, procurement and an index of real files takes weeks. An API wrapper takes a weekend and then a longer security review than people budgeted.

Decision tree

Start at the top. Stop at the first decisive yes.

1. Must the workflow run with no outbound internet?
   Yes → local (or a private network you already treat as the universe).
   No → continue.

2. Is the text classified such that a public AI vendor is not an allowed processor?
   Yes → local or private cloud. Choose based on whether you want to own the box.
   No → continue.

3. Is frontier quality mandatory for this workflow, even with good retrieval?
   Yes → public API, possibly with retrieval still local (hybrid: documents stay, prompts for the last step may still leave — know that).
   No → continue.

4. Is usage low or bursty (a few people, a few times a week)?
   Yes → public API, unless step 2 already forbade it.
   No → continue.

5. Will the GPU (owned or rented) actually stay busy, and is there an owner for the stack?
   Yes → local or private cloud, based on capex versus opex and where the data must sit.
   No → public API, or do not build this.

6. Do you have mixed answers for mixed workloads?
   Yes → hybrid. Sensitive audio and documents locally; non-sensitive drafting on an API. This is the common adult outcome.

If you cannot get through the tree because the workflow is unnamed, go back to when a local LLM makes sense and name the input and the output first.

A note on “self-hosted LLM vs API”

Self-hosted does not mean free, private, or better. It means you are the platform team. APIs do not mean reckless. They mean you accepted a processor.

The boring architecture wins: pin the data flows, run a pilot on real files, compare quality and cost on the same samples, then choose. Anyone who already knows the answer before seeing your corpus is selling something, including me if I skip that step.

When you want that comparison done as a piece of work rather than a slide, use the audit on the private AI for business page. The deliverable is a recommendation that is allowed to be “stay on the API”.