Private AI development
Private, on-premises language models running on hardware you control, for documents that cannot leave your network. And an honest answer about when the cloud is the better buy.
- Data leaves network
- Never
- Typical GPU server
- From ~10-15k EUR
- Open models
- Llama, Mistral, Qwen
- Most common design
- Hybrid
What on-premises AI means
On-premises AI means the language model runs on hardware your company owns or rents exclusively, inside your own network. Prompts, documents and customer records are processed locally and never sent to an external provider. The alternative is a cloud API, where the text leaves the building and is handled by a third party under contract.
We deploy both, and we have no commercial preference between them. What follows is the actual decision framework we use with clients, including the parts that argue against buying hardware.
When data genuinely cannot leave the company
There are three situations where the cloud is ruled out by law, contract or risk rather than by technology.
- Personal and special-category data. GDPR requires a lawful basis and a data processing agreement. That is solvable with EU-region cloud providers, but healthcare, legal and finance clients frequently decide not to transfer the data to a third party at any price.
- Manufacturing know-how. Drawings, formulations, pricing logic, production processes. Things whose leak costs you a competitive position. Even with a no-training guarantee in the contract, some boards judge the transfer risk itself unacceptable.
- Your clients' contracts. The clause everyone forgets. Your customer's agreement may state their data cannot be passed to subcontractors or processed outside the EU. Read what you signed before sending their documents to any API.
If one of these applies, it does not follow that everything moves on-premises. It follows that that specific slice of data gets processed internally.
What an on-premises LLM actually involves
"We will build our own ChatGPT" sounds like a software decision. In practice it is an infrastructure project with three ongoing cost centres.
Hardware
A serious open model in the 70B parameter class needs one or more professional GPUs. The investment often starts around ten to fifteen thousand euros and can be several times that depending on concurrency. Models in the 7-14B range run on much more modest hardware, and for classification, extraction and internal document search they are frequently sufficient.
Models
Llama, Mistral, Qwen and the other open families have improved considerably over the past year. They generally do not match the newest commercial models, particularly on harder reasoning and on smaller languages. This has to be tested against your real data, not decided from a benchmark table or a demo.
Maintenance
On top of the hardware: electricity, cooling, backups, security updates and a named person who owns the thing. An on-premises system without a responsible administrator becomes no-man's-land inside a year, and the degradation is quiet.
On-premises, cloud or hybrid
| On-premises | Cloud API | Hybrid | |
|---|---|---|---|
| Data leaves network | Never | Yes, under contract | Only non-sensitive data |
| Time to first result | Weeks, hardware first | Days | Days for the cloud half |
| Cost at low volume | Poor | Excellent | Good |
| Cost at high steady volume | Excellent | Poor | Good |
| Access to frontier models | No | Yes | Yes, where permitted |
| Handles seasonal spikes | Badly | Automatically | Automatically |
| Needs an administrator | Yes | No | Yes, lighter |
The hybrid route, which is what most companies end up with
All-or-nothing is rarely the right answer. The architecture that survives contact with reality looks like this: a small local model anonymises or classifies documents before they leave the company network, or the whole sensitive analysis stays inside. Everything else, meaning customer communication, drafting, analysis of public data, goes to the strongest cloud model available. You get frontier quality where the data is not sensitive and full control where it is, at several times less than moving everything onto your own servers.
What we deploy
- Model serving. Open-weight models behind a stable internal API, so your applications do not care which model is underneath and you can swap it later.
- Retrieval over your documents. Your file server, wiki and ERP exports indexed locally, so answers cite an internal source rather than being generated from memory.
- A routing layer. Rules that decide, per request, whether it may go to the cloud or must stay local. This is the component that makes hybrid safe rather than theoretical.
- Monitoring and updates. Latency, throughput, GPU health and a review cadence for swapping in newer models as they land.
When on-premises is the wrong answer
If you have no IT administrator, if the workload is a few thousand requests a month, or if the honest reason for wanting it is that on-premises simply sounds safer, do not buy hardware. Cloud with an EU region, a signed data processing agreement, access control and sensitive fields stripped before sending will cost a fraction and perform better. Revisit on-premises when a specific legal or contractual requirement appears, or when steady volume makes the maths work.
Five questions that settle it
- Does the process touch personal or contractually restricted data? If yes, that part is a candidate for local processing or for anonymisation before it leaves.
- What is the volume? Under a few thousand requests a month, cloud usage fees beat owning hardware nearly every time.
- How steady is the load? Steady and high favours on-premises. Spiky and seasonal favours the cloud.
- Who will maintain it? No administrator means no on-premises, regardless of the other four answers.
- How good does the model have to be? If the task needs frontier reasoning, open models may not be there yet. Test before committing capital.
If you want those five answered against your actual situation rather than in the abstract, the AI readiness audit produces the architecture recommendation in a week, with the cost comparison attached.
Questions we get asked every time.
What does on-premises AI mean?
On-premises AI means the language model runs on hardware your company owns or rents exclusively, inside your own network. Prompts and documents are never sent to an external provider. The alternative is a cloud API, where your text leaves the building and is processed by a third party under a contract.
Is an on-premises LLM as good as ChatGPT or Claude?
Generally not, and anyone claiming otherwise is selling hardware. Open models such as Llama, Mistral and Qwen have improved enormously and are entirely capable for classification, extraction and internal search. On the hardest reasoning tasks and on smaller languages, frontier commercial models remain ahead. Test on your own data before deciding.
What does on-premises AI cost?
A serious open model in the 70B parameter class needs a server with one or more professional GPUs, which often starts around ten to fifteen thousand euros and can be several times that. Then add electricity, cooling, backups, security updates and a person to own it. Smaller 7-14B models run on far more modest hardware with correspondingly modest capability.
Does using a cloud AI provider break GDPR?
Not by itself. GDPR requires a lawful basis and a data processing agreement with the provider, and the major providers offer both, along with EU data regions. It becomes a problem when there is no agreement in place, when the data is special-category, or when your own client contracts forbid passing their data to subcontractors.
Can we run some things locally and some in the cloud?
Yes, and for most companies that is the right architecture. A small local model anonymises or classifies sensitive documents before anything leaves the network, and everything non-sensitive goes to the strongest cloud model available. You get frontier quality where it is safe and full control where it is not, at a fraction of all-on-prem cost.
Keep reading.
Does your data actually need to stay in the building?
Half the companies that ask us for on-premises AI do not need it, and we would rather say so on a free call than after you have bought a GPU server.
Book a free 30-minute call