Hosting an open source LLM: vLLM, Ollama, and sovereignty

Using an LLM via an API (OpenAI, Mistral) is straightforward. But as soon as the conversation turns to strict sovereignty or very high volume, another path emerges: self-hosting an open source model. Two tools stand out, Ollama and vLLM, for two distinct use cases.

Why self-host an LLM

Three reasons, rarely just one:

• Sovereignty: your data never leaves your infrastructure. Critical in healthcare, finance, defense, and the public sector.
• Cost: at very high volume, the per-call cost of an API exceeds that of a well-utilized dedicated GPU.
• Control: fixed model version, no dependency on a provider that might change prices or models.

That reasoning is better grounded in verifiable facts than in intuition, because the most widespread fear is not the right one. OpenAI states in its API data documentation that data sent through the API is not used to train its models unless you explicitly opt in; content does, however, pass through abuse-monitoring logs retained for up to 30 days, and exclusion from those logs is subject to prior approval. That is where the real trade-off sits: not “does the model learn from my data?” but “where are the logs hosted, for how long, under which jurisdiction?”. Self-hosting removes that question; it creates others.

Ollama: simple, for getting started and local use

Ollama makes running an open source model (Llama, Mistral, etc.) on a machine effortless. Ideal for prototyping, local use, or moderate volume. Its limitation: it’s not designed to handle thousands of concurrent requests in production.

A useful clarification about Ollama’s scope, to avoid a classic disappointment: the tool is excellent for developing, trying models and running an assistant on a workstation, but it is not designed to serve dozens of simultaneous users. It largely processes requests one after another. On a development machine that is perfect; in multi-user production, latency collapses after a few concurrent requests. An open-weight model under Apache 2.0, as part of the Mistral catalogue is, can be deployed with either engine: the engine choice is independent of the model choice.

vLLM: high-throughput production

vLLM is an inference engine optimized for throughput. On GPU (Scaleway, OVH), it serves many requests in parallel with controlled latency, thanks to techniques like the continuous batching. It’s the tool when self-hosting needs to handle real load.

That throughput gain rests on two mechanisms worth naming, because they explain the gap. vLLM manages attention memory in pages rather than contiguous blocks, which avoids reserving memory for contexts that will never be filled, and it batches requests continuously: a finished request immediately frees its slot for a new one, without waiting for the batch to end. The result is that the same GPU serves markedly more requests per second than sequential execution would. That is the difference between a demo and a service.

When to self-host, when to use an API

• API (Mistral in the EU) for most projects: quick start, latest models, no GPU ops. See our Mistral agency page.
• Self-hosted (Ollama/vLLM) when sovereignty is strict, volume is very high, or both.

The cost calculation is worth doing in both directions, because it inverts with volume. An API is paid per token: the bill is zero when nobody uses the service, and grows linearly with usage. A GPU is paid per hour: it costs the same idle as at full load. So there is a crossover point, specific to each project, obtained by dividing the GPU’s monthly cost by the equivalent API’s price per million tokens. Below that point the API is cheaper; above it, self-hosting becomes cheaper. Add, on the self-hosted side, costs that are rarely budgeted: monitoring, model updates, and someone on call.

The choice depends on the required confidentiality level and the actual cost at your volume, it’s one of the trade-offs in our AI assistants connected to your data.

Need sovereignty constraints for your AI data? We’ll size the infrastructure with you.

Beyond hosting, we design your enterprise RAG on your internal data.