Ollama vs vLLM vs llama.cpp: Choosing a Self-Hosted Inference Server

Updated 2026-10-26 · 11 min read

How the three most common self-hosted LLM inference servers differ in architecture, hardware support, concurrency and operations, and which one fits your workload.

Ollama vs vLLM vs llama.cpp is the first real decision when you self-host an LLM, and the three are less interchangeable than their shared OpenAI-compatible API suggests. They sit at different layers: llama.cpp is an inference engine, Ollama is a packaged server built around the same ggml technology, and vLLM is a GPU serving engine designed for many concurrent users.

Short answer: use Ollama for a single developer, a small team or a quick internal chatbot where convenience matters most. Use llama.cpp (llama-server) when you want the same GGUF efficiency with full control over every flag, or you run on CPU, Apple Silicon or mixed hardware. Use vLLM when you serve many concurrent requests on NVIDIA or AMD data-center GPUs and throughput per GPU is what you pay for.

The rest of this guide explains why, so you can make the call for your own hardware and traffic.

How the three are built

llama.cpp: the engine and the GGUF format

llama.cpp is a C/C++ inference engine built on ggml, a tensor library with backends for CPU (with AVX/NEON paths), CUDA, ROCm/HIP, Metal, Vulkan, SYCL and more. It defined the GGUF file format: a single file holding the weights, tokenizer and chat template, with a family of quantization types (Q80, Q6K, Q5KM, Q4KM and others) designed to run well on ordinary hardware.

The project ships llama-server, an HTTP server with an OpenAI-compatible API (/v1/chat/completions, /v1/completions, /v1/embeddings, /v1/models), a /health endpoint and optional Prometheus metrics. It handles several requests at once through slots and continuous batching, which is enabled by default. Layers can be split between GPU and CPU (-ngl), which is the main reason llama.cpp runs models that do not fit in VRAM at all.

Ollama: packaging around ggml and llama.cpp

Ollama is a Go server that wraps model execution in a product: a model registry (ollama pull qwen2.5:7b), Modelfiles for custom system prompts and parameters, automatic GPU detection, loading and unloading of models on demand, and a native API on port 11434 alongside an OpenAI-compatible /v1 API.

Under the hood it started as a wrapper around llama.cpp. Today it runs GGUF models on ggml with a mix of llama.cpp code and its own model implementations, and since version 0.19 it has an MLX-based engine on Apple Silicon in preview. For you as an operator the important part is the same: Ollama runs quantized GGUF-style models, and its performance profile is close to llama.cpp's, with fewer knobs exposed.

vLLM: a GPU serving engine

vLLM is a Python/PyTorch serving engine built for throughput. Its two core ideas:

• PagedAttention. The KV cache (the per-token attention state every running request needs) is stored in fixed-size blocks, like virtual memory pages, instead of one contiguous buffer per request sized for the maximum context. This wastes far less memory, so more requests fit on the GPU at once, and identical prompt prefixes can share blocks.
• Continuous batching. The scheduler adds new requests to the running batch at every decoding step instead of waiting for a whole batch to finish. A short request does not wait behind a long one.

On top of that come automatic prefix caching, CUDA graphs and torch.compile, speculative decoding, tensor and pipeline parallelism across GPUs, and a broad set of GPU quantization formats. vLLM loads models in Hugging Face format (safetensors) directly from the Hub or a local path and serves an OpenAI-compatible API on port 8000.

The API is the same, the semantics are not

All three speak POST /v1/chat/completions, so LangChain, LlamaIndex, Open WebUI or the official OpenAI SDKs work with a changed base_url. Differences show up at the edges: tool-calling parsers, structured output support, how the model name is resolved, and authentication. vLLM and llama-server accept --api-key. Ollama's local server has no API key setting, so put it behind a reverse proxy or keep it on a private network.

Hardware support

| Hardware | llama.cpp | Ollama | vLLM | |---|---|---|---| | x86 / ARM CPU only | Yes, well optimized | Yes | Yes, CPU backend, slower and less common in production | | Apple Silicon | Yes, Metal, first-class | Yes, Metal, plus MLX engine (preview) | CPU backend, GPU through the separate vLLM-Metal plugin | | NVIDIA | Yes, CUDA | Yes, CUDA | Yes, primary target | | AMD | Yes, ROCm/HIP and Vulkan | Yes, ROCm (rocm image tag) and Vulkan | Yes, ROCm, best on data-center Instinct GPUs | | Intel GPU | SYCL, Vulkan | Vulkan | XPU backend | | Partial GPU/CPU offload | Yes, per layer or per tensor | Yes, automatic | Limited (--cpu-offload-gb), slow |

Two practical notes. First, Docker on macOS cannot pass the Apple GPU into a Linux container, so on a Mac run Ollama or llama.cpp natively and containerize only the rest of the stack. Second, on consumer AMD cards, check the ROCm support list for your exact GPU. Vulkan is the fallback when ROCm does not cover it.

Concurrency and throughput

This is where the three differ most, and where marketing benchmarks mislead most. Two phases matter:

• Prefill processes the prompt. It is compute-bound and parallelizes well.
• Decode generates one token at a time. At batch size 1 it is bound by memory bandwidth: every new token reads all the weights once.

For one user at a time, all three engines are limited by the same physics. A 4-bit GGUF in llama.cpp or Ollama reads fewer bytes per token than an FP16 model in vLLM, so a single chat session can feel just as fast or faster on the GGUF engines. The gap opens when many requests arrive together. Batching lets one read of the weights serve many sequences, so throughput rises with concurrency until compute or KV cache memory runs out. vLLM's scheduler and paged KV cache are built exactly for that regime.

Ollama

OLLAMANUMPARALLEL sets how many requests one loaded model processes at once. The default is 1, and further requests queue (up to OLLAMAMAXQUEUE, default 512, then HTTP 503). Raising it multiplies KV cache memory: the docs state that required memory scales with OLLAMANUMPARALLEL × OLLAMACONTEXTLENGTH. Ollama also loads several models at once if they fit (OLLAMAMAXLOADEDMODELS) and unloads idle ones after OLLAMAKEEP_ALIVE (default 5 minutes). That is excellent for a team switching between models and a poor fit for a high-traffic endpoint.

llama.cpp

llama-server runs N slots (-np) with continuous batching. When you set the slot count yourself, the context size -c is divided among the slots, so -c 32768 -np 4 gives each request up to 8,192 tokens. With automatic slots the server uses a unified KV buffer shared by all sequences. Concurrency works, but the KV cache is still reserved up front rather than paged, so you plan capacity per slot.

vLLM

vLLM pre-allocates a fraction of GPU memory (--gpu-memory-utilization, default 0.92 in current releases), loads the weights, and turns the rest into KV cache blocks. At startup it logs how many tokens of KV cache it has and the maximum concurrency at your --max-model-len. Requests take blocks as they grow and release them when done, and --max-num-seqs caps how many run at once. If the cache fills up, vLLM preempts and re-schedules requests instead of failing them. Under real multi-user load this is the behavior you want.

The trade-off: vLLM takes the GPU for itself, starts slower (model load plus compilation), and is wasteful for a single user.

Quantization formats

| Format | llama.cpp / Ollama | vLLM | Notes | |---|---|---|---| | GGUF K-quants (Q4KM, Q5KM, Q6K) | Native | Through the out-of-tree GGUF plugin, documented as experimental | Q4KM is the common default for quality per byte | | GGUF Q80 | Native | Plugin | Near-FP16 quality at roughly half the size | | FP16 / BF16 safetensors | Can convert to GGUF F16/BF16 | Native | Baseline quality | | FP8 (W8A8) | No | Native on GPUs with FP8 support | Good default for vLLM on recent NVIDIA and AMD GPUs | | AWQ, GPTQ (4-bit) | No | Native | Widely available on the Hub | | NVFP4 / MXFP4 | Limited | Supported on matching hardware | Newer GPU generations | | KV cache quantization | -ctk/-ctv q80, OLLAMAKVCACHETYPE=q8_0 | --kv-cache-dtype fp8 | Halves KV memory at a small quality cost |

The practical consequence: if your model source is GGUF from the Ollama library or a Hugging Face GGUF repo, you are in llama.cpp/Ollama territory. If you start from safetensors, FP8 or AWQ checkpoints, vLLM is the natural home. How much memory each format needs is worked through in our VRAM guide.

Running them in Docker

All three ship official images. Pin an exact tag in anything you deploy. latest changes under you on the next pull, and these projects release often: Ollama and vLLM ship new versions every few weeks, llama.cpp several builds a day. A pinned tag means a restart never upgrades the engine, the model format support or the default flags without you noticing. The tags below were current when this guide was written; check the release pages before you copy them.

Ollama

services:
  ollama:
    image: ollama/ollama:0.40.11
    ports:
      - "11434:11434"
    environment:
      OLLAMA_CONTEXT_LENGTH: "8192"
      OLLAMA_NUM_PARALLEL: "4"
      OLLAMA_KEEP_ALIVE: "-1"        # keep models loaded
      OLLAMA_KV_CACHE_TYPE: "q8_0"
    volumes:
      - ollama:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  ollama:

docker compose exec ollama ollama pull qwen2.5:7b
docker compose exec ollama ollama ps   # shows GPU/CPU split and context per model

Model management is Ollama's strongest point: pull by name, models live in the volume, and ollama ps tells you whether a model spilled onto the CPU. Set OLLAMACONTEXTLENGTH explicitly, because recent versions pick the default context based on available VRAM, which makes behavior differ between machines. For AMD, use the 0.40.11-rocm tag with /dev/kfd and /dev/dri passed through.

llama.cpp

services:
  llama:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda-b11889
    ports:
      - "8080:8080"
    volumes:
      - ./models:/models:ro
    command:
      - "-m"
      - "/models/Qwen2.5-7B-Instruct-Q4_K_M.gguf"
      - "--host"
      - "0.0.0.0"
      - "--port"
      - "8080"
      - "-c"
      - "32768"
      - "-np"
      - "4"
      - "-ngl"
      - "all"
      - "--api-key"
      - "${LLAMA_API_KEY}"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Set --port explicitly, since the built-in default is not 8080 in current builds. Other image variants follow the same pattern: server-rocm-, server-vulkan- and server-* for CPU only. There is no registry; you download GGUF files yourself (for example from bartowski's or the model vendor's GGUF repos on Hugging Face) or let the server fetch one with -hf :Q4KM. That is more work, and it also means you know exactly which file is running.

vLLM

services:
  vllm:
    image: vllm/vllm-openai:v0.32.0
    ports:
      - "8000:8000"
    ipc: host
    environment:
      HF_TOKEN: "${HF_TOKEN}"
    volumes:
      - hf-cache:/root/.cache/huggingface
      - vllm-cache:/root/.cache/vllm
    command:
      - "Qwen/Qwen2.5-7B-Instruct"
      - "--max-model-len"
      - "16384"
      - "--max-num-seqs"
      - "64"
      - "--api-key"
      - "${VLLM_API_KEY}"
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

volumes:
  hf-cache:
  vllm-cache:

ipc: host (or a large shm_size) is needed because PyTorch uses shared memory between processes, especially with tensor parallelism. Mount the Hugging Face cache so a restart does not download the weights again, and the vLLM cache so compiled artifacts are reused. Always set --max-model-len: by default vLLM uses the model's full context window, which may not fit in KV cache and makes startup fail. The image comes in several CUDA variants (for example v0.32.0-cu129); match it to your host driver. For multiple GPUs add --tensor-parallel-size 2. AMD has its own ROCm images (rocm/vllm).

Health checks: vLLM and llama-server both expose GET /health, and Ollama answers on GET /api/version. vLLM serves Prometheus metrics on /metrics out of the box, and llama-server does when started with --metrics.

Ollama vs vLLM vs llama.cpp: decision table

| Situation | Pick | Why | |---|---|---| | Laptop, Mac or workstation, one user | Ollama | Fastest path from zero to a working model | | Small team chatbot, a few concurrent users, several models | Ollama | On-demand loading and a model registry; raise OLLAMANUMPARALLEL | | CPU-only server or a model larger than your VRAM | llama.cpp | Best CPU paths and fine-grained GPU/CPU offload | | Apple Silicon in production | llama.cpp or Ollama | Metal is first-class; vLLM on Mac is not a production target | | Exact control of flags, reproducible builds, edge devices | llama.cpp | One binary or image, every option exposed | | Many concurrent users on NVIDIA or AMD data-center GPUs | vLLM | Paged KV cache and continuous batching keep the GPU busy | | RAG backend with high request volume and long shared prompts | vLLM (or SGLang) | Prefix caching reuses the shared system prompt and context | | FP8, AWQ or GPTQ checkpoints, multi-GPU tensor parallel | vLLM | Native support for these formats and parallelism modes |

A common pattern is to use both: Ollama on developer machines and vLLM in the shared environment, with the application talking to /v1/chat/completions in both cases. Keep the model family the same and test that tool calling and chat templates behave the same on both, because that is where the engines differ most.

When to look at TGI or SGLang

Text Generation Inference (TGI) from Hugging Face was for a long time the main alternative to vLLM. Its repository now states that it is in maintenance mode, accepting only minor fixes, and points users to vLLM and SGLang. Keep an existing, tested TGI deployment if it works, but do not start a new one on it.

SGLang is a serious alternative to vLLM for GPU serving. Its RadixAttention keeps a radix tree of cached prefixes, which pays off when many requests share long prefixes: agents that resend the same tool definitions and history, RAG prompts with a fixed system prompt, few-shot templates. It also has strong structured-output support and is widely used for large mixture-of-experts models. It runs from the lmsysorg/sglang image (for example lmsysorg/sglang:v0.5.22) with python3 -m sglang.launch_server --model-path --host 0.0.0.0 --port 30000 and also exposes an OpenAI-compatible API. If your workload is agent-heavy, benchmark SGLang and vLLM against your own traffic before you commit; results depend heavily on the model and prompt shape.

Next steps

• Size the hardware first: How much VRAM you need to run an LLM gives the formulas for weights and KV cache.
• Start from a working setup: the LLM chatbot stack runs Ollama with a chat UI, and the RAG pipeline stack wires an LLM server to a vector database.
• Moving to Kubernetes: Deploy Open WebUI and Ollama with Helm.
• Browse all chatbot stacks or the full stack catalog.