Self-hosted AI chatbot stacks

Chatbot stacks

• LLM Chatbot (chatbot) – Conversational AI chatbot with memory and streaming

About Chatbot stacks

A self-hosted chatbot stack combines an LLM inference server (such as Ollama or vLLM) with a chat front end and a store for conversation history. Because the model runs on your machines, chats are not sent to a third-party API, and you choose which open-weight model to use.

Things to decide before you deploy:

• Model size versus the GPU memory you have
• Single user (Ollama is simple) or many concurrent users (vLLM batches requests efficiently)
• Authentication in front of the chat UI
• Whether answers should come from your documents, in which case you want a RAG stack instead

Frequently asked questions

Can I run a ChatGPT alternative on my own server?

Yes. Open-weight models served by Ollama or vLLM, combined with a chat UI such as Open WebUI, give you a private assistant that runs entirely on your own infrastructure.

Which hardware do I need for a local chatbot?

It depends on the model. Small quantized models run on a laptop or CPU server; larger models need a GPU with enough VRAM for the weights plus the context cache.

Ollama or vLLM for a chatbot?

Ollama is the easiest way to run models for a few users. vLLM is built for throughput and serves many concurrent requests more efficiently on NVIDIA GPUs.

Guides

• Ollama vs vLLM vs llama.cpp: Choosing a Self-Hosted Inference Server – How the three most common self-hosted LLM inference servers differ in architecture, hardware support, concurrency and operations, and which one fits your workload.
• How Much VRAM Do You Need for an LLM? Formulas and Tables – Calculate GPU memory for LLM inference from the model's parameter count and config: weights, KV cache and overhead, with estimate tables for 8B, 14B, 32B and 70B models.
• Deploy Open WebUI and Ollama on Kubernetes with Helm – A production-shaped Helm setup for Open WebUI and Ollama on Kubernetes, from GPU node preparation to TLS, sign-up control, upgrades and the failures you will most likely hit.
• Terraform GPU Instance on AWS for LLM Inference (EC2 + vLLM) – A minimal, pinned Terraform config for one GPU EC2 instance that serves an LLM with vLLM or Ollama, reachable through SSM instead of open ports, plus the quota and cost traps to avoid.
• Self-Hosting LLMs for GDPR: A Practical Checklist – Running the model on your own servers removes one big data flow, but you are still the controller for every prompt, log and embedding. This checklist covers the rest.