Self-Hosted AI Agents: Local LLM Tool Calling That Works

Updated 2026-10-26 · 12 min read

How tool calling works on Ollama and vLLM, which open-weight models handle it, and how to wrap it in an agent loop that is safe to point at real systems.

A self-hosted AI agent is an LLM running on your own hardware that can decide to call functions you define, read the results and keep going until the task is done. Local LLM tool calling works the same way as with hosted APIs: you send a list of tools as JSON Schema, the model replies with a structured tool_calls entry instead of text, your code runs the function and sends the result back. Ollama supports this out of the box for models tagged for tools; vLLM needs --enable-auto-tool-choice plus a model-specific --tool-call-parser. The model never executes anything itself. Your code does, which is where most of the engineering effort goes.

This guide covers the protocol, the server flags, a minimal agent loop in Python, MCP, the main open source AI agent frameworks and the parts that bite in production: permissions, prompt injection, observability and cost.

How tool calling works with OpenAI-compatible APIs

Both Ollama and vLLM expose an OpenAI-compatible /v1/chat/completions endpoint. Tool calling on top of it is a four-step exchange:

1. You send messages plus a tools array. Each tool has a name, a description and parameters as a JSON Schema object.
2. The model answers with an assistant message whose tool_calls field lists one or more calls, each with an id, a function name and arguments as a JSON string.
3. Your code parses the arguments, runs the function and appends a message with role: "tool", the matching toolcallid and the result as a string.
4. You call the API again with the extended history. The model either calls more tools or answers in plain text.

Under the hood, the model emits tokens in whatever format it was trained on (Hermes-style XML tags, Llama JSON, Mistral's [TOOL_CALLS] prefix, pythonic calls and so on). The server's chat template renders your tools into the prompt, and a parser turns the model's output back into the OpenAI structure. If the template or parser does not match the model, you get tool calls leaking into content as raw text, which is the most common failure when people start.

Enabling tool calling in Ollama and vLLM

Ollama

Ollama handles templates and parsing per model, so there are no server flags. Pick a model whose library page carries the "tools" tag and send tools in the request, either to the native /api/chat endpoint or to the OpenAI-compatible one at http://localhost:11434/v1/. The OpenAI client needs an API key value; Ollama ignores it, so any string works.

docker run -d --name ollama -p 11434:11434 -v ollama:/root/.ollama ollama/ollama:0.40.11
docker exec ollama ollama pull qwen3:8b

Pin the image tag instead of using latest. Tool-call parsing has changed between Ollama releases, and an unplanned upgrade is a classic way to break an agent that worked yesterday. Check the Docker Hub tags for the current release before you copy the version above.

One documented gap: Ollama's OpenAI-compatible endpoint accepts tools but not tool_choice. You cannot force a specific tool or require a call there. If you need that, use vLLM.

vLLM

vLLM needs two flags to turn on automatic tool calling:

• --enable-auto-tool-choice lets the model decide when to call tools.
• --tool-call-parser tells vLLM how to extract calls from that model's output.

Some models also need --chat-template pointing at a tool-aware Jinja template. Hermes, Mistral and Llama checkpoints usually ship a compatible template in tokenizer_config.json; for others, vLLM provides templates in its examples/ directory.

docker run -d --name vllm --gpus all -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.32.0 \
  --model Qwen/Qwen3-8B \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

A few parser names from the vLLM documentation, to show how model-specific this is:

| Parser | Model families (per vLLM docs) | |---|---| | hermes | Hermes 2 Pro / 3, Qwen2.5, QwQ-32B (Qwen3 instruct models use the same format) | | llama3json | Llama 3.1, 3.2 and 4 with JSON-style calls | | llama4pythonic | Llama 4 (recommended over JSON) | | mistral | Mistral function-calling models | | openai | gpt-oss-20b, gpt-oss-120b | | granite4 | Granite 4.0 | | qwen3xml | Qwen3-Coder | | deepseekv31 | DeepSeek-V3.1 |

The list grows with almost every release, so check the tool calling page of the vLLM docs for your version. vLLM also supports tool_choice: "required" and forcing a named function, both implemented with structured outputs. The first request with a new schema can be slower while the grammar compiles.

If you are still deciding between the two servers, the trade-offs are covered in Ollama vs vLLM vs llama.cpp. Short version: Ollama for a single developer or a small internal tool, vLLM when several users or agents hit the model concurrently.

Which open-weight models handle tools well

Treat any ranking as perishable. What holds up reasonably well:

• Use an instruction-tuned model with an official tool-calling template. Base models and many community fine-tunes do not have one. If the model card does not mention tools or function calling, assume it will be unreliable.
• Families that ship tool support: Qwen3 (and Qwen2.5), Llama 3.1 and later, Mistral Small and other Mistral instruct models, OpenAI's open-weight gpt-oss, IBM Granite, DeepSeek V3.x and GLM-4.5/4.6. All of them have dedicated parsers in vLLM.
• Size matters more than for chat. Small models (roughly under 7B parameters) often pick the right tool for simple single-step tasks but struggle to chain several calls, recover from tool errors or respect argument types. As a starting point, test with something in the 8B to 32B range and move down only if your evals pass.
• Quantization can hurt structured output. Aggressive 3-bit or 4-bit quants sometimes produce malformed JSON arguments before they visibly hurt prose. Validate arguments and log parse failures so you notice.
• Reasoning models (Qwen3 in thinking mode, gpt-oss, DeepSeek-R1 variants) tend to choose tools more carefully but spend many more tokens per step. That matters for latency in a loop.

Memory planning for these sizes is in How much VRAM you need to run an LLM. Remember that agents carry long histories: tool results go back into the context on every step, so budget KV cache for a larger context than plain chat needs.

A minimal agent loop in Python

You do not need a framework to understand or build an agent. The loop below uses the official openai Python package against any OpenAI-compatible server and has one real tool: checking free disk space on an allowlisted set of paths. It is read-only and harmless, which is exactly the kind of tool to start with.

import json
import shutil

from openai import OpenAI

# Ollama: http://localhost:11434/v1   vLLM: http://localhost:8000/v1
client = OpenAI(base_url="http://localhost:11434/v1", api_key="unused")
MODEL = "qwen3:8b"  # on vLLM use the served name, e.g. "Qwen/Qwen3-8B"

ALLOWED_PATHS = {"/", "/var/lib/docker", "/data"}
MAX_STEPS = 6


def disk_usage(path: str) -> dict:
    if path not in ALLOWED_PATHS:
        return {"error": f"path not allowed: {path}"}
    total, used, free = shutil.disk_usage(path)
    gib = 1024 ** 3
    return {
        "path": path,
        "total_gib": round(total / gib, 1),
        "used_gib": round(used / gib, 1),
        "free_gib": round(free / gib, 1),
    }


TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "disk_usage",
            "description": "Return total, used and free disk space in GiB for one mount path.",
            "parameters": {
                "type": "object",
                "properties": {
                    "path": {
                        "type": "string",
                        "enum": sorted(ALLOWED_PATHS),
                        "description": "Mount path to inspect",
                    }
                },
                "required": ["path"],
            },
        },
    }
]

REGISTRY = {"disk_usage": disk_usage}


def run(task: str) -> str:
    messages = [
        {"role": "system", "content": "You are an ops assistant. Use tools to get facts. Be brief."},
        {"role": "user", "content": task},
    ]
    for _ in range(MAX_STEPS):
        resp = client.chat.completions.create(
            model=MODEL, messages=messages, tools=TOOLS, temperature=0
        )
        msg = resp.choices[0].message
        if not msg.tool_calls:
            return msg.content or ""
        messages.append(msg.model_dump(exclude_none=True))
        for call in msg.tool_calls:
            fn = REGISTRY.get(call.function.name)
            try:
                args = json.loads(call.function.arguments or "{}")
                result = fn(**args) if fn else {"error": "unknown tool"}
            except (json.JSONDecodeError, TypeError) as exc:
                result = {"error": f"bad arguments: {exc}"}
            messages.append(
                {"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)}
            )
    return "Stopped: step limit reached."


if __name__ == "__main__":
    print(run("Which of my disks has the least free space?"))

The details that matter more than they look:

• **MAX_STEPS** is your first cost and safety control. Models sometimes loop, calling the same tool with the same arguments. A hard cap turns that into a clean failure.
• Errors go back to the model as data. Returning {"error": ...} lets it correct itself. Raising an exception kills the run.
• The allowlist lives in code, not in the prompt. The enum in the schema helps the model choose valid values, but the check in disk_usage is what actually enforces it. Never rely on the schema or system prompt for security.
• temperature=0 makes tool selection more repeatable, which helps debugging and evals.

MCP (Model Context Protocol) in brief

MCP is an open protocol, introduced by Anthropic in late 2024, that standardizes how an application discovers and calls tools, reads resources and fetches prompt templates from external servers. An MCP server exposes tools with JSON Schema inputs; an MCP client (your agent host) lists them and passes them to the model as ordinary tool definitions. Transports are stdio for local processes and Streamable HTTP for remote servers.

For a self-hosted setup, MCP is useful for two reasons. It lets you reuse existing servers for GitHub, Postgres, file systems and similar, and it separates tool code from agent code so each can be deployed and permissioned on its own. It does not change the model side at all: the LLM still sees a list of functions, so everything above about parsers and model choice still applies.

Treat third-party MCP servers like any other dependency that runs code with your credentials. Read what each tool does, pin versions and run them with the narrowest token you can create. A tool description is text that goes into the model's context, so a malicious or compromised server can inject instructions through it.

Open source AI agent frameworks

| Framework | What it is | Good fit | |---|---|---| | LangGraph | Python and JS library for agents as state graphs, with checkpoints and human-in-the-loop interrupts | Multi-step workflows that need durable state, approval steps or branching | | smolagents | Small Hugging Face library; its CodeAgent writes Python code as actions, ToolCallingAgent uses JSON tool calls | Experiments and research agents; code actions need a sandbox | | Flowise | Visual low-code builder for LLM flows and agents, runs as a Node.js server | Teams that want to prototype agents in a UI and call them over an API | | n8n | Workflow automation platform with AI agent nodes and hundreds of integrations; fair-code license | Agents that mostly glue together business systems, triggers and schedules | | Plain loop | The 50 lines above | One to five tools, a single agent, full control |

All of these accept an OpenAI-compatible base URL, so any of them can sit on top of Ollama or vLLM. Our AI agent framework stack packages Flowise for Docker so you can try the visual route quickly, and the agent category lists the related stacks.

An opinion: start with the plain loop until you can name the specific feature you are missing. Frameworks add abstraction layers that make prompt and tool-call debugging harder, and most agent failures are about tool design and model choice, not orchestration.

Sandboxing and permissions for tools

The model decides which tool to call and with which arguments. Assume that decision can be wrong or hostile, and design every tool so the worst call it can make is acceptable.

• Least privilege per tool. A tool that reads tickets gets a read-only token for the ticket system, not an admin key. Separate read and write tools so you can permission them differently.
• Validate arguments in code. Types, ranges, allowlists of hosts, paths and table names. The JSON Schema guides the model; your validation protects you.
• Human approval for side effects. Sending email, merging code, deleting data or spending money should pause for confirmation. LangGraph interrupts and n8n's wait nodes exist for this; in a plain loop, return a pending status and resume after approval.
• Isolate code execution. If a tool runs model-written code or shell commands, run it in a throwaway container or microVM with no network by default, a read-only root file system, CPU and memory limits, a timeout and no host mounts. Do not run it in the agent process.
• Network egress control. Restrict which domains the agent's tools can reach. This is also your main defense against data being sent out (see below).
• Rate limits and budgets per tool. Cap calls per run and per hour, especially for tools that cost money or hit production systems.

Prompt injection risks

Prompt injection means instructions arriving through content the model reads: a web page, a PDF, an email, a ticket, a tool result or an MCP tool description. OWASP lists it as the top risk in its Top 10 for LLM applications. No current model reliably ignores it, and a better system prompt does not fix it.

Simon Willison's "lethal trifecta" is a useful test. An agent is at serious risk when it combines access to private data, exposure to untrusted content and a way to communicate externally (sending HTTP requests, emails, rendering images from URLs). Remove at least one of the three for any given run. Practical patterns:

• Split agents: one reads untrusted content and produces a constrained summary, another with privileged tools never sees the raw content.
• Block or allowlist outbound URLs, including Markdown image links in rendered answers.
• Require approval for any write or send action triggered in a run that touched untrusted input.
• Log full tool-call arguments so you can spot exfiltration attempts after the fact.

Self-hosting helps with data residency. It does not help with injection at all.

Observability

An agent run is a tree of model calls and tool calls. Log it as such:

• Traces per run: every model request (prompt tokens, completion tokens, latency, model, temperature), every tool call (name, arguments, result size, duration, error) and the final outcome. OpenTelemetry has semantic conventions for generative AI spans, and self-hostable tools such as Langfuse and Arize Phoenix accept that data.
• Server metrics: vLLM exposes Prometheus metrics at /metrics (request queue, KV cache usage, token throughput). Alert on queue depth, not only on errors.
• Evals: keep a set of tasks with expected tool sequences or outcomes and run them on every change of model, prompt, parser or server version. A model upgrade that improves chat can still break tool selection.

Prompts and tool results often contain personal data, so traces need the same retention and access rules as your production data. The GDPR checklist for self-hosted LLMs covers that side.

Cost and latency control

On your own GPUs the cost is capacity, not tokens, but the same levers apply:

• Step count dominates latency. Total time is roughly the sum over steps of prompt processing, generation and tool time. Fewer, more capable tools usually beat many tiny ones.
• Context grows every step. Truncate or summarize large tool results before they go back to the model. Return only the fields it needs, not a full API response.
• Prefix caching helps. The system prompt and tool definitions repeat on every call. vLLM's automatic prefix caching (on by default in recent versions) reuses that computation; keep the static part of the prompt stable and at the start.
• Route by difficulty. A small model can classify the request or handle simple single-tool tasks, and a larger one takes the multi-step cases.
• Cap everything. Max steps, max tokens per response, timeouts per tool and per run.

Next steps

• Pick a server and model, then run the loop above against one read-only tool from your own environment.
• See how others apply agents in the use case library.
• Before connecting write tools, go through the permissions and injection sections again with a colleague and decide which actions need human approval.