A self-hosted ChatGPT

July 12, 2026 7 min read

llm local-inference self-hosting self-sovereign rag mcp qdrant open-webui qwen

Intro

Previously I described how I deployed a coding agent running entirely on the home LAN – a spare box, codenamed weebeastie, serving mid-range Qwen3-Coder-30B-A3B model over llama-server, reached from a laptop by Qwen-Code through an SSH tunnel.

I wanted to make the same model usable by e.g. spouse, hence these two additions, both guaranteeing the same self-sovereign posture (loopback-bound, nothing leaves my LAN):

A chat UI for everyone. Open WebUI in a container, so any phone, tablet or laptop browser on the LAN gets a ChatGPT-style window onto the local Qwen – with per-user accounts and per-user memory.
A grounded document assistant. A retrieval-augmented-generation (RAG) server over European Parliament committee documents, built primarily for my spouse’s work.

The interesting design decision is that the RAG isn’t bolted onto the chat, even though Open WebUI comes with some RAG capabilities. Instead it is a shared RAG server spoken to over MCP, so that the same server can be used both by the household chat and my coding agent.

That is because eventually more collections will be indexed in the RAG db - e.g. my notes, household documents, TODO list, vacay plans …

The chat

The whole security story from last time was that the inference port never touches the LAN – llama-server binds 127.0.0.1:8080 and is only reachable over an SSH tunnel. Adding a web UI must not break that.

Open WebUI runs in Docker with network_mode: host. The UI container reaches the model over loopback and exposes its port 3000 to the LAN, making it visible on 192.168.1.22:3000. The llama-server port stays as private as before.

Open WebUI answering from the local Qwen, with the llama-server journal alongside

On the left, the chat at 192.168.1.22:3000 answering from Qwen3-Coder-30B-A3B-Instruct-IQ4_XS; on the right, logs on weebeastie showing the request stream through the model’s slots – ~100 tokens/s of prompt eval, ~19 tokens/s generated.

A few decisions made along the way:

Reboot survival for free. Just a docker container with restart: unless-stopped means the chat auto-returns.
Fixed accounts, signup locked. Three accounts (me as admin, spouse, a shared guest), ENABLE_SIGNUP=false. Separate accounts give separate, per-user memories automatically.
Sandbox gap. The model only turns tokens into tokens; any code it emits runs in the viewer’s browser (WASM/pyodide), not on weebeastie; the container isolates filesystem and processes. The deliberate hole is that host networking removes network isolation, so I keep the dangerous switches (server-side code execution, admin-installed tools) off and apply the cheap hardening (no-new-privileges, cap_drop: ALL).
Only one thing leaves the LAN, and only on consent. Web search via Exa is on, behind an explicit per-query confirmation. Version checks, telemetry and community sharing are all off.

The RAG MCP server

The document assistant is a Rust pipeline: fetch EP committee PDFs from the Open Data Portal, parse, chunk, embed and upsert into a Qdrant vector database, also hosted locally. Each query is then embedded under the identical recipe as at the insert time, top-k passages are pulled, and the local Qwen is asked to answer using only those passages, citing each one, or saying “I don’t know.”

In my first version the RAG was packaged as an OpenAI-compatible model (retrieval and generation coupled together) – a second entry in the chat’s model dropdown that owned the whole loop (embed → Qdrant → ground → call the generator → cited answer). It worked, but it welded retrieval to this one specific generator. So eventually it got re-cast as a shared MCP retrieval server, ep-rag-mcp, attached to the host (the thing running the agent loop), not to the model server.

llama-server is just a text generator with no MCP client. The decide-to-call → dispatch → feed-back cycle lives above the model, in the host. So the two hosts – Open WebUI and Qwen-Code – each hold their own MCP client, both point at the same retrieval server, and both independently use llama-server as their generator. The win is loose coupling: retrieval is implemented once and reused; the generator stays a swappable dropdown in the chat.

One shared RAG-over-MCP server, two hosts, one generator

Only :3000 faces the LAN. :8082 (MCP), :6334 (Qdrant) and :8080 (the model) all stay loopback. ep-rag-mcp is deployed exactly like llama-server – a loopback systemd unit, Restart=always, enabled at boot.

Tool call routing: native tool-calling vs. classify-then-answer

The household model needs a way to decide whether a message even needs the RAG.

Both paths start from the same artefact: a small decision tree expressed as JSON, versioned and auditable, whose node questions are natural language the model evaluates (“Is the user asking a substantive question about the content or positions of EMPL/REGI/IMCO committee documents?”). Two ways to consume it are:

Mode A – native tool-calling. Hand the model the tree as policy plus a search_ep_committee_docs tool, and let it emit a tool_call when it decides to ground. One turn.
Mode B – walk the tree, emit a JSON decision. Ask the model to walk the tree and return a compact {"reached": …, "tool": …, "reason": …} object. The code then reads that JSON and decides. A separate classify turn before generation.

I was not sure if a smaller model like Qwen-30B will be able to reliably answer in Mode B (although prefferable, given one less model turn).

So I just ran both against the same 10 labelled queries – 4 in-domain EP questions, 4 clearly off-topic (including a carbonara recipe), and 2 deliberately hard “generic-EU-fact” borderline cases (“Who is the current President of the Commission?”) that look on-topic but should not hit the document index:

Mode	Routing correct	Format	Note
B – walk tree → JSON decision	10/10	10/10 valid JSON	correct on both hard borderline cases; ~2.5 s per classify (warm)
A – native tool-calling	8/10	10/10 valid tool args	0 missed EP questions; the 2 errors were over-firing on borderline generic-EU questions

The read: Qwen can obey the tree flawlessly when asked to reason through it explicitly (B). Native tool-choice (A) short-circuits that reasoning – it pattern-matches “EU institution → call the EU tool” and skips the walk, over-firing on exactly the borderline questions where restraint matters. This tracks, as the Open WebUI’s own docs warn that small local models are unreliable at native tool-calling, and Qwen-30B sits right at the border of a small and a medium model.

So the household path pays a ~2.5-second classify turn to buy a 10/10 vs 8/10 routing score, and does it deterministically – the model never emits a tool_call in the chat; the code decides from parsed JSON, so the routing is as reliable as the classifier and all the grounding logic lives in one place in Rust (“code is law”). For a spouse asking policy questions, correct-and-a-bit-slower beats fast-and-occasionally-wrong.

Answer and a citation, chunk directly from the Qdrant db:

Factual answer

Where it stands

The box that used to just autocomplete my code now has a face the whole household can talk to from any device, a per-user memory, and a grounded, cited answer path over real EP committee documents – all still behind a single LAN-facing port, with the model itself as private as the day it was set up. Setup, the Rust RAG crates, and the routing spike are in the repository.