Self-Hosted Local LLMs with Ollama and Open WebUI

Run open-weight language models entirely on your own hardware, with a ChatGPT-style web interface, no API key, no per-token billing, and no prompt data leaving your network.

By DaemonPress

What you'll build and why

Ollama packages open-weight language models (Llama, Qwen, Gemma, and others) so they run locally with a single command -- no cloud API key, no per-token billing, and no prompt data leaving your machine. Open WebUI puts a familiar chat interface in front of it, so using it feels like a hosted chatbot while everything actually runs on hardware you control.

Don't build this if: you need the absolute best available model quality for demanding tasks -- local models, even large ones, generally trail the largest hosted commercial models in capability, and if that gap matters for your use case, a hosted API is the better tool. This is worth building specifically when private/offline inference, cost, or just wanting to run your own infrastructure matters more than having the single best model available.

How it works

  You (browser)
│
▼
Open WebUI  --  chat interface, conversation history, user accounts
│
│  API calls
▼
Ollama  --  loads the model, runs inference (CPU or GPU),
returns the response
│
▼
Model files on disk (downloaded once via "ollama pull",
reused for every conversation after that)

Nothing here calls out to the internet once a model is downloaded -- inference happens entirely on your hardware. The model file itself is just weights (numbers), not something that "phones home" or needs a live connection to function.

Before you start

Decision: which model size fits your hardware? Model families and specific recommended releases change every few months as new ones (Llama, Qwen, Gemma, and others each release new generations regularly) come out -- treat the specific model names below as a snapshot, not a permanent list, and check ollama.com/library for what's currently popular in each size class when you actually do this:

RAM available Model size class Notes
8GB or less, no GPU ~1-4B parameters Fast, noticeably less capable, still genuinely usable for simple tasks
16GB, modest or no GPU ~7-8B parameters The sweet spot for most homelab hardware
A GPU with 12GB+ VRAM ~13-14B+ parameters Meaningfully better output quality, needs the VRAM to actually benefit

Steps

Step 1: Install Ollama

$ curl -fsSL https://ollama.com/install.sh | sh

Ollama's own official Linux installer; macOS and Windows have native app installers from ollama.com/download instead. Installs Ollama as a background service listening on localhost:11434.

Step 2: Pull and run a model

Pick a model matching your hardware from the Before you start table. As concrete current starting points (check ollama.com/library for what's current when you do this):

$ ollama pull llama3.1:8b
$ ollama run llama3.1:8b

ollama run drops you into an interactive terminal prompt -- useful to confirm the model works before adding a web UI on top. Type /bye to exit.

Step 3: Run Open WebUI

services:
open-webui:
image: ghcr.io/open-webui/open-webui:v0.11.3
container_name: open-webui
ports:
- "3000:8080"
volumes:
- "./open-webui-data:/app/backend/data"
environment:
OLLAMA_BASE_URL: "http://host.docker.internal:11434"
extra_hosts:
- "host.docker.internal:host-gateway"
restart: unless-stopped

Pinned to v0.11.3 (current as of this guide) rather than :main/:latest, so an upstream update doesn't silently change your running version underneath you. extra_hosts is what lets the container reach Ollama running on the host itself, outside Docker -- without it, host.docker.internal won't resolve on Linux.

$ docker compose up -d

Step 4: Set up Open WebUI

Visit http://<host-ip>:3000. The first account created becomes the admin account. The model you pulled in Step 2 should already be selectable in the chat interface's model dropdown, since Open WebUI talks to the same Ollama instance.

Step 5 (optional): Add more models

$ ollama pull <another-model>:<size>

Each pulled model becomes selectable in Open WebUI's dropdown automatically, no extra configuration.

Verify it works

  • ollama list shows the model(s) you pulled
  • http://<host-ip>:3000 loads Open WebUI, and you can log in with the admin account you created
  • Sending a message gets a real response from the model, not an error
  • Switching the model dropdown to a different pulled model and sending another message uses that model (confirm via response style, or docker logs open-webui / Ollama's own logs for which model was invoked)
  • Failure test: stop Ollama (sudo systemctl stop ollama on Linux, or quit the app on macOS/Windows). Confirm Open WebUI shows a clear connection error rather than hanging silently -- confirms you'll actually notice if the underlying service goes down, not just get a confusing blank response

Secure it

  • What's exposed: Ollama's API on port 11434 has no authentication by default -- anyone who can reach that port can use your models and consume your compute. This guide assumes LAN-only access; don't port-forward 11434 or 3000 to the internet.
  • Open WebUI's own accounts: the first account created is the admin. Open WebUI's own settings control whether new signups are open or require admin approval -- check Admin Panel → Settings → General if you don't want anyone on your network able to self-register an account.
  • Default credentials: none -- the first account you create during Step 4 is the only one, with the password you set then.
  • Update strategy: Ollama ships frequent releases (roughly weekly at times) -- check github.com/ollama/ollama/releases periodically; Open WebUI similarly at github.com/open-webui/open-webui/releases.
  • The failure that hurts most: treating local inference as a privacy guarantee for data you paste into it if you've also exposed the service beyond your own network -- the model itself doesn't call home, but an exposed, unauthenticated API on your network is still a real access-control gap if anyone else can reach it.

Back it up and maintain it

What matters: Open WebUI's ./open-webui-data directory holds accounts, chat history, and settings -- back it up if any of that matters to you. Downloaded model files (managed by Ollama, typically under ~/.ollama/models or the container-equivalent path) are large but fully re-downloadable from the model library -- no need to back those up, just re-pull if lost.

Update cadence: both projects move fast; update deliberately (bump the pinned version, read release notes) rather than tracking :latest silently.

What to monitor: disk space -- models are large (multiple GB each), and it's easy to accumulate several before realizing how much space they use (ollama list shows what's installed; ollama rm <model> removes one).

Troubleshooting

Logs: docker logs open-webui for the web UI; journalctl -u ollama (Linux) or Ollama's own app logs (macOS/Windows) for the inference engine itself.

Symptom Likely cause Diagnostic Fix
Open WebUI loads but shows no models / can't reach Ollama OLLAMA_BASE_URL + extra_hosts misconfigured (Step 3) curl http://localhost:11434 on the host first (isolates whether Ollama itself is the problem); docker exec open-webui curl http://host.docker.internal:11434 Fix the compose file's OLLAMA_BASE_URL/extra_hosts pairing
Responses are extremely slow CPU-only inference on a model too large for comfortable CPU speed -- expected, not broken Try a smaller model from the Before you start table to confirm the pipeline itself works Drop to a smaller model, or add GPU acceleration if the workload justifies it
Ollama runs out of memory and a response cuts off or fails The model doesn't fit in available RAM/VRAM ollama ps shows whether a running model is on GPU or CPU, and roughly how much it's using Drop to a smaller model size
A pulled model doesn't show up in Open WebUI's dropdown Open WebUI's connection to Ollama is stale, or the pull happened on a different Ollama instance than the one it's pointed at ollama list on the host running Ollama to confirm the model is actually there Refresh Open WebUI's model list (often a simple page reload); confirm both are pointed at the same Ollama instance if you're running more than one
GPU acceleration doesn't seem to be used even though you have one Ollama doesn't have GPU access, or drivers aren't correctly installed on the host ollama ps while a model is running -- shows CPU vs GPU Confirm GPU drivers are installed and working outside Ollama first (e.g. nvidia-smi for Nvidia), then re-check
Open WebUI is reachable by more people on your network than you expected Open signup is enabled by default in some configurations Admin Panel → Settings → General, check the signup policy Disable open signup, or require admin approval for new accounts

Undo

docker compose down in Open WebUI's directory removes the container; delete ./open-webui-data to remove all its state. ollama rm <model> removes a specific downloaded model; uninstalling Ollama itself follows its own OS-specific uninstall process (see ollama.com/download for your platform).

Go further

  • Explore Ollama's support for custom system prompts and "Modelfiles" to tune a model's default behavior for a specific use case
  • Open WebUI's built-in RAG (Retrieval-Augmented Generation) feature lets you ask questions against your own uploaded documents rather than only chatting from the model's training data
  • Check ollama.com/library periodically -- this is a genuinely fast-moving space, and a model that's a good fit for your hardware today may have a meaningfully better same-size successor in a few months

Resources

Official documentation:

Source and releases:

Community:

  • Ollama Discord -- linked from the Ollama project
  • r/LocalLLaMA -- active, not Ollama-specific, broadly useful for local-inference questions

Go deeper:

Last verified: 2026-09-21, checked against official Ollama/Open WebUI documentation and current release listings (see Resources above).

Changelog:

  • 2026-09-21 -- Full rewrite: pinned every software version to a specific current release (Ollama 0.34.2, Open WebUI v0.11.3) instead of latest, the clearest violation of the editorial standard's versioning rule in the original. Updated model recommendations against the current Ollama library (Llama 4, Qwen 3, and Gemma 4 now exist where the original referenced only Llama 3.x/Qwen 2.5-era models) while explicitly framing the specific names as a snapshot that goes stale, not a permanent list -- this is a genuinely fast-moving ecosystem, and the durable guidance is the size-class table, not specific model names. Added an architecture diagram, a "don't build this if," Secure it (Ollama's API has no auth by default, and Open WebUI's own signup policy is worth checking), Back it up and maintain it, Undo, and a full Resources section. Expanded Troubleshooting from 3 to 6. Moved from development to self-hosted-services.
  • 2026-09-21 (earlier) -- Original version published with every software version pinned to latest.
LLMsSelf-hostingOpen sourceDocker & containers

Related projects