Self-Hosted Local LLMs with Ollama and Open WebUI

Ollama packages open-weight language models (Llama, Mistral, Gemma, Qwen, and others) so they run locally with a single command, no cloud API key, no per-token billing, and no prompt data ever leaving your machine. Open WebUI puts a familiar chat interface in front of it, so it feels like using a hosted chatbot while everything actually runs on your own hardware.

This is genuinely useful, not just a novelty: local models are smaller and less capable than the largest hosted ones, but modern mid-size open models are strong enough for a real range of everyday tasks -- summarizing, drafting, coding help, answering questions against your own documents -- and doing it privately and for free is a real trade worth making for a lot of use cases.

Step 1: Install Ollama

$ curl -fsSL https://ollama.com/install.sh | sh

This is Ollama's own official installer for Linux; macOS and Windows have native app installers from ollama.com/download instead. It installs Ollama as a background service listening on localhost:11434.

Step 2: Pull and run a model

Pick a model that matches your hardware. As a starting point:

• 8GB RAM or less / no GPU: llama3.2:3b or gemma2:2b -- small, fast, noticeably less capable than larger models, but genuinely usable
• 16GB+ RAM, modest or no GPU: llama3.1:8b or qwen2.5:7b -- the sweet spot for most homelab hardware
• A GPU with 12GB+ VRAM: qwen2.5:14b or larger, for meaningfully better output quality

$ ollama pull llama3.1:8b
$ ollama run llama3.1:8b

ollama run drops you into an interactive prompt directly in the terminal -- useful to confirm the model works before adding a web UI on top. Type /bye to exit.

Step 3: Run Open WebUI

services:
open-webui:
image: ghcr.io/open-webui/open-webui:main
container_name: open-webui
ports:
- "3000:8080"
volumes:
- "./open-webui-data:/app/backend/data"
environment:
OLLAMA_BASE_URL: "http://host.docker.internal:11434"
extra_hosts:
- "host.docker.internal:host-gateway"
restart: unless-stopped

extra_hosts is what lets the container reach Ollama running on the host machine itself, outside Docker -- without it, host.docker.internal won't resolve on Linux. Start it:

$ docker compose up -d

Step 4: Set up Open WebUI

Visit http://<host-ip>:3000. The first account created becomes the admin account. Once logged in, the model you pulled in Step 2 should already be selectable in the chat interface's model dropdown, since Open WebUI talks to the same Ollama instance.

Step 5: (Optional) Add more models as needed

$ ollama pull mistral
$ ollama pull qwen2.5-coder:7b

Each pulled model becomes selectable in Open WebUI's dropdown without any extra configuration -- Open WebUI just reflects whatever Ollama has available.

Verify it works

• ollama list shows the model(s) you pulled
• http://<host-ip>:3000 loads Open WebUI, and you can log in with the admin account you created
• Sending a message in the chat interface gets a real response from the model, not an error
• Switching the model dropdown to a different pulled model and sending another message uses that model instead (confirm via response style, or check Ollama's logs for which model was invoked)

Troubleshooting

Open WebUI loads but shows no models / can't reach Ollama

Almost always the OLLAMA_BASE_URL + extra_hosts combination in Step 3. Confirm curl http://localhost:11434 works directly on the host first (if that fails, the problem is Ollama itself, not Open WebUI); then confirm the container can reach it with docker exec open-webui curl http://host.docker.internal:11434.

Responses are extremely slow

CPU-only inference on a large model is legitimately slow -- this is expected, not broken. Try a smaller model from the Step 2 list first to confirm the pipeline itself works, then decide whether a GPU is worth adding for your actual use case.

Ollama runs out of memory and the response cuts off or fails

The model doesn't fit in available RAM/VRAM. Drop to a smaller model size, or if you're on a supported GPU, confirm Ollama is actually detecting and using it (ollama ps shows whether a running model is on GPU or CPU).

Next steps

Once the basics work, look into Ollama's support for custom system prompts and "Modelfiles" to tune a model's default behavior, and Open WebUI's built-in RAG (Retrieval-Augmented Generation) feature for asking questions against your own uploaded documents rather than just chatting from the model's training data alone.

Sources

Based on the official Ollama documentation and the official Open WebUI documentation.