Run your own ChatGPT: self-hosted LLMs with Docker Compose
I use the big commercial chat assistants every day and they’re genuinely good at what they do. But not everything I’d want to ask belongs on someone else’s servers. Personal data, private notes, half-formed ideas I’m still chewing on: for those I’d rather keep the conversation on my own network. So alongside the cloud tools, I run a local assistant of my own. A small Docker Compose stack gives me a private LLM with a chat UI that looks a lot like the commercial ones, and for that kind of prompt it means no per-token bill and nothing leaving the house.
The whole thing is two services: Ollama to run the models and Open WebUI to talk to them. This is the stack I actually run, why each piece is there, and the bits that trip people up (GPU passthrough being the big one).
What you get
- Ollama is the inference engine. It pulls models, keeps them on disk, and serves them over a simple HTTP API on port
11434. Think of it as the thing that does the thinking. - Open WebUI is the front end. It’s a polished, self-hosted chat interface with conversations, multiple models, user accounts and file uploads. It talks to Ollama over that API, so you get a ChatGPT-style experience pointed entirely at your own hardware.
Two containers, one network, two volumes for persistence. That’s the entire design.
Before you start
You need Docker and Docker Compose, which you almost certainly already have if you run anything in a homelab.
For anything beyond small models you also want an NVIDIA GPU and the NVIDIA Container Toolkit installed on the host. That toolkit is what lets a container see the GPU. Without it, the GPU block in the Compose file below will fail to start, and you’ll want the CPU-only variant instead (I cover that later). You can check the toolkit is working with:
1
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
If that prints your GPU, you’re ready. If it errors, fix the toolkit first, because the LLM stack relies on exactly that plumbing.
The Compose file
Here’s the stack in full. I’ll break down the parts that matter underneath.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
services:
ollama:
image: ollama/ollama
command: serve
expose:
- 11434/tcp
healthcheck:
test: ollama --version || exit 1
volumes:
- ollama:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["all"]
capabilities: [gpu]
webui:
image: ghcr.io/open-webui/open-webui:main
ports:
- 8080:8080/tcp
environment:
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- open-webui:/app/backend/data
depends_on:
- ollama
volumes:
ollama:
open-webui:
Save it as docker-compose.yml and you’re a docker compose up -d away from a running stack. Before you do, a few parts are worth understanding rather than copying blind.
How the two services find each other
Notice OLLAMA_BASE_URL=http://ollama:11434. That ollama is not a hostname I’ve configured anywhere: it’s the service name, and Compose gives every service a DNS entry on the default network for the project. So Open WebUI reaches the inference engine by name, over the internal network, with no host ports or IP addresses involved.
That’s also why Ollama uses expose rather than ports. expose makes 11434 reachable to other containers on the same network but does not publish it to the host. The only thing I actually publish is Open WebUI’s 8080, because that’s the one interface a human needs to reach. Keeping the raw model API off the host is a small but real bit of hygiene, and I’ll come back to why it matters.
GPU passthrough, the part everyone gets stuck on
This block is what hands the GPU to the container:
1
2
3
4
5
6
7
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["all"]
capabilities: [gpu]
A few things worth knowing:
capabilities: [gpu]is required. If you omit it, Docker will not grant GPU access even with the rest of the block present.device_ids: ["all"]gives the container every GPU. If you have more than one and want to pin Ollama to a specific card, swap it for something likedevice_ids: ["0"].- This is the Compose equivalent of
docker run --gpus all. It only works if the NVIDIA Container Toolkit from the previous section is installed. This is where “it won’t start” almost always traces back to.
Get this right and Ollama will load models into VRAM and run them on the GPU, which is the difference between a usable assistant and one you wait on.
Persistence and health
The two named volumes are what stop you re-downloading several gigabytes of model every time a container restarts:
ollama:/root/.ollamakeeps your pulled models.open-webui:/app/backend/datakeeps your accounts, chats and settings.
The healthcheck on Ollama gives Docker a real readiness signal instead of “the process is running, probably”. It’s simple, but it means depends_on and your own tooling can reason about whether the engine is actually up.
Bringing it up
1
2
docker compose up -d
docker compose ps
Open WebUI is now on http://localhost:8080. The first account you create becomes the admin, so do that before you expose it anywhere.
The stack is running, but Ollama has no models yet. Pull one:
1
docker compose exec ollama ollama pull llama3.1:8b
llama3.1:8b is a good general starting point that fits comfortably on an 8 GB card. If you’re tight on VRAM, try a smaller model such as qwen2.5:3b; if you’ve got headroom, pull something larger. Once it’s down, it shows up in the model picker in Open WebUI and you can start chatting. You can also pull models straight from the UI, but doing it once on the command line makes it obvious where the work happens.
No GPU? The CPU-only variant
You don’t need a GPU to try this. Drop the entire deploy: block from the ollama service and everything else stays the same:
1
2
3
4
5
6
7
8
9
ollama:
image: ollama/ollama
command: serve
expose:
- 11434/tcp
healthcheck:
test: ollama --version || exit 1
volumes:
- ollama:/root/.ollama
It will run happily on CPU. Be realistic about it, though: a small model will feel sluggish and a large one will feel broken. CPU-only is great for kicking the tyres and fine for light use with a 3B-class model. For anything you’ll use daily, the GPU is what makes it pleasant.
Don’t put this on the internet naked
Open WebUI has its own authentication, which is good. It is still not something I’d publish straight to the internet on a raw port, and the Ollama API has no auth at all, which is exactly why the Compose file above never publishes 11434 to the host.
If you want to reach this from outside the house, put it behind a reverse proxy that terminates TLS and, ideally, an auth layer in front (I run mine behind an authenticating proxy so there are two doors, not one). The rule I stick to: the only thing that ever gets published is Open WebUI’s port, and even that goes through the proxy rather than straight out. The inference API stays on the internal Docker network where only the UI can see it.
Is running it yourself actually worth it?
Honest answer: it depends on what you want from it.
- Cost. After the hardware you already own, inference is free. No per-token billing, no surprise invoice because a script got chatty. If you tinker constantly, that adds up in your favour.
- Privacy. Nothing leaves your network. For anything sensitive, or just on principle, that’s the whole point. Your prompts are yours.
- Latency. On a decent GPU, first-token latency is genuinely competitive, and there’s no network round trip to a provider.
- Model quality. This is the honest trade-off. The open models you can run at home are very good and getting better fast, but the largest frontier models still have an edge on the hardest tasks. For summarising, drafting, coding help and general questions, local models are more than good enough. For the absolute cutting edge, they’re not there yet.
For me the split is easy. The cloud tools stay in the mix for the heavy, cutting-edge work, and anything personal or private goes to the local model instead, where the traffic never leaves the house. The stack above is what makes routing it that way a five-minute decision rather than a project.
Wrapping up
Two services, one Compose file, and you’ve got a private ChatGPT-alike running on your own hardware. The pieces that matter are small: point Open WebUI at Ollama by its service name, hand the GPU over with the deploy block (and the NVIDIA toolkit behind it), keep your models and chats in named volumes, and never publish the inference API. Get those right and the rest is just picking which models to pull.