part 7 - Build Your Own Private AI Station: Llama server with Docker+ Open WebUI with Docker

Open WebUI will begin using your local Llama.cpp server as a backend!

Overview

Set up your Llama.cpp server with docker

Llama.cpp server Load large models locally

Integrate with Open WebUI for a seamless interface

#docker images
REPOSITORY                      TAG       IMAGE ID       CREATED         SIZE
ghcr.io/ggml-org/llama.cpp      server    bde659bfc300   8 days ago      1.21GB
ghcr.io/open-webui/open-webui   v0.11.0   72c0ba641ba7   2 weeks ago     7.15GB

 

Step 1: Install Llama.cpp

 

Step 2: Download a Supported Model

 

Step 3: Serve the Model with Llama.cpp

docker run -it --rm \
  --name llama \
  -v /datadocker/llama-cpp/models:/models \
  -p 8091:8080 \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/google_gemma-4-E2B-it-Q4_0.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --threads 4 \
  --jinja

 

web ui url:

http://192.168.2.134:8091/

 

 

Step 4: connect to a AI  Provider

Add the API Connection in Open WebUI

Connect Llama.cpp server to Open WebUI

Open WebUI will begin using your local Llama.cpp server as a backend

Open Open WebUI in your browser

Go to ⚙️ Admin Settings → Connections → OpenAI.

Click ➕ Add Connection.

 

http://192.168.2.134:8091/v1

add http://192.168.2.134:8091/v1,it is llama.cpp with gemma4 model

 

 

useful links

 

https://docs.openwebui.com/getting-started/quick-start/connect-a-provider/starting-with-llama-cpp/

 

add ha mcp server

https://homeassistant-ai.github.io/ha-mcp/setup/?method=ha-component&client=open-webui&scope=local