part 7 - Build Your Own Private AI Station: Llama server with Docker+ Open WebUI with Docker

Published Aug 13, 2026

I am running llama.cpp server with Gemma 4 (E2B) model on an rp55 8GB.

Connecting llama.cpp running Gemma 4 (E2B) to Home Assistant via HA-MCP allows the AI model to perform local, tool-calling smart home control.

RPi5 8GB +docker llama.cpp server + Gemma 4 E2B + docker Home Assistant + ha-mcp-integration(build-in ha-mcp server )+docker Open WebUI.

Open WebUI will begin using your local Llama.cpp server as a backend!

Overview
How it work?
Install
Connections
- Step 4: Connect Llama.cpp server via Open WebUI
- Step 5: Connect ha-mcp server via Open WebUI

Overview

Runs the full ha-mcp server inside Home Assistant.Transport: use Streamable HTTP

Set up your Llama.cpp server with docker

Llama.cpp server Load large models locally

Enable 4GB of ZRAM swap space to prevent hard OOM crashes during heavy context processing.

Integrate with Open WebUI for a seamless interface

#docker images
REPOSITORY                      TAG       IMAGE ID       CREATED         SIZE
ghcr.io/ggml-org/llama.cpp      server    bde659bfc300   8 days ago      1.21GB
ghcr.io/open-webui/open-webui   v0.11.0   72c0ba641ba7   2 weeks ago     7.15GB

How it work?

Architecture Overview

Open WebUI Docker(port 3000)
    ↓ OpenAI-compatible API
llama.cpp server (port 8080) ← Gemma 4 E2B Q4
    ↓ (tool calls via MCP)
ha-mcp (inside HA or add-on)
    ↓
Home Assistant Docker

┌─────────────────┐       MCP       ┌─────────────────┐      REST/WS      ┌─────────────────┐
│ MCP Host Client │ ──────────────> │  HA-MCP Server  │ ────────────────> │ Home Assistant  │
│ (Open WebUI /   │                 │   (:9584)       │                   │   (:8123)       │
│  MCP Agent)     │                 └─────────────────┘                   └─────────────────┘
└────────┬────────┘
         │ OpenAI API
         ▼
┌─────────────────┐
│  llama.cpp      │ (Gemma 4 E2B)
│  (:8080)        │
└─────────────────┘

Note

1.llama.cpp acts as an inference server (provider)

2.HA-MCP acts as an MCP tool provider

3.Open WebUI sends user prompts to llama.cpp server

Install

Step 1: Install Llama.cpp server with docker

Step 2: Download a Supported Model

cd /datadocker/llama-cpp

docker pull ghcr.io/ggml-org/llama.cpp:server-b10499

docker run \
  --name llama \
  --restart=unless-stopped \
  -v /datadocker/llama-cpp/models:/models \
  -p 8091:8080 \
  ghcr.io/ggml-org/llama.cpp:server:server-b10499 \
  -m /models/google_gemma-4-E2B-it-Q4_0.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --threads 4 \
  --jinja \
  --reasoning-budget 0 \
  --chat-template-kwargs '{"enable_thinking": false}' \
  --ui-mcp-proxy

you will get api:

http://192.168.2.134:8091/v1

Step 2: Install Open WebUI with docker

docker pull ghcr.io/open-webui/open-webui:v0.11.0

cd /datadocker/open-webui/

docker run -d -p 3000:8080 --name open-webui \
 -v /datadocker/open-webui/data:/app/backend/data \
 ghcr.io/open-webui/open-webui:v0.11.0


web ui

http://192.168.2.125:3000/

Step 3: Install ha-mcp-integration(build-in ha-mcp server ) in home assistant

first,Install HA-MCP Custom Component integration in ha

then Add HA-MCP Server in HA-MCP Custom Component integration.

you will get api:

http://192.168.2.125:9584/private_QQz2CLU3n8xR

Connections

Step 4: Connect Llama.cpp server via Open WebUI

connect to a AI Provider

Add the API Connection in Open WebUI

Connect Llama.cpp server to Open WebUI

Open WebUI will begin using your local Llama.cpp server as a backend

Open Open WebUI in your browser

Go to ⚙️ Admin Settings → Connections → OpenAI.

Click ➕ Add Connection.

http://192.168.2.134:8091/v1

add http://192.168.2.134:8091/v1,it is llama.cpp with gemma4 model

Step 5: Connect ha-mcp server via Open WebUI

add ha mcp server

Open Open WebUI in your browser

Go to ⚙️ Admin Settings → Tools → Integrations.

Click ➕ Add External Tool Servers.

http://192.168.2.125:9584/private_QQz2CLU3n8xR

Note

you can get Connect URL via HA-MCP Custom Component integration in ha.

http://192.168.2.125:8123/config/integrations/integration/ha_mcp_tools

debug step by step

http://192.168.2.134:8091/v1

cqrs

0.00.013.586 W srv  llama_server: CORS is set to allow all origins ('*') and no API key is set
0.00.013.586 W srv  llama_server: this can be a security risk (cross-origin attacks)
0.00.013.587 W srv  llama_server: more info: https://github.com/ggml-org/llama.cpp/pull/25655

useful links

https://docs.openwebui.com/getting-started/quick-start/connect-a-provider/starting-with-llama-cpp/