how to run locally llama.cpp(llama-server) for home assistant on rpi5

Published Apr 21, 2026

Control your Home Assistant smart home with a completely local Large Language Model.

This is my current setup:

Gemma 4 runs in llama.cpp server, exposed through the OpenAI-compatible http://192.168.2.125:8091/v1 API.

Home Assistant connects llama.cpp server using the Local OpenAI LLM integration .

To run docker llama.cpp locally on a Raspberry Pi 5 for Home Assistant, you need to set up a standalone docker llama.cpp server that provides an API on your Pi's base OS，that Home Assistant can communicate with via an API.

Home Assistant does not have a "llama.cpp" brand integration by default.

Connect Home Assistant to it using a compatible integration. such as https://github.com/skye-harris/hass_local_openai_llm.

20260811 updated

Nabu Casa officially introduced the native llama.cpp integration in the Home Assistant 2026.8 release.

https://www.home-assistant.io/integrations/llama_cpp/

How to Run an LLM->llama.cpp on a Raspberry Pi 5
Download LLM Models
Docker Run llama-server
Test
Connect llama.cpp to Home Assistant
- Integration - Add Integration
Voice assistant - Create conversation agent

How to Run an LLM->llama.cpp on a Raspberry Pi 5

Flash Raspberry Pi OS Lite 64-bit (Bookworm) and increase swap to 4 GB.

Run llama.cpp server (Docker).

Download a Gemma 4 E2B GGUF model (2B parameters) from Hugging Face.

Test interactive inference with llama-cli using all four CPU cores.

Start the docker llama-server to expose an OpenAI-compatible API on port 8080.

Integrate llama-server to Home Assistant by sending JSON requests to the API.

Prerequisites

Docker must be installed and running on your system.

Create a folder to store big models

Device: Raspberry Pi 5 (8GB)

OS: Debian 12

Runtime: Docker

Inference Engine: llama-server

Model: Gemma 4 E2B (GGUF, quantized)

Home assistant Integration: Local OpenAI LLM

RPi5
│
├── Home Assistant (Docker)
├── llama-server (Docker)
├── Open WebUI (Docker)
├── piper (Docker)
├── 
└── whisper (Docker)

In the ghcr.io/ggml-org/llama.cpp repository, the images are split by purpose:

Tag	Primary Contents	Best Use Case
`:light`	`llama-cli`, `llama-completion`	Testing/CLI: Best for running models in the terminal or one-off completions without overhead.
`:server`	`llama-server`	Production/API: Ideal for your Home Assistant setup. It provides the OpenAI-compatible endpoint.
`:full`	CLI, Server, and Python conversion/quantization tools.	Development: Use this if you need to convert `.safetensors` to `.gguf` or quantize a model yourself.

:light: Contains only llama-cli and llama-completion.It does not contain the API server.

:server: Contains only llama-server.It contain the API server not contain llama-cli.A lightweight, OpenAI API compatible, HTTP server for serving LLMs.

:full: Contains everything.

You should use the :server tag (or better yet, the :server-arm64 tag since you are on a Raspberry Pi 5).

Download LLM Models

Docker Run llama-server

The llama-server executable acts as an OpenAI-compatible API that Home Assistant can use.

docker run -it --rm \
  --name llama \
  -v /datadocker/llama-cpp/models:/models \
  -p 8091:8080 \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/google_gemma-4-E2B-it-Q4_0.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --threads 4 \
  --jinja

Parameter Description

--entrypoint /app/llama-cli: processes your input (or waits for one), and then exits. It does not listen for network requests on a port.

--entrypoint /app/llama-server: You were using llama-cli, which is for one-off prompts in the terminal. The llama-server is required to handle API calls like curl.

--host 0.0.0.0: Inside a Docker container, the server must listen on 0.0.0.0 to accept connections from your Raspberry Pi's IP or localhost.

--port 8080: This tells the software inside the container to listen on port 8080 (which you mapped to 8091 on your host).

--jinja: support for OpenAI-style function calling.Tool calling must be enabled in inference engine.Detail

output

...
srv          init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv  update_slots: all slots are idle

Now llama.cpp = local LLM → HTTP API server

Test llama-server

Cli

Once the server logs show "HTTP server listening", run your curl command. Make sure to include a JSON body, otherwise the server might reject the request:

curl http://localhost:8091/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "Hello Gemma!"}]
  }'

web ui

http://192.168.2.125:8091/

Connect llama.cpp to Home Assistant

You can link Home Assistant directly to a local llama.cpp server endpoint.

Integration - Add Integration

Custom Integration - Local OpenAI LLM Integration

https://github.com/skye-harris/hass_local_openai_llm

Wyoming-LLM (The Bridge): A Home Assistant Integration that sits between HA and llama.cpp.

Custom Integration - Configure Integration

Added server URL to the initial server configuration

http://192.168.2.125:8091

Voice assistant - Create conversation agent

Add assistant

To start using the model to turn on lights, adjust thermostats, or ask queries, tie it to your Assist pipeline.

useful links

Get llama.cpp installed on your machine and Run with Docker Offcial Docker documentation

https://github.com/ggml-org/llama.cpp/blob/master/docs/docker.md