How to run a local LLM server step by step

Ollama vs LiteLLM vs llmama.cpp vs vvllm vs lm studio

These tools represent different layers of the AI stack. While they overlap, they generally serve distinct purposes:

Serving (Llama.cpp, vLLM),

Managing (Ollama, LM Studio), 

Routing (LiteLLM).

Table of Contents

 

Managing (Ollama, LM Studio)

Ollama

A local LLM inference/runtime platform.It handles model downloads, storage, and execution with a simple CLI/API. Think of it as a “local LLM server”.

Run AI Models locally  integrate via API

 

LM Studio

A  desktop application.

Run AI Models locally with a Chat UI

 

Serving (Llama.cpp, vLLM)

llama.cpp - run ai model on edge devices.

Run a model on a Raspberry Pi5 8GB.

 

vLLM

Build a high-traffic AI startup or production API.

 

Routing (LiteLLM)

LiteLLM

LiteLLM is not an inference engine; it is a Proxy/Router.A proxy/gateway layer that provides a unified, OpenAI-compatible API for calling many LLM providers (cloud and local).

 

 

This is my current setup:

 

Gemma 4 runs in llama.cpp server, exposed through the OpenAI-compatible http://192.168.2.125:8091/v1 API.

Home Assistant connects  llama.cpp server  using the Local OpenAI LLM integration .

 

I’m using the same model for both:

Assist Conversation Agent (streaming responses + device control via the Assist API)

AI Task automations (camera analysis, BMS battery summaries, pool sensor analysis, etc.)

 

For vision, Home Assistant converts camera snapshots into OpenAI-style image_url (base64) inputs and sends them straight to Gemma. The llama.cpp container loads both the GGUF and the matching mmproj (configured in the container, not in HA).

 

For STT I’m also using Gemma via a Wyoming endpoint. The flow looks like this:

Voice Satellite → HA Assist → Wyoming STT (Gemma) → transcript → Gemma via llama.cpp → TTS

 

Camera analysis is similarly straightforward:

Camera Snapshot → HA AI Task → llama.cpp (GGUF + mmproj) → Vision Analysis

 

Comments


Comments are closed