Ollama vs LiteLLM vs llmama.cpp vs vvllm vs lm studio

How to run a local LLM server step by step

 

There are multiple inference engines to run AI models on your own hardware: including llama.cpp, vllm, ollama, lm studio. Etc.Each one is built for a different use case.  find the one that fits your use. 

 

Ollama vs llama.cpp vs vLLM — Which Should You Use?

Ollama vs LiteLLM vs llmama.cpp vs vvllm vs lm studio

These tools represent different layers of the AI stack. While they overlap, they generally serve distinct purposes:

Serving (Llama.cpp, vLLM),

Managing (Ollama, LM Studio), 

Routing (LiteLLM).

 

Comparing local AI runtimes

	Ollama	llama.cpp	vLLM
Setup time	2 min	10 min	5 min
GPU required	No	No	Yes
Quantization	Auto	Full control	Limited
API	OpenAI-compat	OpenAI-compat	OpenAI-compat
Best for	Getting started	Tweaking	Production

 

 

Managing (Ollama, LM Studio)

Ollama

A local LLM inference/runtime platform.It handles model downloads, storage, and execution with a simple CLI/API. Think of it as a “local LLM server”.

Run AI Models locally  integrate via API。Ollama is popular but not the fastest.

 

LM Studio

A  desktop application.

Run AI Models locally with a Chat UI

 

Serving (Llama.cpp, vLLM)

llama.cpp - run ai model on edge devices.

Run a model on a Raspberry Pi5 8GB.

 

vLLM

Build a high-traffic AI startup or production API.

 

Routing (LiteLLM)

LiteLLM

LiteLLM is not an inference engine; it is a Proxy/Router.A proxy/gateway layer that provides a unified, OpenAI-compatible API for calling many LLM providers (cloud and local).

 

Modles

There are multiple model:

 

fast local model

larger reasoning model

vision model

cloud models

 

Best models for CPU-only

 

RAM available	Best model	Why
4GB	Qwen3.5-2B	Best quality at this size
8GB	Qwen3.5-4B	Good balance
16GB	Qwen3.5-9B	Best quality-per-resource
32GB	Qwen3.5-27B (Q4)	Strong but slow (~2-4 tok/s)

 

Pi 5 vs other hardware

Device	RAM	Price	Best model	Speed
Raspberry Pi 5 (8GB)	8GB	~$80	Qwen3.5-0.8B	~10 tok/s
Raspberry Pi 5 (16GB)	16GB	~$120	Qwen3.5-35B-A3B	~4.5 tok/s
Mac Mini M4 (16GB)	16GB	$599	Qwen3.5-9B	~35 tok/s
Used mini PC (16GB)	16GB	~$150-200	Qwen3.5-4B	~8-12 tok/s

 

works on a Pi 5

Model	Size	Speed on Pi 5 (8GB)	Speed on Pi 5 (16GB)	Quality
Qwen3.5-0.8B	0.8B	~8-10 tok/s	~10-12 tok/s	Basic tasks, fast
Qwen 2.5 0.5B	0.5B	~12-15 tok/s	~15 tok/s	Simple Q&A
TinyLlama 1.1B	1.1B	~6-8 tok/s	~8-10 tok/s	Decent for size
Qwen3.5-4B (Q2)	4B	Too slow	~3-4 tok/s	Good quality, slow
Qwen3.5-35B-A3B (Q2)	3B active	Too slow	~4.5 tok/s	Surprisingly good

 

Summary

Ollama - Two commands, simplest setup, good for beginners

llama.cpp - More control, CPU-optimized, works on any hardware

Unsloth - Python integration, full customization, best for developers

 

This is my current setup:

 

Gemma 4 runs in llama.cpp server, exposed through the OpenAI-compatible http://192.168.2.125:8091/v1 API.

Home Assistant connects  llama.cpp server  using the Local OpenAI LLM integration .

 

I’m using the same model for both:

Assist Conversation Agent (streaming responses + device control via the Assist API)

AI Task automations (camera analysis, BMS battery summaries, pool sensor analysis, etc.)

 

For vision, Home Assistant converts camera snapshots into OpenAI-style image_url (base64) inputs and sends them straight to Gemma. The llama.cpp container loads both the GGUF and the matching mmproj (configured in the container, not in HA).

 

For STT I’m also using Gemma via a Wyoming endpoint. The flow looks like this:

Voice Satellite → HA Assist → Wyoming STT (Gemma) → transcript → Gemma via llama.cpp → TTS

 

Camera analysis is similarly straightforward:

Camera Snapshot → HA AI Task → llama.cpp (GGUF + mmproj) → Vision Analysis

 

useful links

Gemma 4 model overview

https://ai.google.dev/gemma/docs/core