run-local-llm-server-part6 debian vmware on windows 10

Published Aug 5, 2026

Llama.cpp is a high-performance inference engine written in C/C++ designed for running Llama and compatible models in the GGUF format.

vmware on windows 10

Overview

Set up your Llama.cpp server

Load large models locally

docker pull ghcr.io/ggml-org/llama.cpp:server-b10499
docker pull ghcr.io/open-webui/open-webui:v0.11.0

Step 1: Download a model in GGUF format from Hugging Face.

# llama.cpp only works with GGUF

# Create a dedicated folder for models:
mkdir -p /datadocker/llama-cpp

cd /datadocker/llama-cpp

mkdir models

cd /datadocker/llama-cpp/models


https://huggingface.co/unsloth/gemma-4-E2B-it-GGUF/tree/main

https://huggingface.co/bartowski/google_gemma-4-E2B-it-GGUF/tree/main

google_gemma-4-E2B-it-Q4_0.gguf

wget https://huggingface.co/bartowski/google_gemma-4-E2B-it-GGUF/resolve/main/google_gemma-4-E2B-it-Q4_0.gguf?download=true

Step 2: Install Llama.cpp server with docker

cd /datadocker/llama-cpp

docker run \
  --name llama \
  --restart=unless-stopped \
  -v /datadocker/llama-cpp/models:/models \
  -p 8091:8080 \
  ghcr.io/ggml-org/llama.cpp:server \
  -m /models/google_gemma-4-E2B-it-Q4_0.gguf \
  --host 0.0.0.0 \
  --port 8080 \
  --threads 4 \
  --jinja \
  --reasoning-budget 0 \
  --chat-template-kwargs '{"enable_thinking": false}' \
  --webui-mcp-proxy

Step 4: Connect Llama.cpp server via WebUI

Overview

Comments