Voice in Home Assistant

I’m using the ESP32-S3-BOX-3B microphone.Home assistant docker version 2024.10 on a dedicated x64 PC.

Table of Contents

If you want an offline voice assistant in HomeAssistant, you would usually install the wyoming-whisper, wyoming-piper and wyoming-openwakeword addons.

Since on RPI HomeAssistant runs in a container rather than directly on a host, you can’t install addons inside of HA. Instead you have to run these service somewhere else and then connect them to HA over the network.

 

Let me tell a bit more about my journey so far…

I have Home Assistant docker set up on rpi hardware

I have Home Assistant Voice Assistant ESP32-S3-BOX-3B set up

I have whisper and piper set up on a on Separate Server.

 

How do I tell Home Assistant to make use of whisper and piper on that other server?

To get started, you'll need to have already set up an ip and port using wyoming integration.

Connecting to your vacuum is performed in two steps:

Connect to the wyoming-whisper API  to Home Assistant docker  via wyoming integration.

It should ask you for a host and a port now. Enter the IP address of your device into the host-field and enter port in the port-field.

Once the session has started the wyoming-whisper will fire a ready event. At this point you can request information from your Home Assistant or send actions to it.

Wyoming - name of protocol. Accordingly, all components are added to the Wyoming integration with this name.

 

 

So I tried to do that by creating 3 custom service from those images:

docker image of the Whisper, Piper, and OpenWakeWord

https://hub.docker.com/r/rhasspy/wyoming-whisper

https://hub.docker.com/r/rhasspy/wyoming-piper

https://hub.docker.com/r/rhasspy/wyoming-openwakeword

 

 

 

which is working fine and I can successfully add them to HomeAssistant using the Wyoming-Protocol Integration.

I’m using the ESP32-S3-BOX-3B microphone .Rhasspy’s wyoming-piper and wyoming-faster-whisper can be installed in several different ways. The easiest way is with Docker, which will pull a 0.3-0.8GB image.

 

This is the entire solution if you are interested. 

 

 

Conversation agent:Home Assistant

Speech-to-text:wyoming-faster-whisper

Text-to-speech:wyoming-piper 

WakeWord: On device

I have flashed my ESP32-S3-BOX-3B with an offcial firmware . I believe this is the same situation like you. The ESP32-S3-BOX-3B will act as a voice satelite. Here are what I have:

 

Hardware

I’ve used the following components for my Voice assistant:

Home Assistant running on a Raspberry 3B+

Whisper, Piper, and OpenWakeWord container runing on a vps on vultr  

A voice assistant  needs hardware.such as  ESP32-S3-BOX3B

Software

Docker containers：

homeassistant container

Whisper, Piper, and OpenWakeWord container

Esphome container

Intergrations:

wyoming integration

connect to the Whisper, Piper, and OpenWakeWord container

esphome integration

 

 

Quick start

Here are the steps I took in order to get this to work:

 

Step 1. we need to install Whisper and Piper as docker containers.

Open xshell and run the following command:

Install wyoming-piper and wyoming-faster-whisper

Step 2.Run Whisper, Piper, and OpenWakeWord on external server -> external voice services

Run a single-container speech-to-text service on Docker

Run a single-container text-to-speech service on Docker

Step 3.Connect it to home assistant core via  wyoming integration -> connects external voice services to Home Assistant.

Open Home Assistant go to Settings → Integrations → Add Integration → Wyoming Protocol

It should ask you for a host and a port now. Enter the IP address of your device into the host-field and enter port in the port-field.

 

If you install Piper and Whisper as docker containers, you need to set the Host to the IP of the machine running the container, and then Port to 10200 (or whatever you have set for Piper) to add Piper. Then repeat the operation but setting Port to 10300 for Whisper. Mine looks like this:

 

 

Step 4.Exposing your devices to Assist.

 

 

 

 

Control HA with your voice without any cloud integration

I am running Home Assistant Core using the docker container, Is there any way that we HA-core peasants to run the add-ons for Whisper and Piper？

I have been using Raspberry Pi Zero, 3A and 3B models with Rhasspy as voice assistant satellites, and decided now is the time to swap to HA Voice Assist.

https://github.com/rhasspy/piper

or

https://github.com/rhasspy/wyoming-faster-whisper

https://github.com/rhasspy/wyoming-piper  

 

 

 

The easier way is to use official docker image and you can run it on another host if you want, just specify the right IP address and ports when you will configure the integrations.

Im running Home Assistant in a Docker container behind a reverse proxy. I’ve installed external Piper and Whisper containers (rhasspy/wyoming-whisper, rhasspy/wyoming-piper) and connected them through up the wyoming protocol integration. 

Outline:

 

Verify your OS settings

 

Install Docker on guest system

Install Whisper and Piper on guest system

 

Install Whisper and Piper integration on host system

Pass the Whisper and Piper from the guest system

 

Part 1 - Local Voice in Home Assistant core

https://blog.matterxiaomi.com/blog/voice-homeassistant/

 

Part 2 - Run whisper on external server

https://blog.matterxiaomi.com/blog/run-whisper-on-external-server/

 

Part 3 - Run piper on external server

https://blog.matterxiaomi.com/blog/run-piper-on-external-server/

 

Part 4 - Run wakeword on external server

https://blog.matterxiaomi.com/blog/run-wakeword-on-external-server/

 

 

 

How it work

 

Voice-Home-Assistant-1.png

 

1.wakeword detection

a ESP32 with I2S Mic using ESPHome and successfully detect wakeword.

Assist Hardware Passing(streaming) the wav audio data to stt.

2.

 

5.intent recognition and intent execution

 

8.text to speech

 

Speech Processingvoice-pipeline-1.png

https://www.home-assistant.io/images/blog/2024-12-voice-chapter-8/voice-pipeline.png

source:https://www.home-assistant.io/blog/2024/12/19/voice-chapter-8-assist-in-the-home/

 

 

conversation(会话组件)

 会话组件允许您与Home Assistant交谈。

Speech-Processing-3-2.png

 

Speech-Processing-3-2.png

 

step by step

step 1.Run Whisper, Piper, and OpenWakeWord on external server

Run Whisper, Piper, and OpenWakeWord with Docker.Pull the container,change path to your path and run it.

step 2.Connect it to home assistant core via  wyoming integration

 

Just use the ha core  and setup the wyoming integration. For host I selected localhost and for port  pointed at port10200 and 10300

Whisper and piper arre recognized in home assistant.

detail:https://www.matterxiaomi.com/boards/topic/15742/local-voice-functionality-in-home-assistant-corehow-to-manually-install-piper-and-whisper-on-home-as#22680

The Wyoming integration connects external voice services to Home Assistant using a small protocol.

http://localhost:4999/boards/topic/14594/voice-in-home-assistant#22634

 

Note

Note that these images are only supported on 64bit architectures

I’ve tries with Raspberry 3B+ and Raspberry OS (32bit) and seems images are only for ARM64.I can not pulling on a rpi3 b+ because it cannot find the arm version.

The issue is that the standard rpi OS is in 32 bit and the images for piper and whisper are 64 bit.

https://www.matterxiaomi.com/boards/topic/15742/local-voice-functionality-in-home-assistant-corehow-to-manually-install-piper-and-whisper-on-home-as#22675

 

Additional info - 

mkdir /root/whisper-data
# Piper
docker run -d -p 10200:10200 \
  -v /root/piper-data:/data \
  --restart unless-stopped \
  rhasspy/wyoming-piper \
  --voice en_US-lessac-medium

# Whisper
docker run -d -p 10300:10300 \
  -v /root/whisper-data:/data \
  --restart unless-stopped \
  rhasspy/wyoming-whisper \
  --model tiny-int8 --language en

# openWakeWord
docker run -d -p 10400:10400 \
  --restart unless-stopped \
  rhasspy/wyoming-openwakeword \
  --preload-model 'ok_nabu'

 

Additional info -Install Option

Rhasspy’s wyoming-piper and wyoming-faster-whisper can be installed in several different ways. The easiest way is with Docker, which will pull a 0.3-0.8GB image

When install piper in your network you have three  options

Option 1.Install wyoming-piper and wyoming-faster-whisper locally with Python virtual environment detail

pip install git+https://github.com/openai/whisper.git

Option 2.Install piper locally (and also whisper), with docker detail

Install piper locally (and also whisper), with add-on detail

Option 3.build from source detail

 

How to improve voice recognition

1.You have to change the model itself

The bigger the model, the better the recognition, however more resources are required and it may take longer to respond.

After I changed from Tiny to Medium, for example Chinese is sooo much better recognized

 

Useful links

https://www.matterxiaomi.com/boards/topic/15742/local-voice-functionality-in-home-assistant-corehow-to-manually-install-piper-and-whisper-on-home-as

source:https://developers.home-assistant.io/docs/voice/overview/

 

https://community.home-assistant.io/t/how-to-manually-install-piper/568040

 

https://www.home-assistant.io/blog/2024/06/26/voice-chapter-7/

 

https://github.com/rhasspy/wyoming-addons

Comments


Comments are closed