OpenAI

在 Home Assistant 中，提供了OpenAI Conversation 集成 

 

This Python project implements a voice assistant that captures the desktop screen, listens for user prompts via speech, and responds with audio output.

It leverages OpenAI’s API for speech-to-text (STT) and text-to-speech (TTS), and uses Google Generative AI (gemini-1.5-flash-latest) for natural language processing (NLP) to generate responses based on the user’s voice input and desktop content.

 

Server:ChatGPT

Integration:OpenAI Conversation Integration

 

OpenAI's ChatGPT	Deepseek's V3	Google Generative AI	Ollama
OpenAI Conversation Integration	Extended OpenAI Conversation

 

How it works

在 Home Assistant 项目中 OpenAI Conversation Integration 的源码中，可以看到调用的完整代码过程：

 

初始化上下文，包括语言、设备 ID、用户输入等信息。

如果配置了 LLM API，获取对应实例，并准备工具列表。

处理对话 ID，如果没有提供，生成一个新的 ULID，记录对话历史。

生成提示文本（prompt），将系统和用户信息传递给 LLM 进行处理。

调用LLM生成回复，处理工具调用（如设备操作）。

保存对话历史，并返回处理结果给用户。

 

step 1.设备列表和tools发送给模型服务OpenAI API

Home Assistant的 OpenAI Conversation发送给模型的Prompt，Prompt 中，包含：

一些除了指示定义的 system 预设输入，比如当前时间；

可用的设备列表（包括设备名称，类型和当前状态）；

模型可以调用的 tools。意图工具（intent tool）列表，当接收用户输入来控制 Home Assistant 时，应该去调用；

这些 tools，模型可以生成用于这些tool函数的输入参数的 JSON 格式。

 

在 OpenAI Conversation 配置中，默认的模板如下，在与 OpenAI API 进行交互时，它会作为 Prompt 的一部分发送给模型服务：

This smart home is controlled by Home Assistant.

An overview of the areas and the devices in this smart home:
{%- for area in areas() %}
  {%- set area_info = namespace(printed=false) %}
  {%- for device in area_devices(area) -%}
    {%- if not device_attr(device, "disabled_by") and not device_attr(device, "entry_type") and device_attr(device, "name") %}
      {%- if not area_info.printed %}

{{ area_name(area) }}:
        {%- set area_info.printed = true %}
      {%- endif %}
- {{ device_attr(device, "name") }}{% if device_attr(device, "model") and (device_attr(device, "model") | string) not in (device_attr(device, "name") | string) %} ({{ device_attr(device, "model") }}){% endif %}
    {%- endif %}
  {%- endfor %}
{%- endfor %}

Answer the user's questions about the world truthfully.

If the user wants to control a device, reject the request and suggest using the Home Assistant app.

output

This smart home is controlled by Home Assistant.

An overview of the areas and the devices in this smart home:

Living Room:
- Motion Sensor (AS007)
- Motion Sensor (AS011)
- wifipuffer1c (AV Renderer Device)
- Flood and Rain Sensor 8AD3 (RS1BB(MI))
- esp8266_12f (nodemcuv2)
- 起居室 (HomePod Mini)
- Yeelight ceil21 local by Xiaomi Miot atuo (yeelink.light.ceiling21)
- Living room lamp meross (mss210p 7.0.0)
- esp32s (nodemcu-32s)

Kitchen:
- 聆巨蓝牙Mesh通断器(厨房) (linju.switch.sw0a01)

Bedroom:
- Motion Sensor 8283 (RTCGQ02LM)

客厅:
- IPTV机顶盒 (miir.stb.ir01)
- Aqara LED灯泡 T1（可调色温） (lumi.light.acn014)
- Aqara人体传感器-RTCGQ11LM-1-客厅 (lumi.sensor_motion.aq2)
- 人体传感器T1-RTCGQ12LM-1 (lumi.motion.agl02)
- 无线开关 (lumi.sensor_switch.v2)
- 米家多功能智能网关(新加坡版) (lumi.gateway.mihk01)
- 领普人体存在传感器ES1ZB(MI) (linp.sensor_occupy.hb01)
- 领普水浸及雨水传感器RS1BB-MI (linp.flood.rs1bb)
- 吸顶灯 (yeelink.light.ceiling21)
- Xiaomi 中枢网关 (xiaomi.gateway.hub1)
- 领普人体存在传感器ES1ZB(MI) (linp.sensor_occupy.hb01)
- 领普水浸及雨水传感器RS1BB-MI (linp.flood.rs1bb)
- 吸顶灯 (yeelink.light.ceiling21)
- Xiaomi Home Hub (xiaomi.gateway.hub1)

厨房:
- 直驱 | 米家滚筒洗衣机 10kg (mibx5.washer.v6)
- 小米米家智能墙壁插座CNCZ08CMno01 (zimi.plug.zncz01)
- 聆巨蓝牙Mesh通断器(厨房) (linju.switch.sw0a01)
- 小米人体传感器2rtcgq02lm-厨房 (lumi.motion.bmgl01)
- 小米米家智能墙壁插座CNCZ08CMno01 (zimi.plug.zncz01)
- Mijia Direct Drive Washer 10Kg (mibx5.washer.v6)

阳台:
- aqara人体传感器-阳台 (lumi.sensor_motion.aq2)
- 小米人体传感器RTCGQ01LM (lumi.sensor_motion.v2)
- Aqara门窗传感器E1 (lumi.magnet.acn001)

书房:
- aqara智能墙壁插座 H1（USB版） (lumi.plug.sacn03)
- aqara人体传感器RTCGQ11LM No3-书房 (lumi.sensor_motion.aq2)
- Aqara温湿度传感器-WSDCGQ11LM-1 (lumi.weather.v1)
- 光照传感器 T1-no2 (lumi.sen_ill.agl01)
- aqara人体传感器 P1-RTCGQ14LM (lumi.motion.ac02)
- Aqara人体传感器E1-RTCGQ15LM-1 (lumi.motion.acn001)
- 插座cp2 (cuco.plug.cp2)
- 米家智能插座2 蓝牙网关版 2 (chuangmi.plug.212a01)
- 领普压力有无传感器(PS1BB)-no1 (linp.senpres.ps1bb)
- 小米智能插线板2 5位插孔版xmzncxb01qm (qmi.plug.psv3)
- Yeelight吸顶灯c2001 (yeelink.light.ceil26)
- 小爱音箱Pro (xiaomi.wifispeaker.lx06)
- 小米小爱音箱Play 增强版 (xiaomi.wifispeaker.l05c)
- 空调遥控 (miir.aircondition.ir02)
- Mi Smart Power Plug 2 (Wi-Fi and Bluetooth Gateway) 2 (chuangmi.plug.212a01)
- 小米智能插线板2 5位插孔版xmzncxb01qm (qmi.plug.psv3)
- 插座cp2 (cuco.plug.cp2)
- 领普压力有无传感器(PS1BB)-no1 (linp.senpres.ps1bb)
- 情景Mesh色温灯V2S系列(普瑞) (lemesh.light.wy0c15)
- Mi AI Speaker Pro (xiaomi.wifispeaker.lx06)
- Mi AI Speaker Play Plus (xiaomi.wifispeaker.l05c)
- Air Conditioner Remote Control (miir.aircondition.ir02)
- Yeelight吸顶灯c2001 (yeelink.light.ceil26)

客厅卫生间:
- aqara高精度人体传感器-RTCGQ13LM (lumi.motion.agl04)
- 人体传感器T1-RTCGQ12LM-3 (lumi.motion.agl02)
- V5智能一键开关（Mesh） (giot.switch.v51ksm)
- Philips灯泡 (philips.light.cbulb)
- V5 Intelligent One-Button Switch (Mesh) (giot.switch.v51ksm)
- Philips灯泡 (philips.light.cbulb)

走廊:
- 领普人体传感器HS1BB (linp.motion.h1)
- Yeelight m2筒灯 (yeelink.light.ml1)
- 领普人体传感器HS1BB (linp.motion.h1)

书房2:
- 米家灯泡蓝牙网关版 (yeelink.light.mbulb3)
- 米家夜灯2蓝牙版MJYD02YL-A (yeelink.light.nl1)

主卧卫生间:
- linptech人体存在平板灯(主卧卫生间) (linp.light.lp1bc)
- linptech人体存在平板灯(主卧卫生间) (linp.light.lp1bc)
- gosund  电小酷CP2儿童房 (cuco.plug.cp2d)

主卧:
- 小米门窗传感器2-主卧 (isa.magnet.dw2hl)
- 米家空调伴侣2(主卧) (lumi.acpartner.mcn02)
- 小米人体传感器2rtcgq02lm-主卧 (lumi.motion.bmgl01)
- 米家空调伴侣2(主卧) (lumi.acpartner.mcn02)
- 小米门窗传感器2-主卧 (isa.magnet.dw2hl)
- 小米人体传感器2rtcgq02lm-主卧 (lumi.motion.bmgl01)

儿童房:
- Yeelight LED Bulb 1S（Color） (yeelink.light.color8)
- gosund  电小酷CP2儿童房 (cuco.plug.cp2d)

Main Bedroom:
- Door/Window Sensor 2F70 (MCCGQ02HL)

Living Room toilet:
- Aqara-Hub-M1S-EDB6 (ZHWG15LM)
- Motion Sensor (AS017)
- Motion Sensor (AS011)
- philips_cbulb_f4b8_light local by Xiaomi Miot atuo (philips.light.cbulb)

Book Bedroom:
- Motion Sensor (AS038)
- Motion Sensor (AS045)
- Temperature Sensor (AS008)
- Motion Sensor (AS007)
- RM4 mini by Broadlink in Book Bedroom
- RM3 mini by Broadlink in Living room (RM mini 3)
- Light Detection Sensor (AS015)
- 小爱音箱-6318 (S12)
- Outlet (AP013)
- ESP32 S3 Box 3 2d954c (voice-assistant)
- Yeelight Ceil26 0x1c5c4959 (ceil26)
- Bluetooth Proxy 58d270 (esp32dev)
- Bluetooth Proxy 58d270 (D0:EF:76:58:D2:70) (esp32dev)
- Xiaomi Miot plug 2 (chuangmi.plug.212a01)
- 情景Mesh色温灯V2S系列(普瑞) (lemesh.light.wy0c15)
- aqara空调伴侣(升级版) (lumi.acpartner.v3)

hallway:
- Lightbulb (AL039)
- Contact Sensor (AS043)
- Motion Sensor (MS002)
- Motion Sensor (AS007)
- 135* (DEEBOT X5 PRO)

corridor:
- Yeelight m2筒灯 (yeelink.light.ml1)

Son Bedroom:
- Motion Sensor (AS007)
- Contact Sensor (MS001)

Main Bedroom Toilet:
- Motion Sensor BF13 (RTCGQ02LM)

Washer room:
- Light Detection Sensor (AS015)

默认房间:
- 网关 M1S (lumi.gateway.acn01)
- 光照传感器 T1 (lumi.sen_ill.agl01)

儿子房:
- aqara人体传感器RTCGQ11LM No4-书房2 (lumi.sensor_motion.aq2)
- 小米门窗传感器MCCGQ01LM (lumi.sensor_magnet.v2)

Answer the user's questions about the world truthfully.

If the user wants to control a device, reject the request and suggest using the Home Assistant app.

 

step 2.模型给出调用的 tool 响应（根据用户的输入）

模型给出响应，在 tool_calls 的响应中，包含了要执行的动作，调用的 tool 方法：HassTurnOn({'name': 'Keting Light', 'domain': 'switch'})

function=Function(arguments='{"name": "Keting Light", "domain": "switch"}', name='HassTurnOn')

 

step 3.HA执行并回应

Tool call: HassTurnOn({'name': 'Keting Light', 'domain': 'switch'})

 

Tool response: {'speech': {}, 'response_type': 'action_done', 'data': {'targets': [], 'success'

 

Response ChatCompletion(message=ChatCompletionMessage(content='已经为您打开了客厅的灯。

 

 

source

工作原理

https://tonnie17.github.io/posts/home-assistant-chatgpt/ cn