feat(discord): voice-channel mixer — ambient idle bed + verbal acks that overlap TTS (#39659)

* feat(discord): voice-channel mixer — ambient idle bed + verbal acks that overlap TTS

Discord voice mode can now feel conversational: the bot speaks a short
acknowledgement before it starts working, and a subtle ambient 'thinking' bed
plays underneath while tools run, ducking under speech and swelling back — the
Grok-voice-mode feel.

discord.py plays only one audio stream per voice connection, so this adds a
software mixer (VoiceMixer, a discord.AudioSource) installed once per guild on
join. It sums an ambient loop, verbal acks, and TTS replies into that single
20ms/48kHz/stereo stream (numpy int16 add + clip), so they overlap instead of
stop-and-swap. Speech ducks the ambient gain down and releases it smoothly.

- plugins/platforms/discord/voice_mixer.py: VoiceMixer + MixerChild (gain,
  loop, fade, duck/release), decode_to_pcm (ffmpeg), synth_ambient_pcm (no
  asset needed — synthesised pad).
- adapter: install mixer on join, tear down on leave, route
  play_in_voice_channel through the mixer (legacy one-shot path kept as
  fallback), play_ack_in_voice, voice_mixer_active. Defensive getattr for the
  object.__new__ test helpers.
- gateway/run.py: tool_start_callback fires a one-time verbal ack on the first
  tool call of a turn when in a voice channel (independent of the text
  tool-progress gate). No system-prompt or message-flow changes.
- config: discord.voice_fx.* (OFF by default; ambient/duck/speech gains, ack
  phrases). All in config.yaml, not .env.
- docs + tests (mixer unit + adapter integration).

Verified: 19 new tests pass, existing voice suite green (2 pre-existing
davey-module env failures unchanged), and a real-mixer E2E confirms ambient
streams, TTS overlaps it, acks layer in, and teardown is clean.

* fix(discord): make voice mixer numpy import lazy (numpy is voice-extra-only)

numpy ships in the optional 'voice' extra, not [all,dev], so a module-level
'import numpy' broke CI test collection (and would break the always-imported
Discord adapter on any install without the voice extra). Defer numpy to the
functions that actually mix audio via _require_numpy(); guard the test module
with pytest.importorskip('numpy').
This commit is contained in:
Teknium
2026-06-05 03:10:40 -07:00
committed by GitHub
parent 3da44dbda7
commit 8a9ded5b21
6 changed files with 952 additions and 1 deletions
+22
View File
@@ -1841,6 +1841,28 @@ DEFAULT_CONFIG = {
# real memory cost. Default 32 MiB matches the historical hardcoded
# cap. Set to 0 for no cap. Env override: DISCORD_MAX_ATTACHMENT_BYTES.
"max_attachment_bytes": 33554432,
# Voice-channel audio effects (the continuous mixer). OFF by default.
# When enabled, the bot installs a software mixer on the outgoing voice
# stream so a low ambient "thinking" bed, verbal acknowledgements, and
# TTS replies can OVERLAP (ducking the ambient under speech) instead of
# stop-and-swap — the Grok-voice-mode feel. discord.py ships no mixer;
# this is implemented in plugins/platforms/discord/voice_mixer.py.
"voice_fx": {
"enabled": False, # master switch for the mixer subsystem
"ambient_enabled": True, # play the idle "thinking" bed while tools run
"ambient_path": "", # custom loop audio file; "" = synthesised pad
"ambient_gain": 0.18, # idle bed loudness, 0.01.0
"duck_gain": 0.06, # ambient loudness while speech plays
"speech_gain": 1.0, # TTS / ack loudness, 0.01.0
"ack_enabled": True, # speak a short phrase before the first tool call
"ack_phrases": [ # picked at random; set [] to disable phrases
"Let me look into that.",
"One moment.",
"Checking on that now.",
"Give me a sec.",
"On it.",
],
},
},
# WhatsApp platform settings (gateway mode)