j.foucher/PS_AI_Agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ElevenLabs: fix multi-player speaker arbitration with open mic

Browse Source 
With an open mic streaming continuously (Server VAD + interruption), every
chunk refreshed the active speaker's LastSpeakTime, so SpeakerSwitchHysteresis
never elapsed and SpeakerIdleTimeout never fired — the first speaker locked the
floor forever and the agent never turned to anyone else.

Now only chunks with real voice energy (RMS >= 0.02) hold/take the floor: the
active speaker is still forwarded continuously (silence tail for VAD), but their
lock only persists while they actually speak. When they pause, another player's
voiced audio can switch in and the agent turns to them (NetActiveSpeakerPawn
replicated -> gaze). Validated with two players on one NPC.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
j.foucher
2026-06-25 18:13:44 +02:00
parent ab2482a209
commit 777ffa09a8
1 changed files with 31 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_ElevenLabsComponent.cpp
View File

@@ -2369,16 +2369,44 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::ServerSendMicAudioFromPlayer(
}
const double Now = FPlatformTime::Seconds();
LastSpeakTime.FindOrAdd(SpeakerPawn) = Now;
// If this player IS the active speaker, forward immediately.
// Real voice energy of this chunk (int16 mono PCM). Only ACTUAL speech may
// hold or take the floor — not the silence-tail or ambient that an open mic
// streams continuously. Otherwise the first speaker's continuous stream keeps
// refreshing LastSpeakTime, so no one else can ever switch in and the agent
// stays locked on them (multi-player gaze/turn-taking broken).
const int32 NumSamples = PCMBytes.Num() / (int32)sizeof(int16);
double SumSq = 0.0;
if (NumSamples > 0)
{
const int16* S = reinterpret_cast<const int16*>(PCMBytes.GetData());
for (int32 i = 0; i < NumSamples; ++i) { const double v = S[i] / 32768.0; SumSq += v * v; }
}
const double Rms = (NumSamples > 0) ? FMath::Sqrt(SumSq / NumSamples) : 0.0;
const bool bVoiced = (Rms >= 0.02); // speech vs ambient/breath; tune if needed.
// Only real voice asserts holding/taking the floor.
if (bVoiced)
{
LastSpeakTime.FindOrAdd(SpeakerPawn) = Now;
}
// The active speaker is always forwarded (keeps the silence tail flowing to
// ElevenLabs VAD for end-of-turn detection).
if (NetActiveSpeakerPawn == SpeakerPawn)
{
WebSocketProxy->SendAudioChunk(PCMBytes);
return;
}
// Speaker switch: only switch if current speaker has been silent
// A different player can only TAKE the floor with actual voice — their
// silence/tail must not steal it from the current speaker.
if (!bVoiced)
{
return;
}
// Speaker switch: only switch if current speaker has been voice-silent
// for at least SpeakerSwitchHysteresis seconds.
bool bCanSwitch = true;
if (NetActiveSpeakerPawn)