j.foucher/PS_AI_Agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: j.foucher:ab2482a2099c78d79031f2ff0985c962b35f78af
Branches Tags
j.foucher:main
..
compare: j.foucher:fee7d0c679d716332d81aadbf0109990a97e51a0
Branches Tags
j.foucher:main

2 Commits
ab2482a209 ... fee7d0c679

Author SHA1 Message Date
j.foucher
fee7d0c679 Gaze: stateless head/eyes so client matches server (fix ~60° head offset) 
Single-player clients sometimes showed the head turned ~60° off while the
server looked correct. Two client-only causes:

1. The client overwrote the actor's rotation every frame (SetActorRotation)
   and then read back its own output as the "replicated" yaw, so SmoothedBodyYaw
   diverged from the true server body yaw.
2. The head/eye cascade was sticky / path-dependent (TargetHeadYaw updated on
   overflow events), so running it independently on a client fed by lagged,
   stepped replication latched into a different state — the head overflowed to
   MaxHeadYaw and stayed there.

Rewrite head/eyes as a STATELESS function of the actual (authoritative on the
server, replicated on the client) body facing + target direction. No sticky
state, FInterpTo only eases the visual. The client no longer overrides the actor
rotation. Body turn stays authority-only (lazy turn beyond head+eyes range).
Server and client now converge to the same pose by construction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-25 18:13:53 +02:00
j.foucher
777ffa09a8 ElevenLabs: fix multi-player speaker arbitration with open mic 
With an open mic streaming continuously (Server VAD + interruption), every
chunk refreshed the active speaker's LastSpeakTime, so SpeakerSwitchHysteresis
never elapsed and SpeakerIdleTimeout never fired — the first speaker locked the
floor forever and the agent never turned to anyone else.

Now only chunks with real voice energy (RMS >= 0.02) hold/take the floor: the
active speaker is still forwarded continuously (silence tail for VAD), but their
lock only persists while they actually speak. When they pause, another player's
voiced audio can switch in and the agent turns to them (NetActiveSpeakerPawn
replicated -> gaze). Validated with two players on one NPC.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
 2026-06-25 18:13:44 +02:00
2 changed files with 68 additions and 107 deletions
Expand all files Collapse all files

34
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_ElevenLabsComponent.cpp
View File

@@ -2369,16 +2369,44 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::ServerSendMicAudioFromPlayer(
}
const double Now = FPlatformTime::Seconds();
LastSpeakTime.FindOrAdd(SpeakerPawn) = Now;
// If this player IS the active speaker, forward immediately.
// Real voice energy of this chunk (int16 mono PCM). Only ACTUAL speech may
// hold or take the floor — not the silence-tail or ambient that an open mic
// streams continuously. Otherwise the first speaker's continuous stream keeps
// refreshing LastSpeakTime, so no one else can ever switch in and the agent
// stays locked on them (multi-player gaze/turn-taking broken).
const int32 NumSamples = PCMBytes.Num() / (int32)sizeof(int16);
double SumSq = 0.0;
if (NumSamples > 0)
{
const int16* S = reinterpret_cast<const int16*>(PCMBytes.GetData());
for (int32 i = 0; i < NumSamples; ++i) { const double v = S[i] / 32768.0; SumSq += v * v; }
}
const double Rms = (NumSamples > 0) ? FMath::Sqrt(SumSq / NumSamples) : 0.0;
const bool bVoiced = (Rms >= 0.02); // speech vs ambient/breath; tune if needed.
// Only real voice asserts holding/taking the floor.
if (bVoiced)
{
LastSpeakTime.FindOrAdd(SpeakerPawn) = Now;
}
// The active speaker is always forwarded (keeps the silence tail flowing to
// ElevenLabs VAD for end-of-turn detection).
if (NetActiveSpeakerPawn == SpeakerPawn)
{
WebSocketProxy->SendAudioChunk(PCMBytes);
return;
}
// Speaker switch: only switch if current speaker has been silent
// A different player can only TAKE the floor with actual voice — their
// silence/tail must not steal it from the current speaker.
if (!bVoiced)
{
return;
}
// Speaker switch: only switch if current speaker has been voice-silent
// for at least SpeakerSwitchHysteresis seconds.
bool bCanSwitch = true;
if (NetActiveSpeakerPawn)

139
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_GazeComponent.cpp
View File

@@ -405,11 +405,7 @@ void UPS_AI_ConvAgent_GazeComponent::TickComponent(
const FVector ToTarget = TargetPos - EyeOrigin;
// ── 2. Body: smooth interp toward sticky target ────────────────────
//
// TargetBodyWorldYaw is persistent — only updated when head+eyes
// can't reach the target. Same sticky pattern as TargetHeadYaw.
// ── 1b. Horizontal direction / world yaw to the target ─────────────
const FVector HorizontalDir = FVector(ToTarget.X, ToTarget.Y, 0.0f);
float TargetWorldYaw = 0.0f;
if (!HorizontalDir.IsNearlyZero(1.0f))
@@ -417,11 +413,21 @@ void UPS_AI_ConvAgent_GazeComponent::TickComponent(
TargetWorldYaw = HorizontalDir.Rotation().Yaw;
}
// Body smoothly interpolates toward its persistent target.
// Server/standalone: directly rotates the actor.
// Client: does NOT rotate — accepts replicated rotation from server.
if (bEnableBodyTracking && Owner->HasAuthority())
// ── 2. BODY (authority only): lazy turn ────────────────────────────
// Rotate the actor to face the target only once head+eyes can no longer
// cover it. Clients NEVER rotate the body — they receive it replicated.
// Body turn state (TargetBodyWorldYaw) lives on the authority only.
if (bEnableBodyTracking && Owner->HasAuthority() && !HorizontalDir.IsNearlyZero(1.0f))
{
const float BodyFacingYaw = Owner->GetActorRotation().Yaw + MeshForwardYawOffset;
const float DeltaToTarget = FMath::FindDeltaAngleDegrees(BodyFacingYaw, TargetWorldYaw);
// Commit to facing the target once it leaves the head+eyes range.
if (FMath::Abs(DeltaToTarget) > MaxHeadYaw + MaxEyeHorizontal)
{
TargetBodyWorldYaw = TargetWorldYaw - MeshForwardYawOffset;
}
const float BodyDelta = FMath::FindDeltaAngleDegrees(
Owner->GetActorRotation().Yaw, TargetBodyWorldYaw);
if (FMath::Abs(BodyDelta) > 0.1f)
@@ -431,122 +437,49 @@ void UPS_AI_ConvAgent_GazeComponent::TickComponent(
}
}
// ── Smoothed body yaw for cascade computations ──────────────────
// Server: direct copy (no lag). Client: interpolate toward replicated
// rotation to avoid step artifacts from discrete ~30Hz network updates.
// The client also overrides the actor's rotation with the smoothed value
// so the body mesh doesn't visually jump at network tick rate.
if (Owner->HasAuthority())
// Smoothed mirror of the (authoritative/replicated) body yaw — DEBUG
// only. NEVER written back to the actor: the old client override
// (SetActorRotation) corrupted the head input and made the client head
// diverge by up to MaxHeadYaw. The head reads the real actor yaw below.
{
SmoothedBodyYaw = Owner->GetActorRotation().Yaw;
}
else
{
const float ReplicatedYaw = Owner->GetActorRotation().Yaw;
const float ClientDelta = FMath::FindDeltaAngleDegrees(SmoothedBodyYaw, ReplicatedYaw);
SmoothedBodyYaw += FMath::FInterpTo(0.0f, ClientDelta, SafeDeltaTime, BodyInterpSpeed * 3.0f);
// Override actor rotation with smoothed value so the body mesh
// shows the interpolated rotation instead of the raw replicated steps.
// This is local-only — replication will overwrite on next network update,
// and SmoothedBodyYaw will absorb the correction smoothly.
FRotator SmoothedRot = Owner->GetActorRotation();
SmoothedRot.Yaw = SmoothedBodyYaw;
Owner->SetActorRotation(SmoothedRot);
const float ActorYaw = Owner->GetActorRotation().Yaw;
const float YawDelta = FMath::FindDeltaAngleDegrees(SmoothedBodyYaw, ActorYaw);
SmoothedBodyYaw += FMath::FInterpTo(0.0f, YawDelta, SafeDeltaTime, BodyInterpSpeed * 3.0f);
}
// ── 3. Compute DeltaYaw after body interp ──────────────────────────
// When body tracking is disabled (e.g. proximity gaze during spline patrol),
// sync SmoothedBodyYaw and TargetBodyWorldYaw to current actor rotation.
// This ensures the head offset is always relative to the actual body facing,
// not a stale cached value from a previous frame.
// When body tracking is off, keep the body target synced (debug only).
if (!bEnableBodyTracking)
{
SmoothedBodyYaw = Owner->GetActorRotation().Yaw;
TargetBodyWorldYaw = SmoothedBodyYaw;
TargetBodyWorldYaw = Owner->GetActorRotation().Yaw;
}
// ── 3. HEAD + EYES — STATELESS cascade ─────────────────────────────
// Pure function of the ACTUAL body facing (authoritative on the server,
// replicated on the client) and the target direction. No sticky /
// path-dependent state, so server and client converge to the same pose
// regardless of how the body rotation arrived (smooth vs ~30Hz steps).
// FInterpTo only eases the visual; it never latches.
const float BodyFacingYaw = Owner->GetActorRotation().Yaw + MeshForwardYawOffset;
float DeltaYaw = 0.0f;
if (!HorizontalDir.IsNearlyZero(1.0f))
{
const float CurrentFacingYaw = SmoothedBodyYaw + MeshForwardYawOffset;
DeltaYaw = FMath::FindDeltaAngleDegrees(CurrentFacingYaw, TargetWorldYaw);
DeltaYaw = FMath::FindDeltaAngleDegrees(BodyFacingYaw, TargetWorldYaw);
}
// ── 4. Pitch from 3D direction ─────────────────────────────────────
const float HorizontalDist = HorizontalDir.Size();
const float TargetPitch = (HorizontalDist > 1.0f)
? FMath::RadiansToDegrees(FMath::Atan2(ToTarget.Z, HorizontalDist))
: 0.0f;
// ── 5. BODY overflow: check against persistent TargetBodyWorldYaw ──
//
// Same pattern as head: check from body's TARGET position (not current).
// Body triggers when head+eyes combined range is exceeded.
const float BodyTargetFacing = TargetBodyWorldYaw + MeshForwardYawOffset;
const float DeltaFromBodyTarget = FMath::FindDeltaAngleDegrees(
BodyTargetFacing, TargetWorldYaw);
bool bBodyOverflowed = false;
if (bEnableBodyTracking && FMath::Abs(DeltaFromBodyTarget) > MaxHeadYaw + MaxEyeHorizontal)
{
// Body realigns to face target
TargetBodyWorldYaw = TargetWorldYaw - MeshForwardYawOffset;
// Head returns to ~0° since body will face target directly
TargetHeadYaw = 0.0f;
bBodyOverflowed = true;
}
// ── 6. HEAD: realign when eyes overflow (check against body TARGET) ──
//
// Head overflow is checked relative to where the BODY IS GOING
// (TargetBodyWorldYaw), not where it currently is. This prevents
// the head from overcompensating during body interpolation —
// otherwise the head turns to track while body catches up, then
// snaps back when body arrives (two-step animation artifact).
//
// GUARD: Skip eye overflow check when body just overflowed in this
// same frame. Body overflow already set TargetHeadYaw=0 (head will
// recenter as body turns). Allowing eye overflow to also fire would
// snap TargetHeadYaw to a second value in the same frame, causing
// a visible 1-2 frame jerk as FInterpTo chases two targets.
if (!bBodyOverflowed)
{
const float HeadDeltaYaw = FMath::FindDeltaAngleDegrees(
TargetBodyWorldYaw + MeshForwardYawOffset, TargetWorldYaw);
const float EyeDeltaYaw = HeadDeltaYaw - TargetHeadYaw;
if (FMath::Abs(EyeDeltaYaw) > MaxEyeHorizontal)
{
TargetHeadYaw = FMath::Clamp(HeadDeltaYaw, -MaxHeadYaw, MaxHeadYaw);
}
}
// Head smoothly interpolates toward its persistent target
// Head takes as much of the yaw gap as its range allows; eyes the rest.
// (TargetHeadYaw/Pitch are kept only as debug mirrors — no longer sticky.)
TargetHeadYaw = FMath::Clamp(DeltaYaw, -MaxHeadYaw, MaxHeadYaw);
CurrentHeadYaw = FMath::FInterpTo(CurrentHeadYaw, TargetHeadYaw, SafeDeltaTime, HeadInterpSpeed);
// Eyes = remaining gap (during transients, eyes may sit at MaxEye while
// the head catches up — ARKit normalization keeps visual deflection small)
CurrentEyeYaw = FMath::Clamp(DeltaYaw - CurrentHeadYaw, -MaxEyeHorizontal, MaxEyeHorizontal);
// ── 5. PITCH: relative cascade (Eyes → Head, no body pitch) ────────
// Same pattern: check against persistent TargetHeadPitch
const float EyeDeltaPitch = TargetPitch - TargetHeadPitch;
if (FMath::Abs(EyeDeltaPitch) > MaxEyeVertical)
{
// Eyes overflow → head realigns toward target pitch
TargetHeadPitch = FMath::Clamp(TargetPitch, -MaxHeadPitch, MaxHeadPitch);
}
CurrentHeadPitch = FMath::FInterpTo(CurrentHeadPitch, TargetHeadPitch, SafeDeltaTime, HeadInterpSpeed);
// Eyes = remaining pitch gap
CurrentEyePitch = FMath::Clamp(TargetPitch - CurrentHeadPitch, -MaxEyeVertical, MaxEyeVertical);
}
else