Gaze: stateless head/eyes so client matches server (fix ~60° head offset)

Single-player clients sometimes showed the head turned ~60° off while the
server looked correct. Two client-only causes:

1. The client overwrote the actor's rotation every frame (SetActorRotation)
   and then read back its own output as the "replicated" yaw, so SmoothedBodyYaw
   diverged from the true server body yaw.
2. The head/eye cascade was sticky / path-dependent (TargetHeadYaw updated on
   overflow events), so running it independently on a client fed by lagged,
   stepped replication latched into a different state — the head overflowed to
   MaxHeadYaw and stayed there.

Rewrite head/eyes as a STATELESS function of the actual (authoritative on the
server, replicated on the client) body facing + target direction. No sticky
state, FInterpTo only eases the visual. The client no longer overrides the actor
rotation. Body turn stays authority-only (lazy turn beyond head+eyes range).
Server and client now converge to the same pose by construction.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_ElevenLabsComponent.cpp

@@ -2369,16 +2369,44 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::ServerSendMicAudioFromPlayer(
 	}
 
 	const double Now = FPlatformTime::Seconds();
-	LastSpeakTime.FindOrAdd(SpeakerPawn) = Now;
 
-	// If this player IS the active speaker, forward immediately.
+	// Real voice energy of this chunk (int16 mono PCM). Only ACTUAL speech may
+	// hold or take the floor — not the silence-tail or ambient that an open mic
+	// streams continuously. Otherwise the first speaker's continuous stream keeps
+	// refreshing LastSpeakTime, so no one else can ever switch in and the agent
+	// stays locked on them (multi-player gaze/turn-taking broken).
+	const int32 NumSamples = PCMBytes.Num() / (int32)sizeof(int16);
+	double SumSq = 0.0;
+	if (NumSamples > 0)
+	{
+		const int16* S = reinterpret_cast<const int16*>(PCMBytes.GetData());
+		for (int32 i = 0; i < NumSamples; ++i) { const double v = S[i] / 32768.0; SumSq += v * v; }
+	}
+	const double Rms = (NumSamples > 0) ? FMath::Sqrt(SumSq / NumSamples) : 0.0;
+	const bool bVoiced = (Rms >= 0.02); // speech vs ambient/breath; tune if needed.
+
+	// Only real voice asserts holding/taking the floor.
+	if (bVoiced)
+	{
+		LastSpeakTime.FindOrAdd(SpeakerPawn) = Now;
+	}
+
+	// The active speaker is always forwarded (keeps the silence tail flowing to
+	// ElevenLabs VAD for end-of-turn detection).
 	if (NetActiveSpeakerPawn == SpeakerPawn)
 	{
 		WebSocketProxy->SendAudioChunk(PCMBytes);
 		return;
 	}
 
-	// Speaker switch: only switch if current speaker has been silent
+	// A different player can only TAKE the floor with actual voice — their
+	// silence/tail must not steal it from the current speaker.
+	if (!bVoiced)
+	{
+		return;
+	}
+
+	// Speaker switch: only switch if current speaker has been voice-silent
 	// for at least SpeakerSwitchHysteresis seconds.
 	bool bCanSwitch = true;
 	if (NetActiveSpeakerPawn)

Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_GazeComponent.cpp

@@ -405,11 +405,7 @@ void UPS_AI_ConvAgent_GazeComponent::TickComponent(
 
 		const FVector ToTarget = TargetPos - EyeOrigin;
 
-		// ── 2. Body: smooth interp toward sticky target ────────────────────
+		// ── 1b. Horizontal direction / world yaw to the target ─────────────
-		//
-		// TargetBodyWorldYaw is persistent — only updated when head+eyes
-		// can't reach the target. Same sticky pattern as TargetHeadYaw.
-
 		const FVector HorizontalDir = FVector(ToTarget.X, ToTarget.Y, 0.0f);
 		float TargetWorldYaw = 0.0f;
 		if (!HorizontalDir.IsNearlyZero(1.0f))
@@ -417,11 +413,21 @@ void UPS_AI_ConvAgent_GazeComponent::TickComponent(
 			TargetWorldYaw = HorizontalDir.Rotation().Yaw;
 		}
 
-		// Body smoothly interpolates toward its persistent target.
+		// ── 2. BODY (authority only): lazy turn ────────────────────────────
-		// Server/standalone: directly rotates the actor.
+		// Rotate the actor to face the target only once head+eyes can no longer
-		// Client: does NOT rotate — accepts replicated rotation from server.
+		// cover it. Clients NEVER rotate the body — they receive it replicated.
-		if (bEnableBodyTracking && Owner->HasAuthority())
+		// Body turn state (TargetBodyWorldYaw) lives on the authority only.
+		if (bEnableBodyTracking && Owner->HasAuthority() && !HorizontalDir.IsNearlyZero(1.0f))
 		{
+			const float BodyFacingYaw = Owner->GetActorRotation().Yaw + MeshForwardYawOffset;
+			const float DeltaToTarget = FMath::FindDeltaAngleDegrees(BodyFacingYaw, TargetWorldYaw);
+
+			// Commit to facing the target once it leaves the head+eyes range.
+			if (FMath::Abs(DeltaToTarget) > MaxHeadYaw + MaxEyeHorizontal)
+			{
+				TargetBodyWorldYaw = TargetWorldYaw - MeshForwardYawOffset;
+			}
+
 			const float BodyDelta = FMath::FindDeltaAngleDegrees(
 				Owner->GetActorRotation().Yaw, TargetBodyWorldYaw);
 			if (FMath::Abs(BodyDelta) > 0.1f)
@@ -431,122 +437,49 @@ void UPS_AI_ConvAgent_GazeComponent::TickComponent(
 			}
 		}
 
-		// ── Smoothed body yaw for cascade computations ──────────────────
+		// Smoothed mirror of the (authoritative/replicated) body yaw — DEBUG
-		// Server: direct copy (no lag). Client: interpolate toward replicated
+		// only. NEVER written back to the actor: the old client override
-		// rotation to avoid step artifacts from discrete ~30Hz network updates.
+		// (SetActorRotation) corrupted the head input and made the client head
-		// The client also overrides the actor's rotation with the smoothed value
+		// diverge by up to MaxHeadYaw. The head reads the real actor yaw below.
-		// so the body mesh doesn't visually jump at network tick rate.
-		if (Owner->HasAuthority())
 		{
-			SmoothedBodyYaw = Owner->GetActorRotation().Yaw;
+			const float ActorYaw = Owner->GetActorRotation().Yaw;
-		}
+			const float YawDelta = FMath::FindDeltaAngleDegrees(SmoothedBodyYaw, ActorYaw);
-		else
+			SmoothedBodyYaw += FMath::FInterpTo(0.0f, YawDelta, SafeDeltaTime, BodyInterpSpeed * 3.0f);
-		{
-			const float ReplicatedYaw = Owner->GetActorRotation().Yaw;
-			const float ClientDelta = FMath::FindDeltaAngleDegrees(SmoothedBodyYaw, ReplicatedYaw);
-			SmoothedBodyYaw += FMath::FInterpTo(0.0f, ClientDelta, SafeDeltaTime, BodyInterpSpeed * 3.0f);
-
-			// Override actor rotation with smoothed value so the body mesh
-			// shows the interpolated rotation instead of the raw replicated steps.
-			// This is local-only — replication will overwrite on next network update,
-			// and SmoothedBodyYaw will absorb the correction smoothly.
-			FRotator SmoothedRot = Owner->GetActorRotation();
-			SmoothedRot.Yaw = SmoothedBodyYaw;
-			Owner->SetActorRotation(SmoothedRot);
 		}
 
-		// ── 3. Compute DeltaYaw after body interp ──────────────────────────
+		// When body tracking is off, keep the body target synced (debug only).
-		// When body tracking is disabled (e.g. proximity gaze during spline patrol),
-		// sync SmoothedBodyYaw and TargetBodyWorldYaw to current actor rotation.
-		// This ensures the head offset is always relative to the actual body facing,
-		// not a stale cached value from a previous frame.
 		if (!bEnableBodyTracking)
 		{
-			SmoothedBodyYaw = Owner->GetActorRotation().Yaw;
+			TargetBodyWorldYaw = Owner->GetActorRotation().Yaw;
-			TargetBodyWorldYaw = SmoothedBodyYaw;
 		}
 
+		// ── 3. HEAD + EYES — STATELESS cascade ─────────────────────────────
+		// Pure function of the ACTUAL body facing (authoritative on the server,
+		// replicated on the client) and the target direction. No sticky /
+		// path-dependent state, so server and client converge to the same pose
+		// regardless of how the body rotation arrived (smooth vs ~30Hz steps).
+		// FInterpTo only eases the visual; it never latches.
+		const float BodyFacingYaw = Owner->GetActorRotation().Yaw + MeshForwardYawOffset;
+
 		float DeltaYaw = 0.0f;
 		if (!HorizontalDir.IsNearlyZero(1.0f))
 		{
-			const float CurrentFacingYaw = SmoothedBodyYaw + MeshForwardYawOffset;
+			DeltaYaw = FMath::FindDeltaAngleDegrees(BodyFacingYaw, TargetWorldYaw);
-			DeltaYaw = FMath::FindDeltaAngleDegrees(CurrentFacingYaw, TargetWorldYaw);
 		}
 
-		// ── 4. Pitch from 3D direction ─────────────────────────────────────
-
 		const float HorizontalDist = HorizontalDir.Size();
 		const float TargetPitch = (HorizontalDist > 1.0f)
 			? FMath::RadiansToDegrees(FMath::Atan2(ToTarget.Z, HorizontalDist))
 			: 0.0f;
 
-		// ── 5. BODY overflow: check against persistent TargetBodyWorldYaw ──
+		// Head takes as much of the yaw gap as its range allows; eyes the rest.
-		//
+		// (TargetHeadYaw/Pitch are kept only as debug mirrors — no longer sticky.)
-		// Same pattern as head: check from body's TARGET position (not current).
+		TargetHeadYaw = FMath::Clamp(DeltaYaw, -MaxHeadYaw, MaxHeadYaw);
-		// Body triggers when head+eyes combined range is exceeded.
-
-		const float BodyTargetFacing = TargetBodyWorldYaw + MeshForwardYawOffset;
-		const float DeltaFromBodyTarget = FMath::FindDeltaAngleDegrees(
-			BodyTargetFacing, TargetWorldYaw);
-
-		bool bBodyOverflowed = false;
-		if (bEnableBodyTracking && FMath::Abs(DeltaFromBodyTarget) > MaxHeadYaw + MaxEyeHorizontal)
-		{
-			// Body realigns to face target
-			TargetBodyWorldYaw = TargetWorldYaw - MeshForwardYawOffset;
-			// Head returns to ~0° since body will face target directly
-			TargetHeadYaw = 0.0f;
-			bBodyOverflowed = true;
-		}
-
-		// ── 6. HEAD: realign when eyes overflow (check against body TARGET) ──
-		//
-		// Head overflow is checked relative to where the BODY IS GOING
-		// (TargetBodyWorldYaw), not where it currently is. This prevents
-		// the head from overcompensating during body interpolation —
-		// otherwise the head turns to track while body catches up, then
-		// snaps back when body arrives (two-step animation artifact).
-		//
-		// GUARD: Skip eye overflow check when body just overflowed in this
-		// same frame. Body overflow already set TargetHeadYaw=0 (head will
-		// recenter as body turns). Allowing eye overflow to also fire would
-		// snap TargetHeadYaw to a second value in the same frame, causing
-		// a visible 1-2 frame jerk as FInterpTo chases two targets.
-
-		if (!bBodyOverflowed)
-		{
-			const float HeadDeltaYaw = FMath::FindDeltaAngleDegrees(
-				TargetBodyWorldYaw + MeshForwardYawOffset, TargetWorldYaw);
-
-			const float EyeDeltaYaw = HeadDeltaYaw - TargetHeadYaw;
-
-			if (FMath::Abs(EyeDeltaYaw) > MaxEyeHorizontal)
-			{
-				TargetHeadYaw = FMath::Clamp(HeadDeltaYaw, -MaxHeadYaw, MaxHeadYaw);
-			}
-		}
-
-		// Head smoothly interpolates toward its persistent target
 		CurrentHeadYaw = FMath::FInterpTo(CurrentHeadYaw, TargetHeadYaw, SafeDeltaTime, HeadInterpSpeed);
-
-		// Eyes = remaining gap (during transients, eyes may sit at MaxEye while
-		// the head catches up — ARKit normalization keeps visual deflection small)
 		CurrentEyeYaw = FMath::Clamp(DeltaYaw - CurrentHeadYaw, -MaxEyeHorizontal, MaxEyeHorizontal);
 
-		// ── 5. PITCH: relative cascade (Eyes → Head, no body pitch) ────────
+		TargetHeadPitch = FMath::Clamp(TargetPitch, -MaxHeadPitch, MaxHeadPitch);
-
-		// Same pattern: check against persistent TargetHeadPitch
-		const float EyeDeltaPitch = TargetPitch - TargetHeadPitch;
-
-		if (FMath::Abs(EyeDeltaPitch) > MaxEyeVertical)
-		{
-			// Eyes overflow → head realigns toward target pitch
-			TargetHeadPitch = FMath::Clamp(TargetPitch, -MaxHeadPitch, MaxHeadPitch);
-		}
-
 		CurrentHeadPitch = FMath::FInterpTo(CurrentHeadPitch, TargetHeadPitch, SafeDeltaTime, HeadInterpSpeed);
-
-		// Eyes = remaining pitch gap
 		CurrentEyePitch = FMath::Clamp(TargetPitch - CurrentHeadPitch, -MaxEyeVertical, MaxEyeVertical);
 	}
 	else