Conversation lock no longer prevents switching to a different agent.
When in an active conversation, the player can look at another nearby
agent for ConversationSwitchDelay seconds (default 1s) to switch.
Looking at empty space keeps the current agent selected (no deselect).
Works in multiplayer — each player has independent switch tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add TurnEagerness (Eager/Normal/Patient) and bSpeculativeTurn to agent config
data asset, sent as conversation_config_override at WebSocket connection time
- Add adaptive pre-buffer system: measures inter-chunk TTS timing and decreases
pre-buffer when chunks arrive fast enough (decrease-only, resets each conversation)
- New UPROPERTY: bAdaptivePreBuffer toggle, AudioPreBufferMs as starting/worst-case value
- Rework latency HUD: TTS+Net, PreBuf actual/target with trend indicator, Gen>Ear,
WS Ping, server region display
- Fetch ElevenLabs server region from REST API x-region header
- Add editor Detail Customization: TurnEagerness dropdown + SpeculativeTurn checkbox
in AgentConfig with LLM picker and Language picker
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
user_transcript arrives AFTER agent_response_started in Server VAD mode
(the server detects end of speech via VAD, starts generating immediately,
and STT completes later). This caused Transcript>Gen to show stale values
(19s) and Total < Gen>Audio (impossible).
Now all metrics are anchored to GenerationStartTime (agent_response_started),
which is the closest client-side proxy for "user stopped speaking":
- Gen>Audio: generation start → first audio chunk (LLM + TTS)
- Pre-buffer: wait before playback
- Gen>Ear: generation start → playback starts (user-perceived)
Removed STTToGenMs, TotalMs, EndToEarMs, UserSpeechMs (all depended on
unreliable timestamps). Simpler, always correct, 3 clear metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous approach used TurnEndTime (from StopListening) which was never
set in Server VAD mode. Now all latency measurements are anchored to
TurnStartTime, captured when the first user_transcript arrives from the
ElevenLabs server — the earliest client-side confirmation of user speech.
Timeline: [user speaks] → STT → user_transcript(=T0) → agent_response_started → audio → playback
Metrics shown:
- Transcript>Gen: T0 → generation start (LLM think time)
- Gen>Audio: generation start → first audio chunk (LLM + TTS)
- Total: T0 → first audio chunk (full pipeline)
- Pre-buffer: wait before playback
- End-to-Ear: T0 → playback starts (user-perceived)
Removed UserSpeechMs (unmeasurable without client-side VAD).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TurnStartTime was only set in StartListening(), which is called once.
In Server VAD + interruption mode the mic stays open, so TurnStartTime
was never updated between turns. Now reset TurnStartTime when the agent
stops speaking (normal end + interruption), marking the start of the
next potential user turn.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move latency reset from StopListening to HandleAgentResponseStarted.
In Server VAD + interruption mode, StopListening is never called so
TurnEndTime stayed at 0 and all dependent metrics showed ---. Now
HandleAgentResponseStarted detects whether StopListening provided a
fresh TurnEndTime; if not (Server VAD), it uses Now as approximation.
Also fix DisplayTime from 0 to 1s to prevent HUD flicker.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add bDebugLatency property + CVar (ps.ai.ConvAgent.Debug.Latency)
independent from bDebug to save HUD space
- Reset latencies to zero each turn (StopListening) instead of persisting
- Add UserSpeechMs and PreBufferMs to the latency struct
- Move latency captures outside bDebug guard (always measured)
- Replace single-line latency in DrawDebugHUD with dedicated DrawLatencyHUD()
showing 6 metrics on separate lines with color coding
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Track 4 latencies per conversation turn (computed only when bDebug is active):
- STT→Gen: user stops talking → server starts generating
- Gen→Audio: server generating → first audio chunk received
- Total: user stops talking → first audio chunk (end-to-end)
- End-to-Ear: user stops talking → audio playback starts (includes pre-buffer)
New timestamps: GenerationStartTime (HandleAgentResponseStarted),
PlaybackStartTime (3 OnAudioPlaybackStarted sites). Values persist on
HUD between turns, reset when new turn starts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. StartConversationWithSelectedAgent: remove early return when WebSocket
is already connected (persistent mode). Always call ServerJoinConversation
so the pawn is added to NetConnectedPawns and bNetIsConversing is set.
2. ServerSendMicAudioFromPlayer: bypass speaker arbitration in standalone
mode (<=1 connected pawn). Send audio directly to avoid silent drops
caused by pawn not being in NetConnectedPawns array. Add warning logs
for multi-player drops to aid debugging.
3. OnMicrophoneDataCaptured: restore direct WebSocketProxy->SendAudioChunk
on the server path. This callback runs on the WASAPI audio thread —
accessing game-thread state (NetConnectedPawns, LastSpeakTime) was
causing undefined behavior. Internal mic is always the local player,
no speaker arbitration needed.
4. StopListening flush: send directly to WebSocket (active speaker already
established, no arbitration needed for the tail of the current turn).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GetCurrentBlendshapes() was copying CurrentBlendshapes on the anim worker
thread while the game thread mutated it (TSet::UnhashElements crash).
Use a snapshot pattern: game thread copies to ThreadSafeBlendshapes under
FCriticalSection at end of TickComponent, anim node reads the snapshot.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace exclusive single-player agent lock with shared multi-player model:
- NetConversatingPawn/Player → NetConnectedPawns array + NetActiveSpeakerPawn
- Server-side speaker arbitration with hysteresis (0.3s) prevents gaze ping-pong
- Speaker idle timeout (3.0s) clears active speaker after silence
- Agent gaze follows the active speaker via replicated OnRep_ActiveSpeaker
- New ServerJoinConversation/ServerLeaveConversation RPCs (idempotent join/leave)
- Backward-compatible: old ServerRequest/Release delegate to new Join/Leave
- InteractionComponent no longer skips occupied agents
- DrawDebugHUD shows connected player count and active speaker
- All mic audio paths (FeedExternalAudio, OnMicCapture, StopListening flush) route
through speaker arbitration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- InteractionComponent: GetPawnViewPoint() now prefers UCameraComponent
over PlayerController::GetPlayerViewPoint() — fixes agent selection
in VR where the pawn root stays at spawn while the HMD moves freely
- GazeComponent: ResolveTargetPosition() uses camera first for locally
controlled pawns (VR HMD / FPS eye position), falls back to bone
chain for NPCs and remote players in multiplayer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix DisplayTime=0.0f causing flicker on all debug HUDs (now 1.0f)
- Add per-component CVars (ps.ai.ConvAgent.Debug.*) for console debug toggle
- Add MicrophoneCapture debug HUD with VU meter (RMS/peak/dB bar)
- InteractionComponent reuses existing MicrophoneCaptureComponent on pawn
instead of always creating a new one
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Sync body animation with actual audio playback via new OnAudioPlaybackStarted
delegate instead of OnAgentStartedSpeaking (accounts for pre-buffer delay)
- Fix stale pre-buffer broadcasts by cancelling bPreBuffering on silence detection
and guarding pre-buffer timeout with bAgentSpeaking check
- Smooth body crossfade using FInterpTo instead of linear interpolation
- Add conversation lock in EvaluateBestAgent: keep agent selected during active
conversation regardless of view cone (distance-only check prevents deselect
flicker on fast camera turns)
- Broadcast OnAgentDisconnected in persistent session EndConversation so all
expression components (body, facial, lip sync, gaze) properly deactivate
when the player leaves the interaction zone
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace FInterpConstantTo with FInterpTo (exponential ease-out) in all 4 anim components
- Apply SmoothStep to crossfade alphas for smooth animation transitions
- Add SpeechBlendAlpha to LipSync: fades out mouth curves when not speaking,
letting FacialExpression emotion curves (including mouth) show through
- Remove LipSync AnimNode grace period zeroing to avoid overwriting emotion curves
- Revert BodyExpression to override mode (additive broken with full-pose anims)
- Default ExcludeBones = neck_01 to prevent Gaze/Posture conflicts
- Fix mid-crossfade animation pop in BodyExpression SwitchToNewAnim
- Add Neutral emotion fallback in Body and Facial expression components
- Add SendTextToSelectedAgent convenience method on InteractionComponent
- Add debug HUD display for BodyExpression component
- Update CoreRedirects for Posture → Gaze rename
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add BodyExpression system: emotion-driven body animations with per-emotion
anim lists (Idle/Normal/Medium/Extreme), random selection, auto-cycle on
loop complete, crossfade transitions, upper-body-only or full-body mode
- Replace bExternalMicManagement with auto-detecting ShouldUseExternalMic()
- Add bAutoTargetEyes to GazeComponent: auto-aim at target's eye bones
(MetaHuman), with fallback chain (eyes > head > FallbackEyeHeight)
- Hide bActive from Details panel on all 4 anim components (read-only,
code-managed): FacialExpression, Gaze, LipSync, BodyExpression
- Remove misleading mh_arkit_mapping_pose warning from LipSync
- Add bUpperBodyOnly toggle to BodyExpression AnimNode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three issues prevented posture re-activation on re-entry:
- StartConversation skipped bNetIsConversing/NetConversatingPawn in standalone
(guarded by NM_Standalone check) so ApplyConversationPosture always deactivated
- SetSelectedAgent required !IsConnected() to auto-start conversation, but in
persistent mode the WS stays connected → StartConversation never called on re-entry
- AttachPostureTarget never set Posture->bActive = true, relying solely on
ApplyConversationPosture which could be skipped due to the above condition
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When true (default), the first StartConversation() opens the WebSocket
and EndConversation() only stops the mic/posture — the WebSocket stays
open until EndPlay. The agent remembers the full conversation context.
When false, each Start/EndConversation opens/closes the WebSocket (previous behavior).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Make bAgentGenerating, bWaitingForAgentResponse, bWaitingForResponse,
bFirstAudioResponseLogged, bAgentResponseStartedFired (std::atomic<bool>)
and LastInterruptEventId (std::atomic<int32>) for thread-safety
- Add WebSocket auto-reconnection with exponential backoff (1s→30s cap,
max 5 attempts), distinguishing intentional vs unexpected disconnects
- Add FillCurrentEyeCurves() zero-allocation method using FindOrAdd()
to eliminate per-frame TMap heap allocation in anim thread
- Replace AudioQueue RemoveAt(0,N) with read-offset pattern — O(1) per
underflow callback, periodic compaction when offset > half buffer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix only smoothed cascade inputs (head/eyes) via SmoothedBodyYaw
but the body mesh still followed the raw replicated actor rotation which
jumped at ~30Hz network rate. Now the client calls SetActorRotation with
the smoothed yaw so the mesh visually interpolates. Replication overwrites
on next network update; SmoothedBodyYaw absorbs the correction smoothly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Server-only AddActorWorldRotation (HasAuthority guard) prevents
client/server tug-of-war. Client interpolates toward replicated
rotation via SmoothedBodyYaw (angle-aware FInterpTo at 3x body
speed) to eliminate step artifacts from ~30Hz network updates.
Cascade (DeltaYaw, head, eyes) uses SmoothedBodyYaw on all machines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The pelvis bone rotation approach didn't work in practice. Reverts to
the previous AddActorWorldRotation() body tracking (replicated actor
rotation). Thread-safety fix and deprecated IsValid() removal are kept.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- FacialExpressionComponent: add FCriticalSection around emotion curves
to prevent race between TickComponent (game thread) and anim worker
thread — root cause of EXCEPTION_ACCESS_VIOLATION at Evaluate_AnyThread
- Remove deprecated FBlendedCurve::IsValid() guards (UE 5.5: always true)
- Body tracking: replace AddActorWorldRotation() with animation-only
pelvis bone rotation via AnimNode — eliminates replication tug-of-war
that caused client-side saccades when server overwrote local rotation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create OpusEncoder on ALL machines (was Authority-only) — clients now
encode mic audio, server decodes it; server still encodes agent audio
- FeedExternalAudio / OnMicrophoneDataCaptured: Opus-encode accumulated
PCM buffer before sending via ServerRelayMicAudio RPC on client path
(~200 bytes/100ms instead of 3200 bytes = ~16 Kbits/s vs 256 Kbits/s)
- ServerRelayMicAudio_Implementation: auto-detect Opus (size < raw chunk)
and decode back to PCM before forwarding to WebSocket
- Add public DecompressMicAudio() helper for clean API access from
InteractionComponent relay without exposing private Opus members
- Graceful fallback: if Opus unavailable, raw PCM is sent/received as before
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Attach AudioPlaybackComponent to owner's root component for proper
3D world positioning (was unattached = stuck at origin)
- Enable default inline spatialization with 15m falloff distance when
no external SoundAttenuation asset is set
- External SoundAttenuation asset still overrides the default if set
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-end conversation on deselection: when bAutoStartConversation is
true and the player walks out of range, EndConversation() is called
so the NPC becomes available for other players
- Server-side posture: add ApplyConversationPosture() helper called
from ServerRequestConversation, ServerReleaseConversation,
EndConversation (Authority path), and HandleDisconnected — fixes
NPC not tracking the client on the listen server (OnRep never fires
on Authority)
- Guard DetachPostureTarget: only clear TargetActor if it matches our
pawn, preventing the server IC from overwriting posture set by the
conversation system for a remote client
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Silence gate: skip sending silent mic audio over network RPCs on clients
(~256 Kbits/s saved when not speaking, fixes chaotic teleporting)
- Lazy init: defer InteractionComponent mic creation from BeginPlay to
TickComponent with IsLocallyControlled guard (fixes "No owning connection"
from server-side replicas of remote pawns)
- Body tracking: use bNetIsConversing as fallback for IsConnected() on
clients where WebSocket doesn't exist
- EvaluateBestAgent: null-check NetConversatingPawn before comparison
- MicCaptureComponent: use TWeakObjectPtr in AsyncTask lambda to prevent
FMRSWRecursiveAccessDetector race on component destruction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
In a listen server, the server-side copy of a remote client's pawn also
has an InteractionComponent that ticks. This caused a race condition:
the server-side tick would start conversations using GetFirstPlayerController()
(= server's PC), setting NetConversatingPawn to the server's pawn instead
of the client's. The client's relay RPC arrived too late and was rejected
because bNetIsConversing was already true.
Fix: disable tick and skip mic creation in BeginPlay for non-locally-controlled
pawns. The client handles all interaction locally via relay RPCs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 clients cannot call Server RPCs on actors they don't own. NPC actors
are server-owned, causing "No owning connection" errors when remote clients
try to start conversations, send mic audio, or interrupt agents.
Solution: relay all client→NPC RPCs through the InteractionComponent on
the player's pawn (which IS owned by the client). The relay forwards
commands to the NPC's ElevenLabsComponent on the server side.
Changes:
- InteractionComponent: add Server relay RPCs (Start/End conversation,
mic audio, text message, interrupt) and Client relay RPCs
(ConversationStarted/Failed) with GetLifetimeReplicatedProps
- ElevenLabsComponent: implement FindLocalRelayComponent(), route all
client-side calls through relay (StartConversation, EndConversation,
SendTextMessage, InterruptAgent, FeedExternalAudio, mic capture)
- Fix HandleConnected/ServerRequestConversation to route Client RPCs
through the player pawn's relay instead of the NPC (no owning connection)
- Fix StartListening/FeedExternalAudio/StopListening to accept
bNetIsConversing on clients (WebSocket only exists on server)
- Fix EvaluateBestAgent to use NetConversatingPawn instead of
NetConversatingPlayer (NULL on remote clients due to bOnlyRelevantToOwner)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent starts with bActive=false and waits for
OnConversationConnected, which only fires on the server (WebSocket).
Remote clients never got bActive=true, so CurrentActiveAlpha stayed
at 0 and all head/eye rotation was zeroed out.
Now OnRep_ConversationState also sets Posture->bActive alongside
TargetActor, matching what was already done for FacialExpression
and LipSync components.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs NetConversatingPawn, PostureComponent availability, and TargetActor
assignment to diagnose why head tracking doesn't work on remote clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raw PCM is split into 32KB sub-chunks for network transmission.
On the client, these sub-chunks arrive nearly simultaneously,
triggering the "second chunk arrived" fast-path which cancelled
the pre-buffer after ~10ms instead of the intended 2000ms.
Now the fast-path only applies on Authority (server) where
chunks represent genuine separate TTS batches. Clients always
wait the full pre-buffer duration via TickComponent timer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PlayerControllers have bOnlyRelevantToOwner=true, so NetConversatingPlayer
was always nullptr on remote clients. Added NetConversatingPawn (APawn*)
which IS replicated to all clients. OnRep_ConversationState now uses
NetConversatingPawn as the posture TargetActor, enabling head/eye
tracking on all clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 limits replicated TArrays to 65535 elements. ElevenLabs sends
audio chunks up to ~72K bytes, exceeding this limit when sent as
raw PCM. Split into 32000-byte sub-chunks (1s of 16kHz 16-bit mono)
before calling MulticastReceiveAgentAudio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 5.5's FVoiceModule returns NULL encoder/decoder, breaking
Opus-based audio replication. This adds a transparent fallback:
- Server sends raw PCM when OpusEncoder is null (~32KB/s, fine for LAN)
- Client accepts raw PCM when OpusDecoder is null
- Changed MulticastReceiveAgentAudio from Unreliable to Reliable
to handle larger uncompressed payloads without packet loss
- Added OnlineSubsystemNull config for future Opus compatibility
- Removed premature bAgentSpeaking=true from MulticastAgentStartedSpeaking
to fix race condition with audio initialization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Log at InitOpusCodec: FVoiceModule availability, encoder/decoder creation, net role
- Log at HandleAudioReceived: warn once if encoder is null or role is not Authority
- These fire unconditionally (not behind bDebug) to catch the root cause
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs compressed audio size in HandleAudioReceived to diagnose
why MulticastReceiveAgentAudio (Unreliable) never reaches clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
