Perception:
- Configure senses in BeginPlay + RequestStimuliListenerUpdate (not constructor)
- Filter threats by team attitude: only Hostile actors generate threat
- Skip Friendly and Neutral actors in CalculateThreatLevel and GetHighestThreatActor
- All Hostile actors are valid threats regardless of TargetPriority list
Blackboard:
- Use SetValueAsEnum/GetValueAsEnum for BehaviorState key (was Int)
Patrol:
- Auto NavMesh patrol when no manual waypoints defined
- Project HomeLocation onto NavMesh before searching random points
- Fallback to NPC current position if HomeLocation not on NavMesh
Spline following:
- Choose direction based on NPC forward vector vs spline tangent (not longest distance)
- Skip re-search if already following a spline (prevent BT re-boucle)
- Reverse at end: wait for NPC to catch up before inverting direction
- Use NPC actual CharacterMovement speed for target point advancement
- Face toward target point (not spline tangent) to prevent crab-walking in turns
- Junction switching: direction continuity check, 70% switch chance, world gap tolerance
- Debug draw: green sphere (target), yellow line (gap), cyan arrow (tangent), text overlay
- Add GetWorldDirectionAtDistance to SplinePath
Editor:
- Add Placeable to SplinePath and CoverPoint UCLASS
- Add PersonalityProfileFactory for Data Asset creation
- Add EnsureBlackboardAsset declaration
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add bExpressiveMode toggle and editable ExpressiveModePromptFragment
with audio tag instructions ([laughs], [whispers], [sighs], [slow], [excited])
- BuildAgentPayload: append prompt fragment, set expressive_mode API field
on agent config, auto-override TTS model to eleven_v3_conversational
- OnFetchAgent: strip expressive fragment from prompt (exact + marker fallback),
read expressive_mode bool from API, auto-detect V3 model
- TTS model combo: inject asset's current model if absent from /v1/models list
(covers agent-only models like eleven_v3_conversational)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix A→B→A audio cutoff: when switching back to a pending-leave agent,
cancel the deferred leave instead of force-completing it (was calling
StopAgentAudio on the agent we're returning to)
- Fix deferred leave firing during TTS gaps: use IsAgentSpeakingOrPending()
instead of IsAgentSpeaking() — checks bAgentGenerating and
bAgentResponseReceived to avoid premature leave during inter-batch silence
- Convert silence detection from tick-based to time-based: SilentTickCount
→ SilentTime (float seconds), GeneratingTickCount → GeneratingTime.
Consistent behavior regardless of frame rate (was 5s@120fps vs 20s@30fps)
- Fix lazy binding: add OnAgentConnected/OnAgentDisconnected in LipSync
and FacialExpression TickComponent lazy-bind path (bActive stayed false
forever in packaged builds when component init order differed)
- Fix reconnection: reset bWaitingForAgentResponse and GeneratingTime
before entering reconnect mode to avoid stale state on new session
- Fix event_ID audio filtering: reset LastInterruptEventId in
HandleAgentResponse and SendUserTurnStart so first audio chunks of a
new turn are not silently discarded by stale interrupt filter
- Preserve retained gaze when switching back to same agent (don't
CleanupRetainedGaze if PrevRetained == NewAgent)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ForceDisableConversation/ForceEnableConversation: disable agent conversation
with blend-out monitoring and OnReadyForAction event (with ActionName param)
- ActionSet data asset: configurable action list per agent with editor
customization (Update All Agents button, custom detail panel)
- Passive gaze by proximity: nearby non-selected agents track the player
with configurable head+eyes and body checkboxes (bAutoPassiveGaze,
bPassiveGazeHeadEyes, bPassiveGazeBody)
- Retained gaze on conversation switch now uses the same passive gaze config
- OnPassiveGazeStarted/OnPassiveGazeStopped events on ElevenLabsComponent
- Fix debug HUD key collisions: per-actor key ranges prevent multi-agent
HUD flickering, add actor name to all HUD titles
- Fix retained gaze bug: re-activate gaze after ExecuteLeave before
ApplyConversationGaze kills it
- Safety timeout (5s) for blend-out monitoring
- Guard on AttachGazeTarget when conversation is disabled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Conversation lock no longer prevents switching to a different agent.
When in an active conversation, the player can look at another nearby
agent for ConversationSwitchDelay seconds (default 1s) to switch.
Looking at empty space keeps the current agent selected (no deselect).
Works in multiplayer — each player has independent switch tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add TurnEagerness (Eager/Normal/Patient) and bSpeculativeTurn to agent config
data asset, sent as conversation_config_override at WebSocket connection time
- Add adaptive pre-buffer system: measures inter-chunk TTS timing and decreases
pre-buffer when chunks arrive fast enough (decrease-only, resets each conversation)
- New UPROPERTY: bAdaptivePreBuffer toggle, AudioPreBufferMs as starting/worst-case value
- Rework latency HUD: TTS+Net, PreBuf actual/target with trend indicator, Gen>Ear,
WS Ping, server region display
- Fetch ElevenLabs server region from REST API x-region header
- Add editor Detail Customization: TurnEagerness dropdown + SpeculativeTurn checkbox
in AgentConfig with LLM picker and Language picker
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
user_transcript arrives AFTER agent_response_started in Server VAD mode
(the server detects end of speech via VAD, starts generating immediately,
and STT completes later). This caused Transcript>Gen to show stale values
(19s) and Total < Gen>Audio (impossible).
Now all metrics are anchored to GenerationStartTime (agent_response_started),
which is the closest client-side proxy for "user stopped speaking":
- Gen>Audio: generation start → first audio chunk (LLM + TTS)
- Pre-buffer: wait before playback
- Gen>Ear: generation start → playback starts (user-perceived)
Removed STTToGenMs, TotalMs, EndToEarMs, UserSpeechMs (all depended on
unreliable timestamps). Simpler, always correct, 3 clear metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous approach used TurnEndTime (from StopListening) which was never
set in Server VAD mode. Now all latency measurements are anchored to
TurnStartTime, captured when the first user_transcript arrives from the
ElevenLabs server — the earliest client-side confirmation of user speech.
Timeline: [user speaks] → STT → user_transcript(=T0) → agent_response_started → audio → playback
Metrics shown:
- Transcript>Gen: T0 → generation start (LLM think time)
- Gen>Audio: generation start → first audio chunk (LLM + TTS)
- Total: T0 → first audio chunk (full pipeline)
- Pre-buffer: wait before playback
- End-to-Ear: T0 → playback starts (user-perceived)
Removed UserSpeechMs (unmeasurable without client-side VAD).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TurnStartTime was only set in StartListening(), which is called once.
In Server VAD + interruption mode the mic stays open, so TurnStartTime
was never updated between turns. Now reset TurnStartTime when the agent
stops speaking (normal end + interruption), marking the start of the
next potential user turn.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move latency reset from StopListening to HandleAgentResponseStarted.
In Server VAD + interruption mode, StopListening is never called so
TurnEndTime stayed at 0 and all dependent metrics showed ---. Now
HandleAgentResponseStarted detects whether StopListening provided a
fresh TurnEndTime; if not (Server VAD), it uses Now as approximation.
Also fix DisplayTime from 0 to 1s to prevent HUD flicker.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add bDebugLatency property + CVar (ps.ai.ConvAgent.Debug.Latency)
independent from bDebug to save HUD space
- Reset latencies to zero each turn (StopListening) instead of persisting
- Add UserSpeechMs and PreBufferMs to the latency struct
- Move latency captures outside bDebug guard (always measured)
- Replace single-line latency in DrawDebugHUD with dedicated DrawLatencyHUD()
showing 6 metrics on separate lines with color coding
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Track 4 latencies per conversation turn (computed only when bDebug is active):
- STT→Gen: user stops talking → server starts generating
- Gen→Audio: server generating → first audio chunk received
- Total: user stops talking → first audio chunk (end-to-end)
- End-to-Ear: user stops talking → audio playback starts (includes pre-buffer)
New timestamps: GenerationStartTime (HandleAgentResponseStarted),
PlaybackStartTime (3 OnAudioPlaybackStarted sites). Values persist on
HUD between turns, reset when new turn starts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. StartConversationWithSelectedAgent: remove early return when WebSocket
is already connected (persistent mode). Always call ServerJoinConversation
so the pawn is added to NetConnectedPawns and bNetIsConversing is set.
2. ServerSendMicAudioFromPlayer: bypass speaker arbitration in standalone
mode (<=1 connected pawn). Send audio directly to avoid silent drops
caused by pawn not being in NetConnectedPawns array. Add warning logs
for multi-player drops to aid debugging.
3. OnMicrophoneDataCaptured: restore direct WebSocketProxy->SendAudioChunk
on the server path. This callback runs on the WASAPI audio thread —
accessing game-thread state (NetConnectedPawns, LastSpeakTime) was
causing undefined behavior. Internal mic is always the local player,
no speaker arbitration needed.
4. StopListening flush: send directly to WebSocket (active speaker already
established, no arbitration needed for the tail of the current turn).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GetCurrentBlendshapes() was copying CurrentBlendshapes on the anim worker
thread while the game thread mutated it (TSet::UnhashElements crash).
Use a snapshot pattern: game thread copies to ThreadSafeBlendshapes under
FCriticalSection at end of TickComponent, anim node reads the snapshot.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace exclusive single-player agent lock with shared multi-player model:
- NetConversatingPawn/Player → NetConnectedPawns array + NetActiveSpeakerPawn
- Server-side speaker arbitration with hysteresis (0.3s) prevents gaze ping-pong
- Speaker idle timeout (3.0s) clears active speaker after silence
- Agent gaze follows the active speaker via replicated OnRep_ActiveSpeaker
- New ServerJoinConversation/ServerLeaveConversation RPCs (idempotent join/leave)
- Backward-compatible: old ServerRequest/Release delegate to new Join/Leave
- InteractionComponent no longer skips occupied agents
- DrawDebugHUD shows connected player count and active speaker
- All mic audio paths (FeedExternalAudio, OnMicCapture, StopListening flush) route
through speaker arbitration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- InteractionComponent: GetPawnViewPoint() now prefers UCameraComponent
over PlayerController::GetPlayerViewPoint() — fixes agent selection
in VR where the pawn root stays at spawn while the HMD moves freely
- GazeComponent: ResolveTargetPosition() uses camera first for locally
controlled pawns (VR HMD / FPS eye position), falls back to bone
chain for NPCs and remote players in multiplayer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix DisplayTime=0.0f causing flicker on all debug HUDs (now 1.0f)
- Add per-component CVars (ps.ai.ConvAgent.Debug.*) for console debug toggle
- Add MicrophoneCapture debug HUD with VU meter (RMS/peak/dB bar)
- InteractionComponent reuses existing MicrophoneCaptureComponent on pawn
instead of always creating a new one
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Sync body animation with actual audio playback via new OnAudioPlaybackStarted
delegate instead of OnAgentStartedSpeaking (accounts for pre-buffer delay)
- Fix stale pre-buffer broadcasts by cancelling bPreBuffering on silence detection
and guarding pre-buffer timeout with bAgentSpeaking check
- Smooth body crossfade using FInterpTo instead of linear interpolation
- Add conversation lock in EvaluateBestAgent: keep agent selected during active
conversation regardless of view cone (distance-only check prevents deselect
flicker on fast camera turns)
- Broadcast OnAgentDisconnected in persistent session EndConversation so all
expression components (body, facial, lip sync, gaze) properly deactivate
when the player leaves the interaction zone
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace FInterpConstantTo with FInterpTo (exponential ease-out) in all 4 anim components
- Apply SmoothStep to crossfade alphas for smooth animation transitions
- Add SpeechBlendAlpha to LipSync: fades out mouth curves when not speaking,
letting FacialExpression emotion curves (including mouth) show through
- Remove LipSync AnimNode grace period zeroing to avoid overwriting emotion curves
- Revert BodyExpression to override mode (additive broken with full-pose anims)
- Default ExcludeBones = neck_01 to prevent Gaze/Posture conflicts
- Fix mid-crossfade animation pop in BodyExpression SwitchToNewAnim
- Add Neutral emotion fallback in Body and Facial expression components
- Add SendTextToSelectedAgent convenience method on InteractionComponent
- Add debug HUD display for BodyExpression component
- Update CoreRedirects for Posture → Gaze rename
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add BodyExpression system: emotion-driven body animations with per-emotion
anim lists (Idle/Normal/Medium/Extreme), random selection, auto-cycle on
loop complete, crossfade transitions, upper-body-only or full-body mode
- Replace bExternalMicManagement with auto-detecting ShouldUseExternalMic()
- Add bAutoTargetEyes to GazeComponent: auto-aim at target's eye bones
(MetaHuman), with fallback chain (eyes > head > FallbackEyeHeight)
- Hide bActive from Details panel on all 4 anim components (read-only,
code-managed): FacialExpression, Gaze, LipSync, BodyExpression
- Remove misleading mh_arkit_mapping_pose warning from LipSync
- Add bUpperBodyOnly toggle to BodyExpression AnimNode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three issues prevented posture re-activation on re-entry:
- StartConversation skipped bNetIsConversing/NetConversatingPawn in standalone
(guarded by NM_Standalone check) so ApplyConversationPosture always deactivated
- SetSelectedAgent required !IsConnected() to auto-start conversation, but in
persistent mode the WS stays connected → StartConversation never called on re-entry
- AttachPostureTarget never set Posture->bActive = true, relying solely on
ApplyConversationPosture which could be skipped due to the above condition
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When true (default), the first StartConversation() opens the WebSocket
and EndConversation() only stops the mic/posture — the WebSocket stays
open until EndPlay. The agent remembers the full conversation context.
When false, each Start/EndConversation opens/closes the WebSocket (previous behavior).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Make bAgentGenerating, bWaitingForAgentResponse, bWaitingForResponse,
bFirstAudioResponseLogged, bAgentResponseStartedFired (std::atomic<bool>)
and LastInterruptEventId (std::atomic<int32>) for thread-safety
- Add WebSocket auto-reconnection with exponential backoff (1s→30s cap,
max 5 attempts), distinguishing intentional vs unexpected disconnects
- Add FillCurrentEyeCurves() zero-allocation method using FindOrAdd()
to eliminate per-frame TMap heap allocation in anim thread
- Replace AudioQueue RemoveAt(0,N) with read-offset pattern — O(1) per
underflow callback, periodic compaction when offset > half buffer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix only smoothed cascade inputs (head/eyes) via SmoothedBodyYaw
but the body mesh still followed the raw replicated actor rotation which
jumped at ~30Hz network rate. Now the client calls SetActorRotation with
the smoothed yaw so the mesh visually interpolates. Replication overwrites
on next network update; SmoothedBodyYaw absorbs the correction smoothly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Server-only AddActorWorldRotation (HasAuthority guard) prevents
client/server tug-of-war. Client interpolates toward replicated
rotation via SmoothedBodyYaw (angle-aware FInterpTo at 3x body
speed) to eliminate step artifacts from ~30Hz network updates.
Cascade (DeltaYaw, head, eyes) uses SmoothedBodyYaw on all machines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The pelvis bone rotation approach didn't work in practice. Reverts to
the previous AddActorWorldRotation() body tracking (replicated actor
rotation). Thread-safety fix and deprecated IsValid() removal are kept.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- FacialExpressionComponent: add FCriticalSection around emotion curves
to prevent race between TickComponent (game thread) and anim worker
thread — root cause of EXCEPTION_ACCESS_VIOLATION at Evaluate_AnyThread
- Remove deprecated FBlendedCurve::IsValid() guards (UE 5.5: always true)
- Body tracking: replace AddActorWorldRotation() with animation-only
pelvis bone rotation via AnimNode — eliminates replication tug-of-war
that caused client-side saccades when server overwrote local rotation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create OpusEncoder on ALL machines (was Authority-only) — clients now
encode mic audio, server decodes it; server still encodes agent audio
- FeedExternalAudio / OnMicrophoneDataCaptured: Opus-encode accumulated
PCM buffer before sending via ServerRelayMicAudio RPC on client path
(~200 bytes/100ms instead of 3200 bytes = ~16 Kbits/s vs 256 Kbits/s)
- ServerRelayMicAudio_Implementation: auto-detect Opus (size < raw chunk)
and decode back to PCM before forwarding to WebSocket
- Add public DecompressMicAudio() helper for clean API access from
InteractionComponent relay without exposing private Opus members
- Graceful fallback: if Opus unavailable, raw PCM is sent/received as before
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Attach AudioPlaybackComponent to owner's root component for proper
3D world positioning (was unattached = stuck at origin)
- Enable default inline spatialization with 15m falloff distance when
no external SoundAttenuation asset is set
- External SoundAttenuation asset still overrides the default if set
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-end conversation on deselection: when bAutoStartConversation is
true and the player walks out of range, EndConversation() is called
so the NPC becomes available for other players
- Server-side posture: add ApplyConversationPosture() helper called
from ServerRequestConversation, ServerReleaseConversation,
EndConversation (Authority path), and HandleDisconnected — fixes
NPC not tracking the client on the listen server (OnRep never fires
on Authority)
- Guard DetachPostureTarget: only clear TargetActor if it matches our
pawn, preventing the server IC from overwriting posture set by the
conversation system for a remote client
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Silence gate: skip sending silent mic audio over network RPCs on clients
(~256 Kbits/s saved when not speaking, fixes chaotic teleporting)
- Lazy init: defer InteractionComponent mic creation from BeginPlay to
TickComponent with IsLocallyControlled guard (fixes "No owning connection"
from server-side replicas of remote pawns)
- Body tracking: use bNetIsConversing as fallback for IsConnected() on
clients where WebSocket doesn't exist
- EvaluateBestAgent: null-check NetConversatingPawn before comparison
- MicCaptureComponent: use TWeakObjectPtr in AsyncTask lambda to prevent
FMRSWRecursiveAccessDetector race on component destruction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
In a listen server, the server-side copy of a remote client's pawn also
has an InteractionComponent that ticks. This caused a race condition:
the server-side tick would start conversations using GetFirstPlayerController()
(= server's PC), setting NetConversatingPawn to the server's pawn instead
of the client's. The client's relay RPC arrived too late and was rejected
because bNetIsConversing was already true.
Fix: disable tick and skip mic creation in BeginPlay for non-locally-controlled
pawns. The client handles all interaction locally via relay RPCs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 clients cannot call Server RPCs on actors they don't own. NPC actors
are server-owned, causing "No owning connection" errors when remote clients
try to start conversations, send mic audio, or interrupt agents.
Solution: relay all client→NPC RPCs through the InteractionComponent on
the player's pawn (which IS owned by the client). The relay forwards
commands to the NPC's ElevenLabsComponent on the server side.
Changes:
- InteractionComponent: add Server relay RPCs (Start/End conversation,
mic audio, text message, interrupt) and Client relay RPCs
(ConversationStarted/Failed) with GetLifetimeReplicatedProps
- ElevenLabsComponent: implement FindLocalRelayComponent(), route all
client-side calls through relay (StartConversation, EndConversation,
SendTextMessage, InterruptAgent, FeedExternalAudio, mic capture)
- Fix HandleConnected/ServerRequestConversation to route Client RPCs
through the player pawn's relay instead of the NPC (no owning connection)
- Fix StartListening/FeedExternalAudio/StopListening to accept
bNetIsConversing on clients (WebSocket only exists on server)
- Fix EvaluateBestAgent to use NetConversatingPawn instead of
NetConversatingPlayer (NULL on remote clients due to bOnlyRelevantToOwner)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent starts with bActive=false and waits for
OnConversationConnected, which only fires on the server (WebSocket).
Remote clients never got bActive=true, so CurrentActiveAlpha stayed
at 0 and all head/eye rotation was zeroed out.
Now OnRep_ConversationState also sets Posture->bActive alongside
TargetActor, matching what was already done for FacialExpression
and LipSync components.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs NetConversatingPawn, PostureComponent availability, and TargetActor
assignment to diagnose why head tracking doesn't work on remote clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
