MicrophoneCaptureComponent:
- Local Voice Activity Detection (RMS-based, independent of ElevenLabs)
- Configurable threshold, onset time, silence time
- bIsUserSpeaking flag + OnUserVoiceActivityChanged delegate
- Hysteresis prevents flickering between speech/silence
AIController gaze bridge:
- Resolve MicComponent from player Pawn (not NPC) via reflection
- ConversationPaused BB key blocks movement branches via BT decorator
- NPC stops only when user actually speaks (not just on proximity connect)
- NPC resumes when conversation disconnects
- Spline PauseFollowing/ResumeFollowing on conversation start/end
BT setup required:
- Add Blackboard Condition (ConversationPaused Is Not Set, Aborts=Both)
on spline and patrol branches
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Revert conversation movement pause (needs local VAD, not bNetIsConversing)
- Remove temporary gaze proximity debug log
- Remove bConversationPaused flag (will be reimplemented with VAD)
- TODO: pause NPC movement only when local voice activity is detected
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New BTService_UpdateGaze bridges PS_AI_Behavior to PS_AI_ConvAgent's
GazeComponent via runtime reflection (zero compile dependency).
Priority system:
- Combat/TakingCover: gaze disabled (aim animation handles it)
- Alerted: look at ThreatActor (head + eyes, no body)
- Conversation: skip (ConvAgent manages)
- Proximity: glance at nearest perceived actor within radius
Proximity gaze features:
- Lock on target until release (no jumping)
- Configurable duration, cooldown, radius on AIController
- Front-facing only (dot product filter)
- Skip spectators and hostile actors
GazeComponent fix:
- Sync SmoothedBodyYaw to actual actor rotation when body tracking
is disabled, preventing stale head offset during spline movement
Files:
- New: BTService_UpdateGaze.h/.cpp
- Modified: AIController.h/.cpp (gaze bridge, config, bind, cleanup)
- Modified: GazeComponent.cpp (body yaw sync fix)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- GetTeamAttitudeTowards fallback (interface path) was calling MakeTeamId
without Faction, so enemies of different factions had identical TeamIds
→ always Friendly instead of Hostile
- Now reads Faction from PersonalityProfile when resolving via interface
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move ThreatDecayRate from global Settings to PersonalityProfile (per-archetype)
- Add state hysteresis in EvaluateReaction to prevent Fleeing/Combat flickering
- FindCover: verify distance on arrival, retry movement if too far
- FindAndFollowSpline: sample multiple spline points when closest is blocked
- BTTask_Attack: call BehaviorStartAttack immediately (draw weapon before LOS/range)
- CoverShootCycle: call BehaviorStartAttack on entry (weapon ready during cover approach)
- FindOwningPawn: walk AttachParentActor chain for ChildActor weapons without Owner
- Initialize PreferCover BB key to false at setup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove Loyalty and Discipline from EPS_AI_Behavior_TraitAxis enum (never used in gameplay)
- Clean up PersonalityProfile and PersonalityComponent default trait initialization
- Add descriptive tooltips to remaining traits (Courage, Aggressivity, Caution)
- Lower Aggressivity combat gate from 0.3 to 0.0 (only Aggressivity=0 prevents combat)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BTTask_Attack.h: remove reference to non-existent GetBehaviorOptimalAttackRange, document PersonalityProfile ranges
- AIController.h: update TeamId comment to reflect actual MakeTeamId encoding (nibble-based)
- CoverShootCycle.h: clarify that Peek/Cover durations are base values modulated by personality traits
- FindCover.h: clarify ManualPointBonus is additive score
- CombatComponent.h: clarify AttackRange/AttackCooldown are for ExecuteAttack, not BTTask_Attack
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add CombatCoverCycleDuration to PersonalityProfile (configurable base duration)
- PersonalityComponent cycles bPreferCover based on Aggressivity/Caution ratio
- Combat duration = CycleDuration × Aggressivity/(Aggressivity+Caution)
- Cover duration = CycleDuration × Caution/(Aggressivity+Caution)
- Min 2s per phase, ±20% jitter
- Write PreferCover bool to Blackboard for BT decorator observer aborts
- IsCoverNeeded decorator checks both target type AND ShouldPreferCover()
- Remove TakingCover state from EvaluateReaction — cover is now a sub-mode of Combat
- BT uses Blackboard Condition on PreferCover with Observer Aborts=Both
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PerceptionComponent: omniscient TActorIterator only runs with HasAuthority()
- PersonalityComponent: ApplyReaction and ForceState gate replicated CurrentState writes to server only
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- IsCoverNeeded used Cast<APawn> on ThreatActor which failed for AimTargetActors
→ always returned true (assume dangerous) → enemies took cover against civilians
- Fix: use FindOwningPawn to walk Owner/Instigator chain to the actual Pawn
- Revert inline civilian check in EvaluateReaction (decorator handles it in BT)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- AIController: auto uncrouch when leaving Fleeing/TakingCover state
- FleeFrom: uncrouch at start of flee (civilian was still crouched from HidingSpot)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix EQS SetScore filter bug: pass FloatValueMin/Max instead of hardcoded 0/1 (filters were ignored)
- Add LateralSpread to LineOfSight EQS test: multiple traces for wider LOS check
- Add bDrawDebug to CoverQuality and LineOfSight EQS tests
- Ignore ThreatActor collision in EQS traces (AimTargetActor was blocking its own LOS)
- Add navmesh projection in EQSContext_CoverLocation for cover points inside geometry
- Add omniscient awareness: enemies detect top-priority targets (Protectors) within sight radius without perception cone
- Suppress target switching during TakingCover state to prevent cover invalidation
- Fix flanking check: trace at chest height instead of feet
- Add debug visualization for EQS fallback paths (NO REFINE, NO FIRE POS, IN PLACE)
- Clean up diagnostic logs: verbose for per-trace EQS details, log level for summaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CoverShootCycle: SetFocus only during AtCover (not shooting), flanking detection every 0.5s abandons compromised cover
- CoverQuality EQS: ground hit filter, MaxNearbyHitDist (300cm), lateral spread traces, capsule-relative heights
- EQSContext_CoverLocation: prefer CoverPoint actor location over refined vector
- UpdateThreat: never abandon target during Combat/TakingCover states
- MoveTimeout (5s) on all movement states to prevent stuck NPCs
- FindCover: detailed refinement result logging
- Debug: Peeking timer log for investigating timer-stops-counting bug (needs rebuild + test)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CoverShootCycle: add FiringPositionQuery EQS for peek/shoot positions
(NPC moves between cover position and firing position with LOS)
- Add BTDecorator_IsCoverNeeded: skip cover when target is Civilian
- Add EQSContext_CoverLocation: provides BB CoverLocation to EQS generators
- FindCover: add debug draw toggle and EQS refinement debug spheres
- Definitions: add ECoverShootSubState and CoverPointType::HidingSpot
NOTE: Has compilation errors to fix (signature mismatches in
CoverShootCycle StartPeeking/ReturnToCover, missing forward-declare)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- SetBehaviorCrouch() interface function for cover/hiding crouch control
- CoverShootCycle: continuous LOS check during Peeking (stop if target hides)
- CoverShootCycle: crouch/stand transitions at all cover state changes
- CoverShootCycle: fail when no LOS and no advancing cover (falls to Attack)
- FindCover: crouch on arrival, stand up on abort
- FindCover: optional EQS RefinementQuery to refine exact position around CoverPoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- LastKnownTargetPosition check now filters FLT_MAX/InvalidLocation (not just zero)
- EQS result with AlreadyAtGoal triggers fallback advance instead of no-op
- bProjectGoal enabled for EQS move to handle NavMesh projection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- EQSTest_LineOfSight: new EQS test for firing position queries
- Updated plugin binaries (PS_AI_Behavior + PS_AI_ConvAgent)
- TeamComponent, Settings, Build.cs updates from LOS plan implementation
- PLAN.md updated with LOS combat implementation progress
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Attack/CoverShootCycle: de-escalate to Alerted on failure so decorator
can re-trigger when threat returns (fixes BT stuck on spline branch)
- UpdateThreat: keep BB ThreatActor during brief perception gaps instead
of clearing immediately (use LOS timeout for graceful degradation)
- HasLineOfSight: ignore actors up the attachment chain so trace doesn't
hit the target Pawn's capsule when aiming at its AimTarget child actor
- NoTeam actors (spectators, editor pawns) treated as Neutral instead of
Hostile, plus SpectatorPawn explicit filter in perception
- BB debug key ThreatPawnName shows owning Pawn name (resolved via
perception's LastThreatOwningPawn) instead of cryptic AimTarget name
- FindOwningPawn promoted to public static on PerceptionComponent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split ResolveToPawn into FindOwningPawn + GetThreatTarget so non-Pawn
actors (PS_AimTargetActor) are properly resolved for team/attitude checks
while still being used as BB target. Add attack range hysteresis (10%
buffer), target persistence (80% threshold), melee no-cooldown chase,
ranged midpoint approach. New files: CoverShootCycle task, CheckCombatType
decorator, MinAttackRange/MaxAttackRange in PersonalityProfile.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add TeamComponent for player pawn team identity (Role-based: Civilian/Enemy/Protector)
- Add IsTargetActorValid to interface for dead target filtering
- Fix GetTeamAttitudeTowards to check IGenericTeamAgentInterface + TeamComponent
- Guarantee BehaviorStopAttack via OnTaskFinished (all exit paths)
- Prevent Combat state without ThreatActor (stay Alerted until perception catches up)
- Resume spline at closest point from current position after combat
- Sync CurrentSpline and SplineProgress to Blackboard in FollowSpline tick
- Auto-detect Patrol state when NPC has a spline (fixes Idle speed=0 blocking movement)
- Add per-component debug toggles (Personality + SplineFollower independent)
- Use AddMovementInput instead of RequestDirectMove for reliable post-combat movement
- Add bTickInEditor for Personality debug in Simulate mode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Always recalculate TeamId at OnPossess (BP CDOs may reset to 0)
- Remove duplicate _Implementation definitions (UHT auto-generates them)
- Fixes crash when interface functions not overridden in BP
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add BehaviorStartAttack/BehaviorStopAttack to IPS_AI_Behavior_Interface
- Attack task now calls interface instead of CombatComponent directly
- Task stays InProgress permanently, Decorator Observer Aborts handles exit
- Remove CombatComponent dependency from Attack task
- Pawn handles actual aiming/shooting via its own systems
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BT Attack task:
- Stay InProgress permanently instead of returning Succeeded after each attack
- Stay InProgress on MoveToActor failure instead of Failed (retry next tick)
- Add verbose logging for attack state (target, range, distance)
BT EvaluateReaction service:
- Auto-detect hostility changes via IPS_AI_Behavior interface
- Dynamically update TeamId when IsBehaviorHostile() changes (infiltrator reveal)
AIController:
- Remove GetBehaviorTeamId from interface (TeamId is now 100% automatic)
- TeamId derived from NPCType + hostile state, no user implementation needed
- Add BB State change logging for debug
- Use SetValueAsEnum consistently for BehaviorState key
Interface:
- Remove GetBehaviorTeamId — TeamId is computed by the plugin automatically
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Perception:
- Configure senses in BeginPlay + RequestStimuliListenerUpdate (not constructor)
- Filter threats by team attitude: only Hostile actors generate threat
- Skip Friendly and Neutral actors in CalculateThreatLevel and GetHighestThreatActor
- All Hostile actors are valid threats regardless of TargetPriority list
Blackboard:
- Use SetValueAsEnum/GetValueAsEnum for BehaviorState key (was Int)
Patrol:
- Auto NavMesh patrol when no manual waypoints defined
- Project HomeLocation onto NavMesh before searching random points
- Fallback to NPC current position if HomeLocation not on NavMesh
Spline following:
- Choose direction based on NPC forward vector vs spline tangent (not longest distance)
- Skip re-search if already following a spline (prevent BT re-boucle)
- Reverse at end: wait for NPC to catch up before inverting direction
- Use NPC actual CharacterMovement speed for target point advancement
- Face toward target point (not spline tangent) to prevent crab-walking in turns
- Junction switching: direction continuity check, 70% switch chance, world gap tolerance
- Debug draw: green sphere (target), yellow line (gap), cyan arrow (tangent), text overlay
- Add GetWorldDirectionAtDistance to SplinePath
Editor:
- Add Placeable to SplinePath and CoverPoint UCLASS
- Add PersonalityProfileFactory for Data Asset creation
- Add EnsureBlackboardAsset declaration
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add bExpressiveMode toggle and editable ExpressiveModePromptFragment
with audio tag instructions ([laughs], [whispers], [sighs], [slow], [excited])
- BuildAgentPayload: append prompt fragment, set expressive_mode API field
on agent config, auto-override TTS model to eleven_v3_conversational
- OnFetchAgent: strip expressive fragment from prompt (exact + marker fallback),
read expressive_mode bool from API, auto-detect V3 model
- TTS model combo: inject asset's current model if absent from /v1/models list
(covers agent-only models like eleven_v3_conversational)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix A→B→A audio cutoff: when switching back to a pending-leave agent,
cancel the deferred leave instead of force-completing it (was calling
StopAgentAudio on the agent we're returning to)
- Fix deferred leave firing during TTS gaps: use IsAgentSpeakingOrPending()
instead of IsAgentSpeaking() — checks bAgentGenerating and
bAgentResponseReceived to avoid premature leave during inter-batch silence
- Convert silence detection from tick-based to time-based: SilentTickCount
→ SilentTime (float seconds), GeneratingTickCount → GeneratingTime.
Consistent behavior regardless of frame rate (was 5s@120fps vs 20s@30fps)
- Fix lazy binding: add OnAgentConnected/OnAgentDisconnected in LipSync
and FacialExpression TickComponent lazy-bind path (bActive stayed false
forever in packaged builds when component init order differed)
- Fix reconnection: reset bWaitingForAgentResponse and GeneratingTime
before entering reconnect mode to avoid stale state on new session
- Fix event_ID audio filtering: reset LastInterruptEventId in
HandleAgentResponse and SendUserTurnStart so first audio chunks of a
new turn are not silently discarded by stale interrupt filter
- Preserve retained gaze when switching back to same agent (don't
CleanupRetainedGaze if PrevRetained == NewAgent)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ForceDisableConversation/ForceEnableConversation: disable agent conversation
with blend-out monitoring and OnReadyForAction event (with ActionName param)
- ActionSet data asset: configurable action list per agent with editor
customization (Update All Agents button, custom detail panel)
- Passive gaze by proximity: nearby non-selected agents track the player
with configurable head+eyes and body checkboxes (bAutoPassiveGaze,
bPassiveGazeHeadEyes, bPassiveGazeBody)
- Retained gaze on conversation switch now uses the same passive gaze config
- OnPassiveGazeStarted/OnPassiveGazeStopped events on ElevenLabsComponent
- Fix debug HUD key collisions: per-actor key ranges prevent multi-agent
HUD flickering, add actor name to all HUD titles
- Fix retained gaze bug: re-activate gaze after ExecuteLeave before
ApplyConversationGaze kills it
- Safety timeout (5s) for blend-out monitoring
- Guard on AttachGazeTarget when conversation is disabled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Conversation lock no longer prevents switching to a different agent.
When in an active conversation, the player can look at another nearby
agent for ConversationSwitchDelay seconds (default 1s) to switch.
Looking at empty space keeps the current agent selected (no deselect).
Works in multiplayer — each player has independent switch tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add TurnEagerness (Eager/Normal/Patient) and bSpeculativeTurn to agent config
data asset, sent as conversation_config_override at WebSocket connection time
- Add adaptive pre-buffer system: measures inter-chunk TTS timing and decreases
pre-buffer when chunks arrive fast enough (decrease-only, resets each conversation)
- New UPROPERTY: bAdaptivePreBuffer toggle, AudioPreBufferMs as starting/worst-case value
- Rework latency HUD: TTS+Net, PreBuf actual/target with trend indicator, Gen>Ear,
WS Ping, server region display
- Fetch ElevenLabs server region from REST API x-region header
- Add editor Detail Customization: TurnEagerness dropdown + SpeculativeTurn checkbox
in AgentConfig with LLM picker and Language picker
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
user_transcript arrives AFTER agent_response_started in Server VAD mode
(the server detects end of speech via VAD, starts generating immediately,
and STT completes later). This caused Transcript>Gen to show stale values
(19s) and Total < Gen>Audio (impossible).
Now all metrics are anchored to GenerationStartTime (agent_response_started),
which is the closest client-side proxy for "user stopped speaking":
- Gen>Audio: generation start → first audio chunk (LLM + TTS)
- Pre-buffer: wait before playback
- Gen>Ear: generation start → playback starts (user-perceived)
Removed STTToGenMs, TotalMs, EndToEarMs, UserSpeechMs (all depended on
unreliable timestamps). Simpler, always correct, 3 clear metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous approach used TurnEndTime (from StopListening) which was never
set in Server VAD mode. Now all latency measurements are anchored to
TurnStartTime, captured when the first user_transcript arrives from the
ElevenLabs server — the earliest client-side confirmation of user speech.
Timeline: [user speaks] → STT → user_transcript(=T0) → agent_response_started → audio → playback
Metrics shown:
- Transcript>Gen: T0 → generation start (LLM think time)
- Gen>Audio: generation start → first audio chunk (LLM + TTS)
- Total: T0 → first audio chunk (full pipeline)
- Pre-buffer: wait before playback
- End-to-Ear: T0 → playback starts (user-perceived)
Removed UserSpeechMs (unmeasurable without client-side VAD).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TurnStartTime was only set in StartListening(), which is called once.
In Server VAD + interruption mode the mic stays open, so TurnStartTime
was never updated between turns. Now reset TurnStartTime when the agent
stops speaking (normal end + interruption), marking the start of the
next potential user turn.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move latency reset from StopListening to HandleAgentResponseStarted.
In Server VAD + interruption mode, StopListening is never called so
TurnEndTime stayed at 0 and all dependent metrics showed ---. Now
HandleAgentResponseStarted detects whether StopListening provided a
fresh TurnEndTime; if not (Server VAD), it uses Now as approximation.
Also fix DisplayTime from 0 to 1s to prevent HUD flicker.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add bDebugLatency property + CVar (ps.ai.ConvAgent.Debug.Latency)
independent from bDebug to save HUD space
- Reset latencies to zero each turn (StopListening) instead of persisting
- Add UserSpeechMs and PreBufferMs to the latency struct
- Move latency captures outside bDebug guard (always measured)
- Replace single-line latency in DrawDebugHUD with dedicated DrawLatencyHUD()
showing 6 metrics on separate lines with color coding
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Track 4 latencies per conversation turn (computed only when bDebug is active):
- STT→Gen: user stops talking → server starts generating
- Gen→Audio: server generating → first audio chunk received
- Total: user stops talking → first audio chunk (end-to-end)
- End-to-Ear: user stops talking → audio playback starts (includes pre-buffer)
New timestamps: GenerationStartTime (HandleAgentResponseStarted),
PlaybackStartTime (3 OnAudioPlaybackStarted sites). Values persist on
HUD between turns, reset when new turn starts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. StartConversationWithSelectedAgent: remove early return when WebSocket
is already connected (persistent mode). Always call ServerJoinConversation
so the pawn is added to NetConnectedPawns and bNetIsConversing is set.
2. ServerSendMicAudioFromPlayer: bypass speaker arbitration in standalone
mode (<=1 connected pawn). Send audio directly to avoid silent drops
caused by pawn not being in NetConnectedPawns array. Add warning logs
for multi-player drops to aid debugging.
3. OnMicrophoneDataCaptured: restore direct WebSocketProxy->SendAudioChunk
on the server path. This callback runs on the WASAPI audio thread —
accessing game-thread state (NetConnectedPawns, LastSpeakTime) was
causing undefined behavior. Internal mic is always the local player,
no speaker arbitration needed.
4. StopListening flush: send directly to WebSocket (active speaker already
established, no arbitration needed for the tail of the current turn).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GetCurrentBlendshapes() was copying CurrentBlendshapes on the anim worker
thread while the game thread mutated it (TSet::UnhashElements crash).
Use a snapshot pattern: game thread copies to ThreadSafeBlendshapes under
FCriticalSection at end of TickComponent, anim node reads the snapshot.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace exclusive single-player agent lock with shared multi-player model:
- NetConversatingPawn/Player → NetConnectedPawns array + NetActiveSpeakerPawn
- Server-side speaker arbitration with hysteresis (0.3s) prevents gaze ping-pong
- Speaker idle timeout (3.0s) clears active speaker after silence
- Agent gaze follows the active speaker via replicated OnRep_ActiveSpeaker
- New ServerJoinConversation/ServerLeaveConversation RPCs (idempotent join/leave)
- Backward-compatible: old ServerRequest/Release delegate to new Join/Leave
- InteractionComponent no longer skips occupied agents
- DrawDebugHUD shows connected player count and active speaker
- All mic audio paths (FeedExternalAudio, OnMicCapture, StopListening flush) route
through speaker arbitration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- InteractionComponent: GetPawnViewPoint() now prefers UCameraComponent
over PlayerController::GetPlayerViewPoint() — fixes agent selection
in VR where the pawn root stays at spawn while the HMD moves freely
- GazeComponent: ResolveTargetPosition() uses camera first for locally
controlled pawns (VR HMD / FPS eye position), falls back to bone
chain for NPCs and remote players in multiplayer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
