Single-player clients sometimes showed the head turned ~60° off while the
server looked correct. Two client-only causes:
1. The client overwrote the actor's rotation every frame (SetActorRotation)
and then read back its own output as the "replicated" yaw, so SmoothedBodyYaw
diverged from the true server body yaw.
2. The head/eye cascade was sticky / path-dependent (TargetHeadYaw updated on
overflow events), so running it independently on a client fed by lagged,
stepped replication latched into a different state — the head overflowed to
MaxHeadYaw and stayed there.
Rewrite head/eyes as a STATELESS function of the actual (authoritative on the
server, replicated on the client) body facing + target direction. No sticky
state, FInterpTo only eases the visual. The client no longer overrides the actor
rotation. Body turn stays authority-only (lazy turn beyond head+eyes range).
Server and client now converge to the same pose by construction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With an open mic streaming continuously (Server VAD + interruption), every
chunk refreshed the active speaker's LastSpeakTime, so SpeakerSwitchHysteresis
never elapsed and SpeakerIdleTimeout never fired — the first speaker locked the
floor forever and the agent never turned to anyone else.
Now only chunks with real voice energy (RMS >= 0.02) hold/take the floor: the
active speaker is still forwarded continuously (silence tail for VAD), but their
lock only persists while they actually speak. When they pause, another player's
voiced audio can switch in and the agent turns to them (NetActiveSpeakerPawn
replicated -> gaze). Validated with two players on one NPC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Client mic uplink confirmed working in a packaged build across two PCs
(networked Server VAD): the agent now replies to the player's voice.
Remove the [MIC-DBG]/[MIC-DBG-SRV] instrumentation added to trace it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Networked clients could hear the agent but the agent never replied to them
(worked in standalone/listen-server host). Root causes, client-only paths
that never run on the authority:
1. Deferred StartListening was never retried once the conversation went live.
OnRep_ConversationState now (re)starts listening when bNetIsConversing
replicates and the local player is in the conversation — closes the
RPC/replication ordering race so bIsListening reliably opens the uplink.
2. The client stripped ALL silent mic chunks to save bandwidth, but ElevenLabs
Server VAD needs the trailing silence to detect end-of-turn — so the agent
heard the words but never the pause and never responded. Replaced the hard
silence drop with a silence "tail": keep streaming for MicSilenceTailSeconds
(default 2s, tunable) after the last voiced chunk, then drop during idle.
The authority path is unchanged (always streams the full mic).
Also: debug HUD now shows "CONNECTED (via server)" on clients instead of a
misleading "DISCONNECTED" (clients have no local WebSocket; IsConnected() is
always false there).
[MIC-DBG] temporary diagnostics are kept in this commit to validate the fix on
another machine; they will be removed in a follow-up once confirmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-enables agent/tool/action-set authoring in the editor after the API
key was removed from Project Settings. Adds an editor-only key source so
it can never be committed-as-config or baked into a packaged build.
- New UPS_AI_ConvAgent_EditorSettings_ElevenLabs (UDeveloperSettings,
config=EditorPerProjectUserSettings, Editor category): single global
field, stored under Saved/Config (never cooked), in the editor module
(never packaged).
- ResolveEditorApiKey(): editor setting -> ELEVENLABS_API_KEY env -> runtime
SaveGame fallback. All three editor customizations route through it.
- Fix the now-stale "API Key not set in Project Settings" messages.
Runtime path is untouched: packaged game still uses the client-provided
key via the SaveGame (Set/GetElevenLabsAPIKey).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Auto pause now triggers as soon as a conversation is genuinely active (agent greeting included) instead of waiting for the user to speak — the NPC no longer walks away during its opening line.
Add DisableBehaviorMovement()/EnableBehaviorMovement() (BlueprintCallable): freeze the NPC's autonomous movement while keeping the Behavior Tree running (perception, threat, gaze, reactions). Movement is paused when EITHER manually disabled OR a conversation is active. Both drive the ConversationPaused Blackboard flag read by the movement-branch decorators; leave the Flee branch un-decorated so fleeing still works.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
StopFollowing() now sets a bManuallyStopped flag (cleared only by an explicit StartFollowing/StartFollowingAtDistance). The FindAndFollowSpline and FollowSpline BT tasks bail while it is set, so the BT no longer auto-resumes a manual stop on the next tick. ResumeFollowing() also respects the flag. PauseFollowing() is unchanged (BT may resume).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds on the client-provided-key SaveGame work to actually secure usage:
- Private agents: connect via a short-lived Signed URL fetched with the stored key (GET /v1/convai/conversation/get-signed-url), instead of a direct agent_id+header connection which ElevenLabs rejects for auth-enabled agents. Falls back to a direct connection only when no key is set (public agents). Enforces 'valid key required': no key -> no signed URL -> no conversation.
- Force agent creation/update to private (platform_settings.auth.enable_auth=true) so a public, key-less agent can't be created by mistake.
- Encrypt the API key at rest in the SaveGame (AES-256); plaintext kept only in the in-memory cache.
- Redact signed-URL query (auth token) from logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The API key is no longer stored in project settings / DefaultEngine.ini (it was committed and baked into shipped builds). It is now provided per-install by the client at runtime and persisted in a SaveGame slot.
- Add UPS_AI_ConvAgent_SaveGame_ElevenLabs (slot PS_AI_ConvAgent_ElevenLabs)
- BlueprintLibrary is the single owner of the key: Set/Get/Has/Clear + in-memory cache backed by the SaveGame
- Add async BP node 'Test ElevenLabs API Key' (GET /v1/models)
- Remove API_Key UPROPERTY from UPS_AI_ConvAgent_Settings_ElevenLabs and the key logic from the module
- Route all consumers (WebSocket auth, region probe, editor sync customizations) through GetElevenLabsAPIKey()
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ElevenLabsComponent::ForceDisableConversation() — removed ActionName param. Now a pure conversation master-switch with no knowledge of actions. OnReadyForAction is fired by the action flow itself: PendingActionName is set when OnAgentActionRequested is broadcasted, then OnReadyForAction fires either after blend-out (if conversation was disabled) or on the next tick (if not).
- Rename bConversationDisabledByAction → bConversationForceDisabled — the old name implied coupling with actions which no longer exists.
- Add bPendingStartListening — when StartListening is called before the WebSocket is connected (race between StartConversation and StartListening from InteractionComponent), defer instead of silently failing. Consumed automatically in HandleConnected / ClientStartConversation. Cleared by StopListening / EndConversation.
- PerceptionComponent — demote the "perceived N actors but ALL filtered out" warning to Verbose; it's a normal case for civilians surrounded by non-hostile actors, not a misconfiguration.
- CLAUDE.md — workflow rule: always describe and request confirmation before code changes.
- Martin.uasset — agent config tweaks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Bulk rename Category/ClassGroup strings to ASTERION|PS_AI_ConvAgent|... and ASTERION|PS_AI_Behavior|... (284 occurrences, 24 files) so functions/properties/components group cleanly in Blueprint menus
- Move bAutoStartBehavior from AIController to PersonalityComponent so the same AIController class can drive multiple NPC archetypes with per-instance scripted-mode override
- Add PowerPoint user documentation for both plugins in Docs/ with generator scripts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Allow NPCs to start without a Behavior Tree and be controlled externally
via Blueprint, Level Sequences, or triggers. Adds Scripted enum state that
prevents BT services from overriding state, plus StartBehavior/StopBehavior
BlueprintCallable functions to toggle BT execution at runtime.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New CVar ps.ai.Behavior.Debug: -1=use bDebug property, 0=force off, 1=force on
- IsDebugEnabled() static helper checks CVar then PersonalityComponent.bDebug
- All diagnostic logs gated by IsDebugEnabled (no spam when debug off)
- Logs use Log level (visible) when enabled, silent when disabled
- Applied to: UpdateThreat, FindAndFollowSpline, FindCover, PerceptionComponent,
PersonalityComponent EvaluateReaction
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BodyExpressionComponent: don't activate on OnConversationConnected, wait for
OnSpeakingStarted (prevents body anims while NPC still walking)
- Demote recurring logs to Verbose: UpdateThreat per-tick, FindAndFollowSpline
debug, FindCover distance check, perception target scoring
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MicrophoneCaptureComponent:
- Local Voice Activity Detection (RMS-based, independent of ElevenLabs)
- Configurable threshold, onset time, silence time
- bIsUserSpeaking flag + OnUserVoiceActivityChanged delegate
- Hysteresis prevents flickering between speech/silence
AIController gaze bridge:
- Resolve MicComponent from player Pawn (not NPC) via reflection
- ConversationPaused BB key blocks movement branches via BT decorator
- NPC stops only when user actually speaks (not just on proximity connect)
- NPC resumes when conversation disconnects
- Spline PauseFollowing/ResumeFollowing on conversation start/end
BT setup required:
- Add Blackboard Condition (ConversationPaused Is Not Set, Aborts=Both)
on spline and patrol branches
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Revert conversation movement pause (needs local VAD, not bNetIsConversing)
- Remove temporary gaze proximity debug log
- Remove bConversationPaused flag (will be reimplemented with VAD)
- TODO: pause NPC movement only when local voice activity is detected
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New BTService_UpdateGaze bridges PS_AI_Behavior to PS_AI_ConvAgent's
GazeComponent via runtime reflection (zero compile dependency).
Priority system:
- Combat/TakingCover: gaze disabled (aim animation handles it)
- Alerted: look at ThreatActor (head + eyes, no body)
- Conversation: skip (ConvAgent manages)
- Proximity: glance at nearest perceived actor within radius
Proximity gaze features:
- Lock on target until release (no jumping)
- Configurable duration, cooldown, radius on AIController
- Front-facing only (dot product filter)
- Skip spectators and hostile actors
GazeComponent fix:
- Sync SmoothedBodyYaw to actual actor rotation when body tracking
is disabled, preventing stale head offset during spline movement
Files:
- New: BTService_UpdateGaze.h/.cpp
- Modified: AIController.h/.cpp (gaze bridge, config, bind, cleanup)
- Modified: GazeComponent.cpp (body yaw sync fix)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- GetTeamAttitudeTowards fallback (interface path) was calling MakeTeamId
without Faction, so enemies of different factions had identical TeamIds
→ always Friendly instead of Hostile
- Now reads Faction from PersonalityProfile when resolving via interface
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move ThreatDecayRate from global Settings to PersonalityProfile (per-archetype)
- Add state hysteresis in EvaluateReaction to prevent Fleeing/Combat flickering
- FindCover: verify distance on arrival, retry movement if too far
- FindAndFollowSpline: sample multiple spline points when closest is blocked
- BTTask_Attack: call BehaviorStartAttack immediately (draw weapon before LOS/range)
- CoverShootCycle: call BehaviorStartAttack on entry (weapon ready during cover approach)
- FindOwningPawn: walk AttachParentActor chain for ChildActor weapons without Owner
- Initialize PreferCover BB key to false at setup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove Loyalty and Discipline from EPS_AI_Behavior_TraitAxis enum (never used in gameplay)
- Clean up PersonalityProfile and PersonalityComponent default trait initialization
- Add descriptive tooltips to remaining traits (Courage, Aggressivity, Caution)
- Lower Aggressivity combat gate from 0.3 to 0.0 (only Aggressivity=0 prevents combat)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BTTask_Attack.h: remove reference to non-existent GetBehaviorOptimalAttackRange, document PersonalityProfile ranges
- AIController.h: update TeamId comment to reflect actual MakeTeamId encoding (nibble-based)
- CoverShootCycle.h: clarify that Peek/Cover durations are base values modulated by personality traits
- FindCover.h: clarify ManualPointBonus is additive score
- CombatComponent.h: clarify AttackRange/AttackCooldown are for ExecuteAttack, not BTTask_Attack
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add CombatCoverCycleDuration to PersonalityProfile (configurable base duration)
- PersonalityComponent cycles bPreferCover based on Aggressivity/Caution ratio
- Combat duration = CycleDuration × Aggressivity/(Aggressivity+Caution)
- Cover duration = CycleDuration × Caution/(Aggressivity+Caution)
- Min 2s per phase, ±20% jitter
- Write PreferCover bool to Blackboard for BT decorator observer aborts
- IsCoverNeeded decorator checks both target type AND ShouldPreferCover()
- Remove TakingCover state from EvaluateReaction — cover is now a sub-mode of Combat
- BT uses Blackboard Condition on PreferCover with Observer Aborts=Both
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PerceptionComponent: omniscient TActorIterator only runs with HasAuthority()
- PersonalityComponent: ApplyReaction and ForceState gate replicated CurrentState writes to server only
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- IsCoverNeeded used Cast<APawn> on ThreatActor which failed for AimTargetActors
→ always returned true (assume dangerous) → enemies took cover against civilians
- Fix: use FindOwningPawn to walk Owner/Instigator chain to the actual Pawn
- Revert inline civilian check in EvaluateReaction (decorator handles it in BT)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- AIController: auto uncrouch when leaving Fleeing/TakingCover state
- FleeFrom: uncrouch at start of flee (civilian was still crouched from HidingSpot)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix EQS SetScore filter bug: pass FloatValueMin/Max instead of hardcoded 0/1 (filters were ignored)
- Add LateralSpread to LineOfSight EQS test: multiple traces for wider LOS check
- Add bDrawDebug to CoverQuality and LineOfSight EQS tests
- Ignore ThreatActor collision in EQS traces (AimTargetActor was blocking its own LOS)
- Add navmesh projection in EQSContext_CoverLocation for cover points inside geometry
- Add omniscient awareness: enemies detect top-priority targets (Protectors) within sight radius without perception cone
- Suppress target switching during TakingCover state to prevent cover invalidation
- Fix flanking check: trace at chest height instead of feet
- Add debug visualization for EQS fallback paths (NO REFINE, NO FIRE POS, IN PLACE)
- Clean up diagnostic logs: verbose for per-trace EQS details, log level for summaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CoverShootCycle: SetFocus only during AtCover (not shooting), flanking detection every 0.5s abandons compromised cover
- CoverQuality EQS: ground hit filter, MaxNearbyHitDist (300cm), lateral spread traces, capsule-relative heights
- EQSContext_CoverLocation: prefer CoverPoint actor location over refined vector
- UpdateThreat: never abandon target during Combat/TakingCover states
- MoveTimeout (5s) on all movement states to prevent stuck NPCs
- FindCover: detailed refinement result logging
- Debug: Peeking timer log for investigating timer-stops-counting bug (needs rebuild + test)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CoverShootCycle: add FiringPositionQuery EQS for peek/shoot positions
(NPC moves between cover position and firing position with LOS)
- Add BTDecorator_IsCoverNeeded: skip cover when target is Civilian
- Add EQSContext_CoverLocation: provides BB CoverLocation to EQS generators
- FindCover: add debug draw toggle and EQS refinement debug spheres
- Definitions: add ECoverShootSubState and CoverPointType::HidingSpot
NOTE: Has compilation errors to fix (signature mismatches in
CoverShootCycle StartPeeking/ReturnToCover, missing forward-declare)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- SetBehaviorCrouch() interface function for cover/hiding crouch control
- CoverShootCycle: continuous LOS check during Peeking (stop if target hides)
- CoverShootCycle: crouch/stand transitions at all cover state changes
- CoverShootCycle: fail when no LOS and no advancing cover (falls to Attack)
- FindCover: crouch on arrival, stand up on abort
- FindCover: optional EQS RefinementQuery to refine exact position around CoverPoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- LastKnownTargetPosition check now filters FLT_MAX/InvalidLocation (not just zero)
- EQS result with AlreadyAtGoal triggers fallback advance instead of no-op
- bProjectGoal enabled for EQS move to handle NavMesh projection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- EQSTest_LineOfSight: new EQS test for firing position queries
- Updated plugin binaries (PS_AI_Behavior + PS_AI_ConvAgent)
- TeamComponent, Settings, Build.cs updates from LOS plan implementation
- PLAN.md updated with LOS combat implementation progress
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Attack/CoverShootCycle: de-escalate to Alerted on failure so decorator
can re-trigger when threat returns (fixes BT stuck on spline branch)
- UpdateThreat: keep BB ThreatActor during brief perception gaps instead
of clearing immediately (use LOS timeout for graceful degradation)
- HasLineOfSight: ignore actors up the attachment chain so trace doesn't
hit the target Pawn's capsule when aiming at its AimTarget child actor
- NoTeam actors (spectators, editor pawns) treated as Neutral instead of
Hostile, plus SpectatorPawn explicit filter in perception
- BB debug key ThreatPawnName shows owning Pawn name (resolved via
perception's LastThreatOwningPawn) instead of cryptic AimTarget name
- FindOwningPawn promoted to public static on PerceptionComponent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split ResolveToPawn into FindOwningPawn + GetThreatTarget so non-Pawn
actors (PS_AimTargetActor) are properly resolved for team/attitude checks
while still being used as BB target. Add attack range hysteresis (10%
buffer), target persistence (80% threshold), melee no-cooldown chase,
ranged midpoint approach. New files: CoverShootCycle task, CheckCombatType
decorator, MinAttackRange/MaxAttackRange in PersonalityProfile.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add TeamComponent for player pawn team identity (Role-based: Civilian/Enemy/Protector)
- Add IsTargetActorValid to interface for dead target filtering
- Fix GetTeamAttitudeTowards to check IGenericTeamAgentInterface + TeamComponent
- Guarantee BehaviorStopAttack via OnTaskFinished (all exit paths)
- Prevent Combat state without ThreatActor (stay Alerted until perception catches up)
- Resume spline at closest point from current position after combat
- Sync CurrentSpline and SplineProgress to Blackboard in FollowSpline tick
- Auto-detect Patrol state when NPC has a spline (fixes Idle speed=0 blocking movement)
- Add per-component debug toggles (Personality + SplineFollower independent)
- Use AddMovementInput instead of RequestDirectMove for reliable post-combat movement
- Add bTickInEditor for Personality debug in Simulate mode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
