The remaining pops were per-MESSAGE, not per-sub-chunk: the Opus encoder
processes whole 640-byte frames only, so each ElevenLabs message dropped its
<1-frame (<20ms) tail. The lost ~10-16ms per message left a waveform
discontinuity at each message splice -> audible click (2-3 per sentence).
Confirmed by log (message sizes not 640-aligned; no underflow) and the
standalone-vs-client asymmetry: both use the same playback queue, but the host
plays raw PCM (no drop) and is clean while clients play the Opus-encoded stream
and pop.
Fix (server-only path): carry the sub-frame remainder to the next message.
Prepend the previous message's leftover before encoding, emit only whole frames,
and stash the new tail. No samples dropped -> continuous stream -> no splice
click. Leftover reset per turn (new-turn guard + StopAgentAudio).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After sub-chunking each ElevenLabs message into N ~0.8s Opus packets, the
client enqueued each sub-chunk separately, re-running the playback state
machine per sub-chunk (spurious Play() restart / pre-buffer race / underrun
at the seam) -> an audible pop at each ~0.8s boundary (2-3 per sentence).
Confirmed via multi-agent analysis + logs (no generation skips, no decode
truncation -> not a codec issue; the discontinuity is on the enqueue side).
- MulticastReceiveAgentAudio gains a bLastSubChunk flag; the server marks the
final sub-chunk of each message (encode path frame-aligned so the last
sub-chunk is well-defined; raw-PCM fallback likewise).
- The client accumulates decoded PCM across a message's sub-chunks and calls
EnqueueAgentAudio (and lip-sync broadcast) ONCE, on the last sub-chunk.
- ClientReassemblyPCM reset in StopAgentAudio so a half-received message can't
leak across a turn/interruption.
No added latency: a message's sub-chunks are sent back-to-back in one server
tick and arrive together. Each RPC still <=40 frames (Reliable, under the
encode/decode limits).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sending Opus over Unreliable multicast caused missing words and crackling:
Opus is a stateful inter-frame codec, so a dropped or reordered packet
desyncs the decoder and corrupts all following frames until the stream
resets. Unreliable was originally adopted only to stop a reliable-buffer
overflow — but that overflow came from 32-152KB raw-PCM RPCs (pre-Opus).
With Opus + sub-chunking each RPC is ~2KB (2-3 partial bunches) at
~1.25 RPC/s, so the 512-bunch reliable buffer stays in single digits and
cannot overflow. Reliable restores ordered, lossless delivery.
- MulticastReceiveAgentAudio: Unreliable -> Reliable.
- Keep the 40-frame (0.8s, ~2KB) sub-chunking — it is what keeps packets
small enough for Reliable to be safe.
- Harden the raw-PCM fallback (only reached if Opus init fails): hard-cap
one message at 320KB and drop the tail so it can never overflow the
reliable buffer, and warn on every fallback (config bug: [Voice]).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Long ElevenLabs audio messages were cut off on remote clients:
- the client Opus decode buffer was only 32000 bytes (~0.88s), so the
engine decoder aborted with "Decompression buffer too small" and
dropped the tail of any packet longer than that;
- the server also encoded a whole message as a single Opus packet, which
the engine encoder caps at 76 frames (~1.52s), silently dropping the rest.
The host/authority plays raw PCM and never decodes, so standalone and
listen-server hosts were unaffected, which masked the bug.
Fix (HandleAudioReceived + MulticastReceiveAgentAudio_Implementation):
- sub-chunk the Opus encode into <=40-frame (25600-byte, 0.8s) blocks,
one RPC each: stays under both the 76-frame encode cap and the client
decode buffer, and keeps each Unreliable RPC small;
- raise the client decode buffer 32000 -> 64*1024 as headroom.
Empirically confirmed: networked client logged 16x "Decompression buffer
too small" with Opus active; standalone (Opus equally active) logged zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- MulticastReceiveAgentAudio: Reliable -> Unreliable. Realtime audio: a lost or
oversized packet is dropped (minor glitch) instead of backing up the reliable
buffer and dropping the client connection.
- Raw-PCM fallback chunk size 32000 -> 4000 bytes so an unreliable RPC is not
auto-promoted to reliable when fragmented into many partial bunches (which
would reintroduce the overflow).
- Add net-audio size log gated behind ps.ai.ConvAgent.Debug.ElevenLabs; fires on
Authority even in solo PIE: "[NET-SRV] Opus audio chunk: N bytes (raw M -> Xx)".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The project had no [Voice] section, so the engine default bEnabled=false left
FVoiceModule::CreateVoiceEncoder() returning null (Encoder/Decoder NULL).
Networked agent audio then fell back to raw PCM in large reliable multicast
RPCs, overflowing the reliable buffer and dropping VR client connections the
moment the avatar spoke. Enabling Voice lets Opus compress (~13-20x), keeping
RPCs small.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A client that joins while a conversation is already active can run OnRep with
NetConnectedPawns/NetActiveSpeakerPawn entries whose replicated pawn references
haven't resolved yet (actor channel not open). NetConnectedPawns.Num() > 0 did
not guarantee a non-null element, so OnRep_ConversationState dereferenced a null
pawn in its log (crash in Dev/PIE) and ApplyConversationGaze set TargetActor to
null (gaze glitch). Both now skip until the reference resolves — OnRep fires
again once it's patched in. More likely with 3+ clients.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Single-player clients sometimes showed the head turned ~60° off while the
server looked correct. Two client-only causes:
1. The client overwrote the actor's rotation every frame (SetActorRotation)
and then read back its own output as the "replicated" yaw, so SmoothedBodyYaw
diverged from the true server body yaw.
2. The head/eye cascade was sticky / path-dependent (TargetHeadYaw updated on
overflow events), so running it independently on a client fed by lagged,
stepped replication latched into a different state — the head overflowed to
MaxHeadYaw and stayed there.
Rewrite head/eyes as a STATELESS function of the actual (authoritative on the
server, replicated on the client) body facing + target direction. No sticky
state, FInterpTo only eases the visual. The client no longer overrides the actor
rotation. Body turn stays authority-only (lazy turn beyond head+eyes range).
Server and client now converge to the same pose by construction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With an open mic streaming continuously (Server VAD + interruption), every
chunk refreshed the active speaker's LastSpeakTime, so SpeakerSwitchHysteresis
never elapsed and SpeakerIdleTimeout never fired — the first speaker locked the
floor forever and the agent never turned to anyone else.
Now only chunks with real voice energy (RMS >= 0.02) hold/take the floor: the
active speaker is still forwarded continuously (silence tail for VAD), but their
lock only persists while they actually speak. When they pause, another player's
voiced audio can switch in and the agent turns to them (NetActiveSpeakerPawn
replicated -> gaze). Validated with two players on one NPC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Client mic uplink confirmed working in a packaged build across two PCs
(networked Server VAD): the agent now replies to the player's voice.
Remove the [MIC-DBG]/[MIC-DBG-SRV] instrumentation added to trace it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Networked clients could hear the agent but the agent never replied to them
(worked in standalone/listen-server host). Root causes, client-only paths
that never run on the authority:
1. Deferred StartListening was never retried once the conversation went live.
OnRep_ConversationState now (re)starts listening when bNetIsConversing
replicates and the local player is in the conversation — closes the
RPC/replication ordering race so bIsListening reliably opens the uplink.
2. The client stripped ALL silent mic chunks to save bandwidth, but ElevenLabs
Server VAD needs the trailing silence to detect end-of-turn — so the agent
heard the words but never the pause and never responded. Replaced the hard
silence drop with a silence "tail": keep streaming for MicSilenceTailSeconds
(default 2s, tunable) after the last voiced chunk, then drop during idle.
The authority path is unchanged (always streams the full mic).
Also: debug HUD now shows "CONNECTED (via server)" on clients instead of a
misleading "DISCONNECTED" (clients have no local WebSocket; IsConnected() is
always false there).
[MIC-DBG] temporary diagnostics are kept in this commit to validate the fix on
another machine; they will be removed in a follow-up once confirmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-enables agent/tool/action-set authoring in the editor after the API
key was removed from Project Settings. Adds an editor-only key source so
it can never be committed-as-config or baked into a packaged build.
- New UPS_AI_ConvAgent_EditorSettings_ElevenLabs (UDeveloperSettings,
config=EditorPerProjectUserSettings, Editor category): single global
field, stored under Saved/Config (never cooked), in the editor module
(never packaged).
- ResolveEditorApiKey(): editor setting -> ELEVENLABS_API_KEY env -> runtime
SaveGame fallback. All three editor customizations route through it.
- Fix the now-stale "API Key not set in Project Settings" messages.
Runtime path is untouched: packaged game still uses the client-provided
key via the SaveGame (Set/GetElevenLabsAPIKey).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Auto pause now triggers as soon as a conversation is genuinely active (agent greeting included) instead of waiting for the user to speak — the NPC no longer walks away during its opening line.
Add DisableBehaviorMovement()/EnableBehaviorMovement() (BlueprintCallable): freeze the NPC's autonomous movement while keeping the Behavior Tree running (perception, threat, gaze, reactions). Movement is paused when EITHER manually disabled OR a conversation is active. Both drive the ConversationPaused Blackboard flag read by the movement-branch decorators; leave the Flee branch un-decorated so fleeing still works.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
StopFollowing() now sets a bManuallyStopped flag (cleared only by an explicit StartFollowing/StartFollowingAtDistance). The FindAndFollowSpline and FollowSpline BT tasks bail while it is set, so the BT no longer auto-resumes a manual stop on the next tick. ResumeFollowing() also respects the flag. PauseFollowing() is unchanged (BT may resume).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds on the client-provided-key SaveGame work to actually secure usage:
- Private agents: connect via a short-lived Signed URL fetched with the stored key (GET /v1/convai/conversation/get-signed-url), instead of a direct agent_id+header connection which ElevenLabs rejects for auth-enabled agents. Falls back to a direct connection only when no key is set (public agents). Enforces 'valid key required': no key -> no signed URL -> no conversation.
- Force agent creation/update to private (platform_settings.auth.enable_auth=true) so a public, key-less agent can't be created by mistake.
- Encrypt the API key at rest in the SaveGame (AES-256); plaintext kept only in the in-memory cache.
- Redact signed-URL query (auth token) from logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The API key is no longer stored in project settings / DefaultEngine.ini (it was committed and baked into shipped builds). It is now provided per-install by the client at runtime and persisted in a SaveGame slot.
- Add UPS_AI_ConvAgent_SaveGame_ElevenLabs (slot PS_AI_ConvAgent_ElevenLabs)
- BlueprintLibrary is the single owner of the key: Set/Get/Has/Clear + in-memory cache backed by the SaveGame
- Add async BP node 'Test ElevenLabs API Key' (GET /v1/models)
- Remove API_Key UPROPERTY from UPS_AI_ConvAgent_Settings_ElevenLabs and the key logic from the module
- Route all consumers (WebSocket auth, region probe, editor sync customizations) through GetElevenLabsAPIKey()
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ElevenLabsComponent::ForceDisableConversation() — removed ActionName param. Now a pure conversation master-switch with no knowledge of actions. OnReadyForAction is fired by the action flow itself: PendingActionName is set when OnAgentActionRequested is broadcasted, then OnReadyForAction fires either after blend-out (if conversation was disabled) or on the next tick (if not).
- Rename bConversationDisabledByAction → bConversationForceDisabled — the old name implied coupling with actions which no longer exists.
- Add bPendingStartListening — when StartListening is called before the WebSocket is connected (race between StartConversation and StartListening from InteractionComponent), defer instead of silently failing. Consumed automatically in HandleConnected / ClientStartConversation. Cleared by StopListening / EndConversation.
- PerceptionComponent — demote the "perceived N actors but ALL filtered out" warning to Verbose; it's a normal case for civilians surrounded by non-hostile actors, not a misconfiguration.
- CLAUDE.md — workflow rule: always describe and request confirmation before code changes.
- Martin.uasset — agent config tweaks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Bulk rename Category/ClassGroup strings to ASTERION|PS_AI_ConvAgent|... and ASTERION|PS_AI_Behavior|... (284 occurrences, 24 files) so functions/properties/components group cleanly in Blueprint menus
- Move bAutoStartBehavior from AIController to PersonalityComponent so the same AIController class can drive multiple NPC archetypes with per-instance scripted-mode override
- Add PowerPoint user documentation for both plugins in Docs/ with generator scripts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Allow NPCs to start without a Behavior Tree and be controlled externally
via Blueprint, Level Sequences, or triggers. Adds Scripted enum state that
prevents BT services from overriding state, plus StartBehavior/StopBehavior
BlueprintCallable functions to toggle BT execution at runtime.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New CVar ps.ai.Behavior.Debug: -1=use bDebug property, 0=force off, 1=force on
- IsDebugEnabled() static helper checks CVar then PersonalityComponent.bDebug
- All diagnostic logs gated by IsDebugEnabled (no spam when debug off)
- Logs use Log level (visible) when enabled, silent when disabled
- Applied to: UpdateThreat, FindAndFollowSpline, FindCover, PerceptionComponent,
PersonalityComponent EvaluateReaction
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BodyExpressionComponent: don't activate on OnConversationConnected, wait for
OnSpeakingStarted (prevents body anims while NPC still walking)
- Demote recurring logs to Verbose: UpdateThreat per-tick, FindAndFollowSpline
debug, FindCover distance check, perception target scoring
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MicrophoneCaptureComponent:
- Local Voice Activity Detection (RMS-based, independent of ElevenLabs)
- Configurable threshold, onset time, silence time
- bIsUserSpeaking flag + OnUserVoiceActivityChanged delegate
- Hysteresis prevents flickering between speech/silence
AIController gaze bridge:
- Resolve MicComponent from player Pawn (not NPC) via reflection
- ConversationPaused BB key blocks movement branches via BT decorator
- NPC stops only when user actually speaks (not just on proximity connect)
- NPC resumes when conversation disconnects
- Spline PauseFollowing/ResumeFollowing on conversation start/end
BT setup required:
- Add Blackboard Condition (ConversationPaused Is Not Set, Aborts=Both)
on spline and patrol branches
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Revert conversation movement pause (needs local VAD, not bNetIsConversing)
- Remove temporary gaze proximity debug log
- Remove bConversationPaused flag (will be reimplemented with VAD)
- TODO: pause NPC movement only when local voice activity is detected
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New BTService_UpdateGaze bridges PS_AI_Behavior to PS_AI_ConvAgent's
GazeComponent via runtime reflection (zero compile dependency).
Priority system:
- Combat/TakingCover: gaze disabled (aim animation handles it)
- Alerted: look at ThreatActor (head + eyes, no body)
- Conversation: skip (ConvAgent manages)
- Proximity: glance at nearest perceived actor within radius
Proximity gaze features:
- Lock on target until release (no jumping)
- Configurable duration, cooldown, radius on AIController
- Front-facing only (dot product filter)
- Skip spectators and hostile actors
GazeComponent fix:
- Sync SmoothedBodyYaw to actual actor rotation when body tracking
is disabled, preventing stale head offset during spline movement
Files:
- New: BTService_UpdateGaze.h/.cpp
- Modified: AIController.h/.cpp (gaze bridge, config, bind, cleanup)
- Modified: GazeComponent.cpp (body yaw sync fix)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- GetTeamAttitudeTowards fallback (interface path) was calling MakeTeamId
without Faction, so enemies of different factions had identical TeamIds
→ always Friendly instead of Hostile
- Now reads Faction from PersonalityProfile when resolving via interface
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move ThreatDecayRate from global Settings to PersonalityProfile (per-archetype)
- Add state hysteresis in EvaluateReaction to prevent Fleeing/Combat flickering
- FindCover: verify distance on arrival, retry movement if too far
- FindAndFollowSpline: sample multiple spline points when closest is blocked
- BTTask_Attack: call BehaviorStartAttack immediately (draw weapon before LOS/range)
- CoverShootCycle: call BehaviorStartAttack on entry (weapon ready during cover approach)
- FindOwningPawn: walk AttachParentActor chain for ChildActor weapons without Owner
- Initialize PreferCover BB key to false at setup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove Loyalty and Discipline from EPS_AI_Behavior_TraitAxis enum (never used in gameplay)
- Clean up PersonalityProfile and PersonalityComponent default trait initialization
- Add descriptive tooltips to remaining traits (Courage, Aggressivity, Caution)
- Lower Aggressivity combat gate from 0.3 to 0.0 (only Aggressivity=0 prevents combat)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BTTask_Attack.h: remove reference to non-existent GetBehaviorOptimalAttackRange, document PersonalityProfile ranges
- AIController.h: update TeamId comment to reflect actual MakeTeamId encoding (nibble-based)
- CoverShootCycle.h: clarify that Peek/Cover durations are base values modulated by personality traits
- FindCover.h: clarify ManualPointBonus is additive score
- CombatComponent.h: clarify AttackRange/AttackCooldown are for ExecuteAttack, not BTTask_Attack
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add CombatCoverCycleDuration to PersonalityProfile (configurable base duration)
- PersonalityComponent cycles bPreferCover based on Aggressivity/Caution ratio
- Combat duration = CycleDuration × Aggressivity/(Aggressivity+Caution)
- Cover duration = CycleDuration × Caution/(Aggressivity+Caution)
- Min 2s per phase, ±20% jitter
- Write PreferCover bool to Blackboard for BT decorator observer aborts
- IsCoverNeeded decorator checks both target type AND ShouldPreferCover()
- Remove TakingCover state from EvaluateReaction — cover is now a sub-mode of Combat
- BT uses Blackboard Condition on PreferCover with Observer Aborts=Both
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PerceptionComponent: omniscient TActorIterator only runs with HasAuthority()
- PersonalityComponent: ApplyReaction and ForceState gate replicated CurrentState writes to server only
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- IsCoverNeeded used Cast<APawn> on ThreatActor which failed for AimTargetActors
→ always returned true (assume dangerous) → enemies took cover against civilians
- Fix: use FindOwningPawn to walk Owner/Instigator chain to the actual Pawn
- Revert inline civilian check in EvaluateReaction (decorator handles it in BT)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- AIController: auto uncrouch when leaving Fleeing/TakingCover state
- FleeFrom: uncrouch at start of flee (civilian was still crouched from HidingSpot)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix EQS SetScore filter bug: pass FloatValueMin/Max instead of hardcoded 0/1 (filters were ignored)
- Add LateralSpread to LineOfSight EQS test: multiple traces for wider LOS check
- Add bDrawDebug to CoverQuality and LineOfSight EQS tests
- Ignore ThreatActor collision in EQS traces (AimTargetActor was blocking its own LOS)
- Add navmesh projection in EQSContext_CoverLocation for cover points inside geometry
- Add omniscient awareness: enemies detect top-priority targets (Protectors) within sight radius without perception cone
- Suppress target switching during TakingCover state to prevent cover invalidation
- Fix flanking check: trace at chest height instead of feet
- Add debug visualization for EQS fallback paths (NO REFINE, NO FIRE POS, IN PLACE)
- Clean up diagnostic logs: verbose for per-trace EQS details, log level for summaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CoverShootCycle: SetFocus only during AtCover (not shooting), flanking detection every 0.5s abandons compromised cover
- CoverQuality EQS: ground hit filter, MaxNearbyHitDist (300cm), lateral spread traces, capsule-relative heights
- EQSContext_CoverLocation: prefer CoverPoint actor location over refined vector
- UpdateThreat: never abandon target during Combat/TakingCover states
- MoveTimeout (5s) on all movement states to prevent stuck NPCs
- FindCover: detailed refinement result logging
- Debug: Peeking timer log for investigating timer-stops-counting bug (needs rebuild + test)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CoverShootCycle: add FiringPositionQuery EQS for peek/shoot positions
(NPC moves between cover position and firing position with LOS)
- Add BTDecorator_IsCoverNeeded: skip cover when target is Civilian
- Add EQSContext_CoverLocation: provides BB CoverLocation to EQS generators
- FindCover: add debug draw toggle and EQS refinement debug spheres
- Definitions: add ECoverShootSubState and CoverPointType::HidingSpot
NOTE: Has compilation errors to fix (signature mismatches in
CoverShootCycle StartPeeking/ReturnToCover, missing forward-declare)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- SetBehaviorCrouch() interface function for cover/hiding crouch control
- CoverShootCycle: continuous LOS check during Peeking (stop if target hides)
- CoverShootCycle: crouch/stand transitions at all cover state changes
- CoverShootCycle: fail when no LOS and no advancing cover (falls to Attack)
- FindCover: crouch on arrival, stand up on abort
- FindCover: optional EQS RefinementQuery to refine exact position around CoverPoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
