On a player barge-in, the engaged agent stops (its mic feeds the player's voice to ElevenLabs, which interrupts it), but the witness agent never hears the player and kept finishing its line (e.g. '...I'll give you the cash...') mid-scene. InterruptForPlayer now calls InterruptAgent() on both CompA and CompB. Conversations stay open; the witness still receives the contextual_updates.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A first-message greeting opens the agent segment but never fires OnAgentStoppedSpeaking (no agent_response), so the segment stayed open and the later reply was appended to it — inheriting the greeting's early StartTime and playing before the player's question on replay.
RecordAgentAudio now flushes the open segment and starts a fresh one when audio resumes after a >1.5s gap, so the greeting keeps its time and the reply gets its own (correct, later) timestamp. Also flush open agent segments at record stop. Added record-side diagnostic logs behind ps.convreplay.Debug (agent-START/STOP/SPLIT, player-SEG, user-TEXT).
Record-side fix: existing sidecars keep the old segmentation; re-record to benefit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Record every player's mic into the replay even with no agent in the scene (listen-server/standalone). The bridge arms each player pawn's InteractionComponent via a replicated flag; the host captures locally, remote clients relay via a new ServerRecordVoice RPC, and the server broadcasts OnPlayerVoiceForRecording into the existing player-audio pipeline. Agent and agent-less paths are mutually exclusive (no double-record).
Replay routing: match a recorded agent by id, else by nearest position (runtime-spawned NPCs get unstable names on replay), keeping lip-sync.
Player audio: per-utterance bidirectional AGC toward a target RMS with an envelope peak limiter (instant attack, ~9ms release) instead of hard clipping. Live CVars: ps.convreplay.PlayerTargetRms (default 0.08), ps.convreplay.PlayerGain, ps.convreplay.Debug (per-record logs off by default).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New PS_AI_ConvReplay bridge plugin records ElevenLabs agent TTS and player mic audio (Opus) into a sidecar next to the .replay, and plays it back during DemoPlay with lip-sync / body-expression / gaze / emotion. Agent audio is re-fed through the agent's own voice; player audio is spatialized at the recorded position.
ConvAgent hooks (generic): PlayExternalAudioPCM, StopExternalAudio, TriggerReplayedAction/TriggerReplayedClientToolCall, and an OnPlayerAudioCaptured delegate broadcast in ServerSendMicAudioFromPlayer.
Agent matching is id-or-nearest-position (runtime-spawned NPCs get unstable names on replay). Player level normalized per-utterance (bidirectional AGC + soft limiter), tunable live via ps.convreplay.PlayerGain.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The agent-to-agent dialogue interrupt was detected only in FeedExternalAudio,
which runs client-side for a remote player. On a DEDICATED server (no host
player) no one could trigger the reliable VAD interrupt -- everyone fell back to
the slower ElevenLabs transcript path.
Also broadcast OnUserVoiceDetected from ServerSendMicAudioFromPlayer, where the
player's mic (the host's OR a remote client's, relayed) always arrives on the
server. Shares the LastUserVoiceOnset debounce so the host does not double-fire.
The orchestrator still attributes the correct player (ResolveSpeakingPlayer
finds the connected client via NetConnectedPawns).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New APS_AI_ConvAgent_AgentDialogue actor drives two ElevenLabs agents in a
text-only conversation (they speak aloud and look at each other), and lets a
player barge in and take over.
Orchestrator (AgentDialogue):
- Single-floor agent<->agent turn-taking over text (no STT); gaze override so
the agents look at each other; TurnDelaySeconds for pacing.
- Voice-onset interrupt: the player is detected via a local mic VAD
(OnUserVoiceDetected), not the unreliable STT round-trip. On interrupt both
agents turn to the player; auto-released once the player leaves both agents.
- 3-way "floor" mode: the engaged agent is reframed ("a real person now
addresses you") while the witness agent overhears the exchange via
contextual_update, so it can reference it when the player later turns to it.
Configurable messages: Interrupt/Bystander context + overheard prefix.
Gaze: honour GazeOverrideTarget on every path that could steal the gaze back to
the player -- OnRep_ActiveSpeaker, InteractionComponent AttachGazeTarget /
SetSelectedAgent, and the PS_AI_Behavior proximity gaze (IsConversationActiveOnPawn
now treats an agent-to-agent dialogue as an active conversation).
Conversation lifecycle: a watching player joining/leaving by look no longer
tears down an orchestrated agent's WebSocket (ServerLeaveConversation guarded by
GazeOverrideTarget) -- fixes agents going silent / lip sync deactivating when
the player looks away.
ElevenLabs: add SendContextualUpdate (contextual_update -- inject context with no
reply triggered) on the component and WebSocket proxy; add OnUserVoiceDetected
local voice-onset event fired from FeedExternalAudio.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Diagnostic for the audio "pops": EnqueueAgentAudio compares the last sample of
the previous enqueued block with the first sample of the new one and logs
[AUDIO-DBG] with the delta + a running count when the jump exceeds a threshold
(gated behind ps.ai.ConvAgent.Debug.ElevenLabs; role-tagged). The host's raw-PCM
stream is continuous so its count stays ~0; a client whose count climbs at
message cadence still has a splice problem. Confirms the carry-over fix
(bcb6c44) at runtime.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The remaining pops were per-MESSAGE, not per-sub-chunk: the Opus encoder
processes whole 640-byte frames only, so each ElevenLabs message dropped its
<1-frame (<20ms) tail. The lost ~10-16ms per message left a waveform
discontinuity at each message splice -> audible click (2-3 per sentence).
Confirmed by log (message sizes not 640-aligned; no underflow) and the
standalone-vs-client asymmetry: both use the same playback queue, but the host
plays raw PCM (no drop) and is clean while clients play the Opus-encoded stream
and pop.
Fix (server-only path): carry the sub-frame remainder to the next message.
Prepend the previous message's leftover before encoding, emit only whole frames,
and stash the new tail. No samples dropped -> continuous stream -> no splice
click. Leftover reset per turn (new-turn guard + StopAgentAudio).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After sub-chunking each ElevenLabs message into N ~0.8s Opus packets, the
client enqueued each sub-chunk separately, re-running the playback state
machine per sub-chunk (spurious Play() restart / pre-buffer race / underrun
at the seam) -> an audible pop at each ~0.8s boundary (2-3 per sentence).
Confirmed via multi-agent analysis + logs (no generation skips, no decode
truncation -> not a codec issue; the discontinuity is on the enqueue side).
- MulticastReceiveAgentAudio gains a bLastSubChunk flag; the server marks the
final sub-chunk of each message (encode path frame-aligned so the last
sub-chunk is well-defined; raw-PCM fallback likewise).
- The client accumulates decoded PCM across a message's sub-chunks and calls
EnqueueAgentAudio (and lip-sync broadcast) ONCE, on the last sub-chunk.
- ClientReassemblyPCM reset in StopAgentAudio so a half-received message can't
leak across a turn/interruption.
No added latency: a message's sub-chunks are sent back-to-back in one server
tick and arrive together. Each RPC still <=40 frames (Reliable, under the
encode/decode limits).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Sending Opus over Unreliable multicast caused missing words and crackling:
Opus is a stateful inter-frame codec, so a dropped or reordered packet
desyncs the decoder and corrupts all following frames until the stream
resets. Unreliable was originally adopted only to stop a reliable-buffer
overflow — but that overflow came from 32-152KB raw-PCM RPCs (pre-Opus).
With Opus + sub-chunking each RPC is ~2KB (2-3 partial bunches) at
~1.25 RPC/s, so the 512-bunch reliable buffer stays in single digits and
cannot overflow. Reliable restores ordered, lossless delivery.
- MulticastReceiveAgentAudio: Unreliable -> Reliable.
- Keep the 40-frame (0.8s, ~2KB) sub-chunking — it is what keeps packets
small enough for Reliable to be safe.
- Harden the raw-PCM fallback (only reached if Opus init fails): hard-cap
one message at 320KB and drop the tail so it can never overflow the
reliable buffer, and warn on every fallback (config bug: [Voice]).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Long ElevenLabs audio messages were cut off on remote clients:
- the client Opus decode buffer was only 32000 bytes (~0.88s), so the
engine decoder aborted with "Decompression buffer too small" and
dropped the tail of any packet longer than that;
- the server also encoded a whole message as a single Opus packet, which
the engine encoder caps at 76 frames (~1.52s), silently dropping the rest.
The host/authority plays raw PCM and never decodes, so standalone and
listen-server hosts were unaffected, which masked the bug.
Fix (HandleAudioReceived + MulticastReceiveAgentAudio_Implementation):
- sub-chunk the Opus encode into <=40-frame (25600-byte, 0.8s) blocks,
one RPC each: stays under both the 76-frame encode cap and the client
decode buffer, and keeps each Unreliable RPC small;
- raise the client decode buffer 32000 -> 64*1024 as headroom.
Empirically confirmed: networked client logged 16x "Decompression buffer
too small" with Opus active; standalone (Opus equally active) logged zero.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- MulticastReceiveAgentAudio: Reliable -> Unreliable. Realtime audio: a lost or
oversized packet is dropped (minor glitch) instead of backing up the reliable
buffer and dropping the client connection.
- Raw-PCM fallback chunk size 32000 -> 4000 bytes so an unreliable RPC is not
auto-promoted to reliable when fragmented into many partial bunches (which
would reintroduce the overflow).
- Add net-audio size log gated behind ps.ai.ConvAgent.Debug.ElevenLabs; fires on
Authority even in solo PIE: "[NET-SRV] Opus audio chunk: N bytes (raw M -> Xx)".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The project had no [Voice] section, so the engine default bEnabled=false left
FVoiceModule::CreateVoiceEncoder() returning null (Encoder/Decoder NULL).
Networked agent audio then fell back to raw PCM in large reliable multicast
RPCs, overflowing the reliable buffer and dropping VR client connections the
moment the avatar spoke. Enabling Voice lets Opus compress (~13-20x), keeping
RPCs small.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A client that joins while a conversation is already active can run OnRep with
NetConnectedPawns/NetActiveSpeakerPawn entries whose replicated pawn references
haven't resolved yet (actor channel not open). NetConnectedPawns.Num() > 0 did
not guarantee a non-null element, so OnRep_ConversationState dereferenced a null
pawn in its log (crash in Dev/PIE) and ApplyConversationGaze set TargetActor to
null (gaze glitch). Both now skip until the reference resolves — OnRep fires
again once it's patched in. More likely with 3+ clients.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Single-player clients sometimes showed the head turned ~60° off while the
server looked correct. Two client-only causes:
1. The client overwrote the actor's rotation every frame (SetActorRotation)
and then read back its own output as the "replicated" yaw, so SmoothedBodyYaw
diverged from the true server body yaw.
2. The head/eye cascade was sticky / path-dependent (TargetHeadYaw updated on
overflow events), so running it independently on a client fed by lagged,
stepped replication latched into a different state — the head overflowed to
MaxHeadYaw and stayed there.
Rewrite head/eyes as a STATELESS function of the actual (authoritative on the
server, replicated on the client) body facing + target direction. No sticky
state, FInterpTo only eases the visual. The client no longer overrides the actor
rotation. Body turn stays authority-only (lazy turn beyond head+eyes range).
Server and client now converge to the same pose by construction.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
With an open mic streaming continuously (Server VAD + interruption), every
chunk refreshed the active speaker's LastSpeakTime, so SpeakerSwitchHysteresis
never elapsed and SpeakerIdleTimeout never fired — the first speaker locked the
floor forever and the agent never turned to anyone else.
Now only chunks with real voice energy (RMS >= 0.02) hold/take the floor: the
active speaker is still forwarded continuously (silence tail for VAD), but their
lock only persists while they actually speak. When they pause, another player's
voiced audio can switch in and the agent turns to them (NetActiveSpeakerPawn
replicated -> gaze). Validated with two players on one NPC.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Client mic uplink confirmed working in a packaged build across two PCs
(networked Server VAD): the agent now replies to the player's voice.
Remove the [MIC-DBG]/[MIC-DBG-SRV] instrumentation added to trace it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Networked clients could hear the agent but the agent never replied to them
(worked in standalone/listen-server host). Root causes, client-only paths
that never run on the authority:
1. Deferred StartListening was never retried once the conversation went live.
OnRep_ConversationState now (re)starts listening when bNetIsConversing
replicates and the local player is in the conversation — closes the
RPC/replication ordering race so bIsListening reliably opens the uplink.
2. The client stripped ALL silent mic chunks to save bandwidth, but ElevenLabs
Server VAD needs the trailing silence to detect end-of-turn — so the agent
heard the words but never the pause and never responded. Replaced the hard
silence drop with a silence "tail": keep streaming for MicSilenceTailSeconds
(default 2s, tunable) after the last voiced chunk, then drop during idle.
The authority path is unchanged (always streams the full mic).
Also: debug HUD now shows "CONNECTED (via server)" on clients instead of a
misleading "DISCONNECTED" (clients have no local WebSocket; IsConnected() is
always false there).
[MIC-DBG] temporary diagnostics are kept in this commit to validate the fix on
another machine; they will be removed in a follow-up once confirmed.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Re-enables agent/tool/action-set authoring in the editor after the API
key was removed from Project Settings. Adds an editor-only key source so
it can never be committed-as-config or baked into a packaged build.
- New UPS_AI_ConvAgent_EditorSettings_ElevenLabs (UDeveloperSettings,
config=EditorPerProjectUserSettings, Editor category): single global
field, stored under Saved/Config (never cooked), in the editor module
(never packaged).
- ResolveEditorApiKey(): editor setting -> ELEVENLABS_API_KEY env -> runtime
SaveGame fallback. All three editor customizations route through it.
- Fix the now-stale "API Key not set in Project Settings" messages.
Runtime path is untouched: packaged game still uses the client-provided
key via the SaveGame (Set/GetElevenLabsAPIKey).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Auto pause now triggers as soon as a conversation is genuinely active (agent greeting included) instead of waiting for the user to speak — the NPC no longer walks away during its opening line.
Add DisableBehaviorMovement()/EnableBehaviorMovement() (BlueprintCallable): freeze the NPC's autonomous movement while keeping the Behavior Tree running (perception, threat, gaze, reactions). Movement is paused when EITHER manually disabled OR a conversation is active. Both drive the ConversationPaused Blackboard flag read by the movement-branch decorators; leave the Flee branch un-decorated so fleeing still works.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
StopFollowing() now sets a bManuallyStopped flag (cleared only by an explicit StartFollowing/StartFollowingAtDistance). The FindAndFollowSpline and FollowSpline BT tasks bail while it is set, so the BT no longer auto-resumes a manual stop on the next tick. ResumeFollowing() also respects the flag. PauseFollowing() is unchanged (BT may resume).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds on the client-provided-key SaveGame work to actually secure usage:
- Private agents: connect via a short-lived Signed URL fetched with the stored key (GET /v1/convai/conversation/get-signed-url), instead of a direct agent_id+header connection which ElevenLabs rejects for auth-enabled agents. Falls back to a direct connection only when no key is set (public agents). Enforces 'valid key required': no key -> no signed URL -> no conversation.
- Force agent creation/update to private (platform_settings.auth.enable_auth=true) so a public, key-less agent can't be created by mistake.
- Encrypt the API key at rest in the SaveGame (AES-256); plaintext kept only in the in-memory cache.
- Redact signed-URL query (auth token) from logs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The API key is no longer stored in project settings / DefaultEngine.ini (it was committed and baked into shipped builds). It is now provided per-install by the client at runtime and persisted in a SaveGame slot.
- Add UPS_AI_ConvAgent_SaveGame_ElevenLabs (slot PS_AI_ConvAgent_ElevenLabs)
- BlueprintLibrary is the single owner of the key: Set/Get/Has/Clear + in-memory cache backed by the SaveGame
- Add async BP node 'Test ElevenLabs API Key' (GET /v1/models)
- Remove API_Key UPROPERTY from UPS_AI_ConvAgent_Settings_ElevenLabs and the key logic from the module
- Route all consumers (WebSocket auth, region probe, editor sync customizations) through GetElevenLabsAPIKey()
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- ElevenLabsComponent::ForceDisableConversation() — removed ActionName param. Now a pure conversation master-switch with no knowledge of actions. OnReadyForAction is fired by the action flow itself: PendingActionName is set when OnAgentActionRequested is broadcasted, then OnReadyForAction fires either after blend-out (if conversation was disabled) or on the next tick (if not).
- Rename bConversationDisabledByAction → bConversationForceDisabled — the old name implied coupling with actions which no longer exists.
- Add bPendingStartListening — when StartListening is called before the WebSocket is connected (race between StartConversation and StartListening from InteractionComponent), defer instead of silently failing. Consumed automatically in HandleConnected / ClientStartConversation. Cleared by StopListening / EndConversation.
- PerceptionComponent — demote the "perceived N actors but ALL filtered out" warning to Verbose; it's a normal case for civilians surrounded by non-hostile actors, not a misconfiguration.
- CLAUDE.md — workflow rule: always describe and request confirmation before code changes.
- Martin.uasset — agent config tweaks.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Bulk rename Category/ClassGroup strings to ASTERION|PS_AI_ConvAgent|... and ASTERION|PS_AI_Behavior|... (284 occurrences, 24 files) so functions/properties/components group cleanly in Blueprint menus
- Move bAutoStartBehavior from AIController to PersonalityComponent so the same AIController class can drive multiple NPC archetypes with per-instance scripted-mode override
- Add PowerPoint user documentation for both plugins in Docs/ with generator scripts
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Allow NPCs to start without a Behavior Tree and be controlled externally
via Blueprint, Level Sequences, or triggers. Adds Scripted enum state that
prevents BT services from overriding state, plus StartBehavior/StopBehavior
BlueprintCallable functions to toggle BT execution at runtime.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New CVar ps.ai.Behavior.Debug: -1=use bDebug property, 0=force off, 1=force on
- IsDebugEnabled() static helper checks CVar then PersonalityComponent.bDebug
- All diagnostic logs gated by IsDebugEnabled (no spam when debug off)
- Logs use Log level (visible) when enabled, silent when disabled
- Applied to: UpdateThreat, FindAndFollowSpline, FindCover, PerceptionComponent,
PersonalityComponent EvaluateReaction
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BodyExpressionComponent: don't activate on OnConversationConnected, wait for
OnSpeakingStarted (prevents body anims while NPC still walking)
- Demote recurring logs to Verbose: UpdateThreat per-tick, FindAndFollowSpline
debug, FindCover distance check, perception target scoring
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
MicrophoneCaptureComponent:
- Local Voice Activity Detection (RMS-based, independent of ElevenLabs)
- Configurable threshold, onset time, silence time
- bIsUserSpeaking flag + OnUserVoiceActivityChanged delegate
- Hysteresis prevents flickering between speech/silence
AIController gaze bridge:
- Resolve MicComponent from player Pawn (not NPC) via reflection
- ConversationPaused BB key blocks movement branches via BT decorator
- NPC stops only when user actually speaks (not just on proximity connect)
- NPC resumes when conversation disconnects
- Spline PauseFollowing/ResumeFollowing on conversation start/end
BT setup required:
- Add Blackboard Condition (ConversationPaused Is Not Set, Aborts=Both)
on spline and patrol branches
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Revert conversation movement pause (needs local VAD, not bNetIsConversing)
- Remove temporary gaze proximity debug log
- Remove bConversationPaused flag (will be reimplemented with VAD)
- TODO: pause NPC movement only when local voice activity is detected
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New BTService_UpdateGaze bridges PS_AI_Behavior to PS_AI_ConvAgent's
GazeComponent via runtime reflection (zero compile dependency).
Priority system:
- Combat/TakingCover: gaze disabled (aim animation handles it)
- Alerted: look at ThreatActor (head + eyes, no body)
- Conversation: skip (ConvAgent manages)
- Proximity: glance at nearest perceived actor within radius
Proximity gaze features:
- Lock on target until release (no jumping)
- Configurable duration, cooldown, radius on AIController
- Front-facing only (dot product filter)
- Skip spectators and hostile actors
GazeComponent fix:
- Sync SmoothedBodyYaw to actual actor rotation when body tracking
is disabled, preventing stale head offset during spline movement
Files:
- New: BTService_UpdateGaze.h/.cpp
- Modified: AIController.h/.cpp (gaze bridge, config, bind, cleanup)
- Modified: GazeComponent.cpp (body yaw sync fix)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- GetTeamAttitudeTowards fallback (interface path) was calling MakeTeamId
without Faction, so enemies of different factions had identical TeamIds
→ always Friendly instead of Hostile
- Now reads Faction from PersonalityProfile when resolving via interface
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Move ThreatDecayRate from global Settings to PersonalityProfile (per-archetype)
- Add state hysteresis in EvaluateReaction to prevent Fleeing/Combat flickering
- FindCover: verify distance on arrival, retry movement if too far
- FindAndFollowSpline: sample multiple spline points when closest is blocked
- BTTask_Attack: call BehaviorStartAttack immediately (draw weapon before LOS/range)
- CoverShootCycle: call BehaviorStartAttack on entry (weapon ready during cover approach)
- FindOwningPawn: walk AttachParentActor chain for ChildActor weapons without Owner
- Initialize PreferCover BB key to false at setup
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Remove Loyalty and Discipline from EPS_AI_Behavior_TraitAxis enum (never used in gameplay)
- Clean up PersonalityProfile and PersonalityComponent default trait initialization
- Add descriptive tooltips to remaining traits (Courage, Aggressivity, Caution)
- Lower Aggressivity combat gate from 0.3 to 0.0 (only Aggressivity=0 prevents combat)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- BTTask_Attack.h: remove reference to non-existent GetBehaviorOptimalAttackRange, document PersonalityProfile ranges
- AIController.h: update TeamId comment to reflect actual MakeTeamId encoding (nibble-based)
- CoverShootCycle.h: clarify that Peek/Cover durations are base values modulated by personality traits
- FindCover.h: clarify ManualPointBonus is additive score
- CombatComponent.h: clarify AttackRange/AttackCooldown are for ExecuteAttack, not BTTask_Attack
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add CombatCoverCycleDuration to PersonalityProfile (configurable base duration)
- PersonalityComponent cycles bPreferCover based on Aggressivity/Caution ratio
- Combat duration = CycleDuration × Aggressivity/(Aggressivity+Caution)
- Cover duration = CycleDuration × Caution/(Aggressivity+Caution)
- Min 2s per phase, ±20% jitter
- Write PreferCover bool to Blackboard for BT decorator observer aborts
- IsCoverNeeded decorator checks both target type AND ShouldPreferCover()
- Remove TakingCover state from EvaluateReaction — cover is now a sub-mode of Combat
- BT uses Blackboard Condition on PreferCover with Observer Aborts=Both
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- PerceptionComponent: omniscient TActorIterator only runs with HasAuthority()
- PersonalityComponent: ApplyReaction and ForceState gate replicated CurrentState writes to server only
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- IsCoverNeeded used Cast<APawn> on ThreatActor which failed for AimTargetActors
→ always returned true (assume dangerous) → enemies took cover against civilians
- Fix: use FindOwningPawn to walk Owner/Instigator chain to the actual Pawn
- Revert inline civilian check in EvaluateReaction (decorator handles it in BT)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- AIController: auto uncrouch when leaving Fleeing/TakingCover state
- FleeFrom: uncrouch at start of flee (civilian was still crouched from HidingSpot)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Fix EQS SetScore filter bug: pass FloatValueMin/Max instead of hardcoded 0/1 (filters were ignored)
- Add LateralSpread to LineOfSight EQS test: multiple traces for wider LOS check
- Add bDrawDebug to CoverQuality and LineOfSight EQS tests
- Ignore ThreatActor collision in EQS traces (AimTargetActor was blocking its own LOS)
- Add navmesh projection in EQSContext_CoverLocation for cover points inside geometry
- Add omniscient awareness: enemies detect top-priority targets (Protectors) within sight radius without perception cone
- Suppress target switching during TakingCover state to prevent cover invalidation
- Fix flanking check: trace at chest height instead of feet
- Add debug visualization for EQS fallback paths (NO REFINE, NO FIRE POS, IN PLACE)
- Clean up diagnostic logs: verbose for per-trace EQS details, log level for summaries
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- CoverShootCycle: SetFocus only during AtCover (not shooting), flanking detection every 0.5s abandons compromised cover
- CoverQuality EQS: ground hit filter, MaxNearbyHitDist (300cm), lateral spread traces, capsule-relative heights
- EQSContext_CoverLocation: prefer CoverPoint actor location over refined vector
- UpdateThreat: never abandon target during Combat/TakingCover states
- MoveTimeout (5s) on all movement states to prevent stuck NPCs
- FindCover: detailed refinement result logging
- Debug: Peeking timer log for investigating timer-stops-counting bug (needs rebuild + test)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- CoverShootCycle: add FiringPositionQuery EQS for peek/shoot positions
(NPC moves between cover position and firing position with LOS)
- Add BTDecorator_IsCoverNeeded: skip cover when target is Civilian
- Add EQSContext_CoverLocation: provides BB CoverLocation to EQS generators
- FindCover: add debug draw toggle and EQS refinement debug spheres
- Definitions: add ECoverShootSubState and CoverPointType::HidingSpot
NOTE: Has compilation errors to fix (signature mismatches in
CoverShootCycle StartPeeking/ReturnToCover, missing forward-declare)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- SetBehaviorCrouch() interface function for cover/hiding crouch control
- CoverShootCycle: continuous LOS check during Peeking (stop if target hides)
- CoverShootCycle: crouch/stand transitions at all cover state changes
- CoverShootCycle: fail when no LOS and no advancing cover (falls to Attack)
- FindCover: crouch on arrival, stand up on abort
- FindCover: optional EQS RefinementQuery to refine exact position around CoverPoints
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- LastKnownTargetPosition check now filters FLT_MAX/InvalidLocation (not just zero)
- EQS result with AlreadyAtGoal triggers fallback advance instead of no-op
- bProjectGoal enabled for EQS move to handle NavMesh projection
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- EQSTest_LineOfSight: new EQS test for firing position queries
- Updated plugin binaries (PS_AI_Behavior + PS_AI_ConvAgent)
- TeamComponent, Settings, Build.cs updates from LOS plan implementation
- PLAN.md updated with LOS combat implementation progress
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Attack/CoverShootCycle: de-escalate to Alerted on failure so decorator
can re-trigger when threat returns (fixes BT stuck on spline branch)
- UpdateThreat: keep BB ThreatActor during brief perception gaps instead
of clearing immediately (use LOS timeout for graceful degradation)
- HasLineOfSight: ignore actors up the attachment chain so trace doesn't
hit the target Pawn's capsule when aiming at its AimTarget child actor
- NoTeam actors (spectators, editor pawns) treated as Neutral instead of
Hostile, plus SpectatorPawn explicit filter in perception
- BB debug key ThreatPawnName shows owning Pawn name (resolved via
perception's LastThreatOwningPawn) instead of cryptic AimTarget name
- FindOwningPawn promoted to public static on PerceptionComponent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Split ResolveToPawn into FindOwningPawn + GetThreatTarget so non-Pawn
actors (PS_AimTargetActor) are properly resolved for team/attitude checks
while still being used as BB target. Add attack range hysteresis (10%
buffer), target persistence (80% threshold), melee no-cooldown chase,
ranged midpoint approach. New files: CoverShootCycle task, CheckCombatType
decorator, MinAttackRange/MaxAttackRange in PersonalityProfile.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add TeamComponent for player pawn team identity (Role-based: Civilian/Enemy/Protector)
- Add IsTargetActorValid to interface for dead target filtering
- Fix GetTeamAttitudeTowards to check IGenericTeamAgentInterface + TeamComponent
- Guarantee BehaviorStopAttack via OnTaskFinished (all exit paths)
- Prevent Combat state without ThreatActor (stay Alerted until perception catches up)
- Resume spline at closest point from current position after combat
- Sync CurrentSpline and SplineProgress to Blackboard in FollowSpline tick
- Auto-detect Patrol state when NPC has a spline (fixes Idle speed=0 blocking movement)
- Add per-component debug toggles (Personality + SplineFollower independent)
- Use AddMovementInput instead of RequestDirectMove for reliable post-combat movement
- Add bTickInEditor for Personality debug in Simulate mode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Always recalculate TeamId at OnPossess (BP CDOs may reset to 0)
- Remove duplicate _Implementation definitions (UHT auto-generates them)
- Fixes crash when interface functions not overridden in BP
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add BehaviorStartAttack/BehaviorStopAttack to IPS_AI_Behavior_Interface
- Attack task now calls interface instead of CombatComponent directly
- Task stays InProgress permanently, Decorator Observer Aborts handles exit
- Remove CombatComponent dependency from Attack task
- Pawn handles actual aiming/shooting via its own systems
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
BT Attack task:
- Stay InProgress permanently instead of returning Succeeded after each attack
- Stay InProgress on MoveToActor failure instead of Failed (retry next tick)
- Add verbose logging for attack state (target, range, distance)
BT EvaluateReaction service:
- Auto-detect hostility changes via IPS_AI_Behavior interface
- Dynamically update TeamId when IsBehaviorHostile() changes (infiltrator reveal)
AIController:
- Remove GetBehaviorTeamId from interface (TeamId is now 100% automatic)
- TeamId derived from NPCType + hostile state, no user implementation needed
- Add BB State change logging for debug
- Use SetValueAsEnum consistently for BehaviorState key
Interface:
- Remove GetBehaviorTeamId — TeamId is computed by the plugin automatically
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Perception:
- Configure senses in BeginPlay + RequestStimuliListenerUpdate (not constructor)
- Filter threats by team attitude: only Hostile actors generate threat
- Skip Friendly and Neutral actors in CalculateThreatLevel and GetHighestThreatActor
- All Hostile actors are valid threats regardless of TargetPriority list
Blackboard:
- Use SetValueAsEnum/GetValueAsEnum for BehaviorState key (was Int)
Patrol:
- Auto NavMesh patrol when no manual waypoints defined
- Project HomeLocation onto NavMesh before searching random points
- Fallback to NPC current position if HomeLocation not on NavMesh
Spline following:
- Choose direction based on NPC forward vector vs spline tangent (not longest distance)
- Skip re-search if already following a spline (prevent BT re-boucle)
- Reverse at end: wait for NPC to catch up before inverting direction
- Use NPC actual CharacterMovement speed for target point advancement
- Face toward target point (not spline tangent) to prevent crab-walking in turns
- Junction switching: direction continuity check, 70% switch chance, world gap tolerance
- Debug draw: green sphere (target), yellow line (gap), cyan arrow (tangent), text overlay
- Add GetWorldDirectionAtDistance to SplinePath
Editor:
- Add Placeable to SplinePath and CoverPoint UCLASS
- Add PersonalityProfileFactory for Data Asset creation
- Add EnsureBlackboardAsset declaration
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add bExpressiveMode toggle and editable ExpressiveModePromptFragment
with audio tag instructions ([laughs], [whispers], [sighs], [slow], [excited])
- BuildAgentPayload: append prompt fragment, set expressive_mode API field
on agent config, auto-override TTS model to eleven_v3_conversational
- OnFetchAgent: strip expressive fragment from prompt (exact + marker fallback),
read expressive_mode bool from API, auto-detect V3 model
- TTS model combo: inject asset's current model if absent from /v1/models list
(covers agent-only models like eleven_v3_conversational)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix A→B→A audio cutoff: when switching back to a pending-leave agent,
cancel the deferred leave instead of force-completing it (was calling
StopAgentAudio on the agent we're returning to)
- Fix deferred leave firing during TTS gaps: use IsAgentSpeakingOrPending()
instead of IsAgentSpeaking() — checks bAgentGenerating and
bAgentResponseReceived to avoid premature leave during inter-batch silence
- Convert silence detection from tick-based to time-based: SilentTickCount
→ SilentTime (float seconds), GeneratingTickCount → GeneratingTime.
Consistent behavior regardless of frame rate (was 5s@120fps vs 20s@30fps)
- Fix lazy binding: add OnAgentConnected/OnAgentDisconnected in LipSync
and FacialExpression TickComponent lazy-bind path (bActive stayed false
forever in packaged builds when component init order differed)
- Fix reconnection: reset bWaitingForAgentResponse and GeneratingTime
before entering reconnect mode to avoid stale state on new session
- Fix event_ID audio filtering: reset LastInterruptEventId in
HandleAgentResponse and SendUserTurnStart so first audio chunks of a
new turn are not silently discarded by stale interrupt filter
- Preserve retained gaze when switching back to same agent (don't
CleanupRetainedGaze if PrevRetained == NewAgent)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- ForceDisableConversation/ForceEnableConversation: disable agent conversation
with blend-out monitoring and OnReadyForAction event (with ActionName param)
- ActionSet data asset: configurable action list per agent with editor
customization (Update All Agents button, custom detail panel)
- Passive gaze by proximity: nearby non-selected agents track the player
with configurable head+eyes and body checkboxes (bAutoPassiveGaze,
bPassiveGazeHeadEyes, bPassiveGazeBody)
- Retained gaze on conversation switch now uses the same passive gaze config
- OnPassiveGazeStarted/OnPassiveGazeStopped events on ElevenLabsComponent
- Fix debug HUD key collisions: per-actor key ranges prevent multi-agent
HUD flickering, add actor name to all HUD titles
- Fix retained gaze bug: re-activate gaze after ExecuteLeave before
ApplyConversationGaze kills it
- Safety timeout (5s) for blend-out monitoring
- Guard on AttachGazeTarget when conversation is disabled
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Conversation lock no longer prevents switching to a different agent.
When in an active conversation, the player can look at another nearby
agent for ConversationSwitchDelay seconds (default 1s) to switch.
Looking at empty space keeps the current agent selected (no deselect).
Works in multiplayer — each player has independent switch tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add TurnEagerness (Eager/Normal/Patient) and bSpeculativeTurn to agent config
data asset, sent as conversation_config_override at WebSocket connection time
- Add adaptive pre-buffer system: measures inter-chunk TTS timing and decreases
pre-buffer when chunks arrive fast enough (decrease-only, resets each conversation)
- New UPROPERTY: bAdaptivePreBuffer toggle, AudioPreBufferMs as starting/worst-case value
- Rework latency HUD: TTS+Net, PreBuf actual/target with trend indicator, Gen>Ear,
WS Ping, server region display
- Fetch ElevenLabs server region from REST API x-region header
- Add editor Detail Customization: TurnEagerness dropdown + SpeculativeTurn checkbox
in AgentConfig with LLM picker and Language picker
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
user_transcript arrives AFTER agent_response_started in Server VAD mode
(the server detects end of speech via VAD, starts generating immediately,
and STT completes later). This caused Transcript>Gen to show stale values
(19s) and Total < Gen>Audio (impossible).
Now all metrics are anchored to GenerationStartTime (agent_response_started),
which is the closest client-side proxy for "user stopped speaking":
- Gen>Audio: generation start → first audio chunk (LLM + TTS)
- Pre-buffer: wait before playback
- Gen>Ear: generation start → playback starts (user-perceived)
Removed STTToGenMs, TotalMs, EndToEarMs, UserSpeechMs (all depended on
unreliable timestamps). Simpler, always correct, 3 clear metrics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous approach used TurnEndTime (from StopListening) which was never
set in Server VAD mode. Now all latency measurements are anchored to
TurnStartTime, captured when the first user_transcript arrives from the
ElevenLabs server — the earliest client-side confirmation of user speech.
Timeline: [user speaks] → STT → user_transcript(=T0) → agent_response_started → audio → playback
Metrics shown:
- Transcript>Gen: T0 → generation start (LLM think time)
- Gen>Audio: generation start → first audio chunk (LLM + TTS)
- Total: T0 → first audio chunk (full pipeline)
- Pre-buffer: wait before playback
- End-to-Ear: T0 → playback starts (user-perceived)
Removed UserSpeechMs (unmeasurable without client-side VAD).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
TurnStartTime was only set in StartListening(), which is called once.
In Server VAD + interruption mode the mic stays open, so TurnStartTime
was never updated between turns. Now reset TurnStartTime when the agent
stops speaking (normal end + interruption), marking the start of the
next potential user turn.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Move latency reset from StopListening to HandleAgentResponseStarted.
In Server VAD + interruption mode, StopListening is never called so
TurnEndTime stayed at 0 and all dependent metrics showed ---. Now
HandleAgentResponseStarted detects whether StopListening provided a
fresh TurnEndTime; if not (Server VAD), it uses Now as approximation.
Also fix DisplayTime from 0 to 1s to prevent HUD flicker.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add bDebugLatency property + CVar (ps.ai.ConvAgent.Debug.Latency)
independent from bDebug to save HUD space
- Reset latencies to zero each turn (StopListening) instead of persisting
- Add UserSpeechMs and PreBufferMs to the latency struct
- Move latency captures outside bDebug guard (always measured)
- Replace single-line latency in DrawDebugHUD with dedicated DrawLatencyHUD()
showing 6 metrics on separate lines with color coding
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Track 4 latencies per conversation turn (computed only when bDebug is active):
- STT→Gen: user stops talking → server starts generating
- Gen→Audio: server generating → first audio chunk received
- Total: user stops talking → first audio chunk (end-to-end)
- End-to-Ear: user stops talking → audio playback starts (includes pre-buffer)
New timestamps: GenerationStartTime (HandleAgentResponseStarted),
PlaybackStartTime (3 OnAudioPlaybackStarted sites). Values persist on
HUD between turns, reset when new turn starts.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1. StartConversationWithSelectedAgent: remove early return when WebSocket
is already connected (persistent mode). Always call ServerJoinConversation
so the pawn is added to NetConnectedPawns and bNetIsConversing is set.
2. ServerSendMicAudioFromPlayer: bypass speaker arbitration in standalone
mode (<=1 connected pawn). Send audio directly to avoid silent drops
caused by pawn not being in NetConnectedPawns array. Add warning logs
for multi-player drops to aid debugging.
3. OnMicrophoneDataCaptured: restore direct WebSocketProxy->SendAudioChunk
on the server path. This callback runs on the WASAPI audio thread —
accessing game-thread state (NetConnectedPawns, LastSpeakTime) was
causing undefined behavior. Internal mic is always the local player,
no speaker arbitration needed.
4. StopListening flush: send directly to WebSocket (active speaker already
established, no arbitration needed for the tail of the current turn).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
GetCurrentBlendshapes() was copying CurrentBlendshapes on the anim worker
thread while the game thread mutated it (TSet::UnhashElements crash).
Use a snapshot pattern: game thread copies to ThreadSafeBlendshapes under
FCriticalSection at end of TickComponent, anim node reads the snapshot.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace exclusive single-player agent lock with shared multi-player model:
- NetConversatingPawn/Player → NetConnectedPawns array + NetActiveSpeakerPawn
- Server-side speaker arbitration with hysteresis (0.3s) prevents gaze ping-pong
- Speaker idle timeout (3.0s) clears active speaker after silence
- Agent gaze follows the active speaker via replicated OnRep_ActiveSpeaker
- New ServerJoinConversation/ServerLeaveConversation RPCs (idempotent join/leave)
- Backward-compatible: old ServerRequest/Release delegate to new Join/Leave
- InteractionComponent no longer skips occupied agents
- DrawDebugHUD shows connected player count and active speaker
- All mic audio paths (FeedExternalAudio, OnMicCapture, StopListening flush) route
through speaker arbitration
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- InteractionComponent: GetPawnViewPoint() now prefers UCameraComponent
over PlayerController::GetPlayerViewPoint() — fixes agent selection
in VR where the pawn root stays at spawn while the HMD moves freely
- GazeComponent: ResolveTargetPosition() uses camera first for locally
controlled pawns (VR HMD / FPS eye position), falls back to bone
chain for NPCs and remote players in multiplayer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix DisplayTime=0.0f causing flicker on all debug HUDs (now 1.0f)
- Add per-component CVars (ps.ai.ConvAgent.Debug.*) for console debug toggle
- Add MicrophoneCapture debug HUD with VU meter (RMS/peak/dB bar)
- InteractionComponent reuses existing MicrophoneCaptureComponent on pawn
instead of always creating a new one
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Sync body animation with actual audio playback via new OnAudioPlaybackStarted
delegate instead of OnAgentStartedSpeaking (accounts for pre-buffer delay)
- Fix stale pre-buffer broadcasts by cancelling bPreBuffering on silence detection
and guarding pre-buffer timeout with bAgentSpeaking check
- Smooth body crossfade using FInterpTo instead of linear interpolation
- Add conversation lock in EvaluateBestAgent: keep agent selected during active
conversation regardless of view cone (distance-only check prevents deselect
flicker on fast camera turns)
- Broadcast OnAgentDisconnected in persistent session EndConversation so all
expression components (body, facial, lip sync, gaze) properly deactivate
when the player leaves the interaction zone
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace FInterpConstantTo with FInterpTo (exponential ease-out) in all 4 anim components
- Apply SmoothStep to crossfade alphas for smooth animation transitions
- Add SpeechBlendAlpha to LipSync: fades out mouth curves when not speaking,
letting FacialExpression emotion curves (including mouth) show through
- Remove LipSync AnimNode grace period zeroing to avoid overwriting emotion curves
- Revert BodyExpression to override mode (additive broken with full-pose anims)
- Default ExcludeBones = neck_01 to prevent Gaze/Posture conflicts
- Fix mid-crossfade animation pop in BodyExpression SwitchToNewAnim
- Add Neutral emotion fallback in Body and Facial expression components
- Add SendTextToSelectedAgent convenience method on InteractionComponent
- Add debug HUD display for BodyExpression component
- Update CoreRedirects for Posture → Gaze rename
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add BodyExpression system: emotion-driven body animations with per-emotion
anim lists (Idle/Normal/Medium/Extreme), random selection, auto-cycle on
loop complete, crossfade transitions, upper-body-only or full-body mode
- Replace bExternalMicManagement with auto-detecting ShouldUseExternalMic()
- Add bAutoTargetEyes to GazeComponent: auto-aim at target's eye bones
(MetaHuman), with fallback chain (eyes > head > FallbackEyeHeight)
- Hide bActive from Details panel on all 4 anim components (read-only,
code-managed): FacialExpression, Gaze, LipSync, BodyExpression
- Remove misleading mh_arkit_mapping_pose warning from LipSync
- Add bUpperBodyOnly toggle to BodyExpression AnimNode
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three issues prevented posture re-activation on re-entry:
- StartConversation skipped bNetIsConversing/NetConversatingPawn in standalone
(guarded by NM_Standalone check) so ApplyConversationPosture always deactivated
- SetSelectedAgent required !IsConnected() to auto-start conversation, but in
persistent mode the WS stays connected → StartConversation never called on re-entry
- AttachPostureTarget never set Posture->bActive = true, relying solely on
ApplyConversationPosture which could be skipped due to the above condition
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When true (default), the first StartConversation() opens the WebSocket
and EndConversation() only stops the mic/posture — the WebSocket stays
open until EndPlay. The agent remembers the full conversation context.
When false, each Start/EndConversation opens/closes the WebSocket (previous behavior).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Make bAgentGenerating, bWaitingForAgentResponse, bWaitingForResponse,
bFirstAudioResponseLogged, bAgentResponseStartedFired (std::atomic<bool>)
and LastInterruptEventId (std::atomic<int32>) for thread-safety
- Add WebSocket auto-reconnection with exponential backoff (1s→30s cap,
max 5 attempts), distinguishing intentional vs unexpected disconnects
- Add FillCurrentEyeCurves() zero-allocation method using FindOrAdd()
to eliminate per-frame TMap heap allocation in anim thread
- Replace AudioQueue RemoveAt(0,N) with read-offset pattern — O(1) per
underflow callback, periodic compaction when offset > half buffer
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previous fix only smoothed cascade inputs (head/eyes) via SmoothedBodyYaw
but the body mesh still followed the raw replicated actor rotation which
jumped at ~30Hz network rate. Now the client calls SetActorRotation with
the smoothed yaw so the mesh visually interpolates. Replication overwrites
on next network update; SmoothedBodyYaw absorbs the correction smoothly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Server-only AddActorWorldRotation (HasAuthority guard) prevents
client/server tug-of-war. Client interpolates toward replicated
rotation via SmoothedBodyYaw (angle-aware FInterpTo at 3x body
speed) to eliminate step artifacts from ~30Hz network updates.
Cascade (DeltaYaw, head, eyes) uses SmoothedBodyYaw on all machines.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The pelvis bone rotation approach didn't work in practice. Reverts to
the previous AddActorWorldRotation() body tracking (replicated actor
rotation). Thread-safety fix and deprecated IsValid() removal are kept.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- FacialExpressionComponent: add FCriticalSection around emotion curves
to prevent race between TickComponent (game thread) and anim worker
thread — root cause of EXCEPTION_ACCESS_VIOLATION at Evaluate_AnyThread
- Remove deprecated FBlendedCurve::IsValid() guards (UE 5.5: always true)
- Body tracking: replace AddActorWorldRotation() with animation-only
pelvis bone rotation via AnimNode — eliminates replication tug-of-war
that caused client-side saccades when server overwrote local rotation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create OpusEncoder on ALL machines (was Authority-only) — clients now
encode mic audio, server decodes it; server still encodes agent audio
- FeedExternalAudio / OnMicrophoneDataCaptured: Opus-encode accumulated
PCM buffer before sending via ServerRelayMicAudio RPC on client path
(~200 bytes/100ms instead of 3200 bytes = ~16 Kbits/s vs 256 Kbits/s)
- ServerRelayMicAudio_Implementation: auto-detect Opus (size < raw chunk)
and decode back to PCM before forwarding to WebSocket
- Add public DecompressMicAudio() helper for clean API access from
InteractionComponent relay without exposing private Opus members
- Graceful fallback: if Opus unavailable, raw PCM is sent/received as before
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Attach AudioPlaybackComponent to owner's root component for proper
3D world positioning (was unattached = stuck at origin)
- Enable default inline spatialization with 15m falloff distance when
no external SoundAttenuation asset is set
- External SoundAttenuation asset still overrides the default if set
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-end conversation on deselection: when bAutoStartConversation is
true and the player walks out of range, EndConversation() is called
so the NPC becomes available for other players
- Server-side posture: add ApplyConversationPosture() helper called
from ServerRequestConversation, ServerReleaseConversation,
EndConversation (Authority path), and HandleDisconnected — fixes
NPC not tracking the client on the listen server (OnRep never fires
on Authority)
- Guard DetachPostureTarget: only clear TargetActor if it matches our
pawn, preventing the server IC from overwriting posture set by the
conversation system for a remote client
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Silence gate: skip sending silent mic audio over network RPCs on clients
(~256 Kbits/s saved when not speaking, fixes chaotic teleporting)
- Lazy init: defer InteractionComponent mic creation from BeginPlay to
TickComponent with IsLocallyControlled guard (fixes "No owning connection"
from server-side replicas of remote pawns)
- Body tracking: use bNetIsConversing as fallback for IsConnected() on
clients where WebSocket doesn't exist
- EvaluateBestAgent: null-check NetConversatingPawn before comparison
- MicCaptureComponent: use TWeakObjectPtr in AsyncTask lambda to prevent
FMRSWRecursiveAccessDetector race on component destruction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
In a listen server, the server-side copy of a remote client's pawn also
has an InteractionComponent that ticks. This caused a race condition:
the server-side tick would start conversations using GetFirstPlayerController()
(= server's PC), setting NetConversatingPawn to the server's pawn instead
of the client's. The client's relay RPC arrived too late and was rejected
because bNetIsConversing was already true.
Fix: disable tick and skip mic creation in BeginPlay for non-locally-controlled
pawns. The client handles all interaction locally via relay RPCs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 clients cannot call Server RPCs on actors they don't own. NPC actors
are server-owned, causing "No owning connection" errors when remote clients
try to start conversations, send mic audio, or interrupt agents.
Solution: relay all client→NPC RPCs through the InteractionComponent on
the player's pawn (which IS owned by the client). The relay forwards
commands to the NPC's ElevenLabsComponent on the server side.
Changes:
- InteractionComponent: add Server relay RPCs (Start/End conversation,
mic audio, text message, interrupt) and Client relay RPCs
(ConversationStarted/Failed) with GetLifetimeReplicatedProps
- ElevenLabsComponent: implement FindLocalRelayComponent(), route all
client-side calls through relay (StartConversation, EndConversation,
SendTextMessage, InterruptAgent, FeedExternalAudio, mic capture)
- Fix HandleConnected/ServerRequestConversation to route Client RPCs
through the player pawn's relay instead of the NPC (no owning connection)
- Fix StartListening/FeedExternalAudio/StopListening to accept
bNetIsConversing on clients (WebSocket only exists on server)
- Fix EvaluateBestAgent to use NetConversatingPawn instead of
NetConversatingPlayer (NULL on remote clients due to bOnlyRelevantToOwner)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent starts with bActive=false and waits for
OnConversationConnected, which only fires on the server (WebSocket).
Remote clients never got bActive=true, so CurrentActiveAlpha stayed
at 0 and all head/eye rotation was zeroed out.
Now OnRep_ConversationState also sets Posture->bActive alongside
TargetActor, matching what was already done for FacialExpression
and LipSync components.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs NetConversatingPawn, PostureComponent availability, and TargetActor
assignment to diagnose why head tracking doesn't work on remote clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raw PCM is split into 32KB sub-chunks for network transmission.
On the client, these sub-chunks arrive nearly simultaneously,
triggering the "second chunk arrived" fast-path which cancelled
the pre-buffer after ~10ms instead of the intended 2000ms.
Now the fast-path only applies on Authority (server) where
chunks represent genuine separate TTS batches. Clients always
wait the full pre-buffer duration via TickComponent timer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PlayerControllers have bOnlyRelevantToOwner=true, so NetConversatingPlayer
was always nullptr on remote clients. Added NetConversatingPawn (APawn*)
which IS replicated to all clients. OnRep_ConversationState now uses
NetConversatingPawn as the posture TargetActor, enabling head/eye
tracking on all clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 limits replicated TArrays to 65535 elements. ElevenLabs sends
audio chunks up to ~72K bytes, exceeding this limit when sent as
raw PCM. Split into 32000-byte sub-chunks (1s of 16kHz 16-bit mono)
before calling MulticastReceiveAgentAudio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 5.5's FVoiceModule returns NULL encoder/decoder, breaking
Opus-based audio replication. This adds a transparent fallback:
- Server sends raw PCM when OpusEncoder is null (~32KB/s, fine for LAN)
- Client accepts raw PCM when OpusDecoder is null
- Changed MulticastReceiveAgentAudio from Unreliable to Reliable
to handle larger uncompressed payloads without packet loss
- Added OnlineSubsystemNull config for future Opus compatibility
- Removed premature bAgentSpeaking=true from MulticastAgentStartedSpeaking
to fix race condition with audio initialization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Log at InitOpusCodec: FVoiceModule availability, encoder/decoder creation, net role
- Log at HandleAudioReceived: warn once if encoder is null or role is not Authority
- These fire unconditionally (not behind bDebug) to catch the root cause
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs compressed audio size in HandleAudioReceived to diagnose
why MulticastReceiveAgentAudio (Unreliable) never reaches clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- OnRep_ConversationState now sets PostureComponent TargetActor from
replicated NetConversatingPlayer so remote clients see head/eyes tracking
- Activate FacialExpressionComponent and LipSyncComponent on remote clients
(OnAgentConnected never fires on clients since WebSocket is server-only)
- Fix audio race condition: MulticastAgentStartedSpeaking no longer sets
bAgentSpeaking prematurely, letting EnqueueAgentAudio handle the full
first-chunk initialization (pre-buffer, Play(), state reset)
- Add diagnostic logging to MulticastReceiveAgentAudio for silent failures
(OpusDecoder invalid, LOD culling, decode failure)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces UPS_AI_ConvAgent_AgentConfig_ElevenLabs data asset to encapsulate
full agent configuration (voice, LLM, prompt, language, emotions) with a
custom Detail Customization providing:
- Voice/TTS Model/LLM/Language pickers with Fetch buttons (ElevenLabs API)
- LLM latency hints in dropdown (~250ms, ~700ms, etc.)
- Create/Update/Fetch Agent buttons for REST API CRUD
- Auto-fetch on editor open, auto-select first voice for new assets
- Prompt fragment management (language, multilingual, emotion tool)
- Smart defaults: gemini-2.5-flash LLM, eleven_turbo_v2_5 TTS, English
- Speed range expanded to 0.7-1.95 (was 0.7-1.2)
- bAutoStartConversation + StartConversationWithSelectedAgent() on InteractionComponent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Saved/ is not staged in packaged builds, so Content/Certificates/ is the
only reliable location. Simplified code by removing Android-specific
writable fallback (Content/ works on all platforms with NonUFS staging).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Posture, FacialExpression, LipSync: bActive + ActivationBlendDuration for
smooth alpha blend in/out (linear interp via FInterpConstantTo).
- Auto-activation: components bind to OnAgentConnected/OnAgentDisconnected,
starting inactive and blending in when conversation begins.
- Without an agent component, bActive defaults to true (backward compatible).
- Add BlueprintFunctionLibrary with SetPostProcessAnimBlueprint helper
(wraps UE5.5 SetOverridePostProcessAnimBP for per-instance BP setup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Whitelist Android in .uplugin Runtime module and handle
read-only APK paths for SSL certificate copy on Android
(ProjectSavedDir fallback). No change to Win64 behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only init, deinit, errors and critical warnings remain unconditional.
All turn-by-turn logs (mic open/close, agent generating/speaking/stopped,
emotion changes, audio chunks, text/viseme processing, latency, bone
resolution, selection, registration) are now gated behind component
bDebug or Settings->bVerboseLogging.
10 files, ~125 lines removed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Packaged build fixes:
- Use EvaluateCurveData() for emotion and lip sync curves (works with
compressed/cooked data instead of raw FloatCurves)
- Add FMemMark wrapper for game-thread curve evaluation (FBlendedCurve
uses TMemStackAllocator)
- Lazy binding in AnimNodes and LipSyncComponent for packaged build
component initialization order
- SetIsReplicatedByDefault(true) instead of SetIsReplicated(true)
- Load AudioCapture module explicitly in plugin StartupModule
- Bundle cacert.pem for SSL in packaged builds
- Add DirectoriesToAlwaysStageAsNonUFS for certificates
- Add EmotionPoseMap and LipSyncPoseMap .cpp implementations
Body tracking:
- Body tracking now activates on conversation start (HandleAgentResponseStarted)
instead of on selection, creating a natural notice→engage two-step:
eyes+head track on selection, body turns when agent starts responding
- SendTextMessage also enables body tracking for text input
Cleanup:
- Remove all temporary [DIAG] and [BODY] debug logs
- Gate PostureComponent periodic debug log behind bDebug flag
- Remove obsolete editor-time curve caches (CachedCurveData, RebuildCurveCache,
FPS_AI_ConvAgent_CachedEmotionCurves, FPS_AI_ConvAgent_CachedPoseCurves)
from EmotionPoseMap and LipSyncPoseMap — no longer needed since
EvaluateCurveData() reads compressed curves directly at runtime
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replicate conversation state (bNetIsConversing, NetConversatingPlayer) for exclusive NPC locking
- Opus encode TTS audio on server, multicast to all clients for shared playback
- Replicate emotion state (OnRep) so clients compute facial expressions locally
- Multicast speaking/interrupted/text events so lip sync and posture run locally
- Route mic audio via Server RPC (client→server→ElevenLabs WebSocket)
- LOD: cull audio beyond 30m, skip lip sync beyond 15m for non-speaker clients
- Auto-detect player disconnection and release NPC on authority
- InteractionComponent: skip occupied NPCs, auto-start conversation on selection
- No changes to LipSync, Posture, FacialExpression, MicCapture or AnimNodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Body tracking is now activated by ElevenLabsComponent directly
in StartListening() and SendTextMessage(), instead of being
managed by InteractionComponent. This ensures the agent turns
its body toward the player on any form of conversation input.
InteractionComponent still disables body tracking on deselection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent gains bEnableBodyTracking flag. When false, only
head and eyes track the target — body stays frozen.
InteractionComponent now:
- On posture attach: sets TargetActor but disables body tracking
(agent notices player with eyes+head only)
- On StartListening: enables body tracking (agent fully engages)
- On StopListening: disables body tracking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Automatically calls StartListening/StopListening on the agent's
ElevenLabsComponent on selection/deselection. Enabled by default.
Disable for manual control (e.g. push-to-talk).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent: body no longer returns to original yaw when
TargetActor is cleared — only head and eyes return to neutral.
InteractionComponent: add AgentEyeLevelOffset (default 150cm) so
the view cone check targets chest height instead of feet, preventing
selection loss at close range.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
InteractionComponent now automatically sets/clears the agent's
PostureComponent TargetActor on selection/deselection, with
configurable attach/detach delays and a master toggle for manual control.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aligns all class/struct/enum/delegate prefixes with the module name
PS_AI_ConvAgent. Removes redundant Conv_ from ElevenLabsComponent
(PS_AI_Agent_Conv_ElevenLabsComponent → PS_AI_ConvAgent_ElevenLabsComponent).
UI strings now use "PS AI ConvAgent". CoreRedirects updated for both
ElevenLabs* and PS_AI_Agent_* legacy references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Separates generic systems (lip sync, posture, facial expression, mic)
from ElevenLabs-specific ones (conv agent, websocket, settings).
Generic classes now use PS_AI_Agent_ prefix; ElevenLabs-specific keep
_ElevenLabs suffix. CoreRedirects in DefaultEngine.ini ensure existing
.uasset references load correctly. Binaries/Intermediate deleted for
full rebuild.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add BodyDriftCompensation parameter (0→1) to counter-rotate head when
body animation moves the torso (bow, lean). Drift correction applied
AFTER swing-twist on the first chain bone only, preventing the
correction from being stripped by tilt decomposition or compounded
through the chain.
- Add bDrawDebugGaze toggle: draws per-eye debug lines from Face mesh
FACIAL_L/R_Eye bones (cyan = desired direction, green = actual bone
Z-axis gaze) to visually verify eye contact accuracy.
- Cache Face mesh separately from Body mesh for correct eye bone transforms.
- Use eye bone midpoint (Face mesh) as EyeOrigin for pitch calculation
instead of head bone, fixing vertical offset.
- Calibrate ARKit eye ranges: horizontal 30→40, vertical 20→35 to match
MetaHuman actual eye deflection per curve unit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix eye compensation to use smooth Lerp blend instead of binary switch.
Uses Curve.Get() to read animation's existing CTRL values, then
Lerp(animCTRL, postureCTRL, Comp) for proportional blending.
- Split AnimationCompensation into HeadAnimationCompensation and
EyeAnimationCompensation for independent control per layer.
- Fix CTRL curve naming: use correct MetaHuman format
(CTRL_expressions_eyeLook{Dir}{L/R}) with proper In/Out→Left/Right mapping.
- Add eye diagnostic modes (disabled) for pipeline debugging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add NeckBoneChain config (FElevenLabsNeckBoneEntry USTRUCT) to distribute
head rotation across multiple bones (e.g. neck_01=0.25, neck_02=0.35,
head=0.40) for more natural neck arc movement.
- Swing-twist now uses bone's actual tilt axis (BoneRot.RotateVector) instead
of hardcoded FVector::RightVector — accounts for MetaHuman ~11.5° reference
pose rotation on head bone.
- Clean up diagnostic logging: reduced to Log level, removed verbose per-frame
quaternion dumps.
- Backward compatible: empty NeckBoneChain falls back to single HeadBoneName.
KNOWN ISSUE: diagonal tilt is more visible with neck chain — needs investigation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix two-step body animation: head overflow now checked against
TargetBodyWorldYaw (body TARGET) instead of current body position,
preventing head overcompensation during body interpolation.
- Fix ARKit eye curve normalization: curves now normalized by fixed
physical range (30°/20°) instead of configurable MaxEye thresholds.
MaxEye controls cascade threshold only, not visual deflection.
- Fix diagonal head tilt: full FQuat pipeline (no FRotator round-trip)
+ swing-twist decomposition using bone's actual tilt axis (accounts
for MetaHuman ~11.5° reference pose rotation on head bone).
- Clean up diagnostic logging: reduced to Log level, removed verbose
per-frame quaternion dumps from investigation phase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite posture system as a relative cascade (Eyes → Head → Body) with
persistent sticky targets. Each layer stays put until the layer below
overflows its max angle, then realigns fully toward the target.
Key changes:
- Thread safety: FCriticalSection protects AnimNode shared data
- Persistent TargetHeadYaw/Pitch: overflow checked against target (not
interpolating current) so head completes full realignment
- Persistent TargetBodyWorldYaw: body only moves when head+eyes combined
range is exceeded (sticky, same pattern as head)
- Quaternion head rotation: compose independent NodQuat × TurnQuat to
avoid diagonal coupling that FRotator causes
- Eye curves: negate CurrentEyeYaw for correct ARKit convention
- AnimNode: enhanced logging, axis diagnostic define (disabled)
- Remove old percentage/degree activation thresholds (max angles serve
as natural thresholds in the relative cascade)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New ElevenLabsPostureComponent with 3-layer rotation distribution:
- Body (60%): rotates entire actor (yaw only) via SetActorRotation
- Head (20%): direct bone transform in AnimNode
- Eyes (10%): 8 ARKit eye look curves (eyeLookUp/Down/In/Out L/R)
Features:
- Configurable rotation percentages and angle limits per layer
- Smooth FInterpTo interpolation with per-layer speeds
- TargetActor + TargetOffset for any actor type (no skeleton required)
- Smooth return to neutral when TargetActor is cleared
- Blue AnimGraph node in ElevenLabs category
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create dedicated UElevenLabsEmotionPoseMap data asset for emotions
- FacialExpressionComponent now uses EmotionPoseMap (not LipSyncPoseMap)
- Remove emotion data (FElevenLabsEmotionPoseSet, EmotionPoses TMap) from LipSyncPoseMap
- Add ElevenLabsEmotionPoseMapFactory for Content Browser asset creation
- Standardize log format to [T+Xs] [Turn N] in ConversationalAgentComponent
- Remove unused #include "Engine/World.h"
- Simplify collision log to single line
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New AnimNode_ElevenLabsFacialExpression: independent AnimBP node for emotion expressions
- New AnimGraphNode (amber color) in ElevenLabs category for AnimBP editor
- Emotion AnimSequences now play in real-time (looping) instead of static pose at t=0
- Smooth crossfade between emotions with configurable duration
- LipSync AnimNode skips near-zero curves so emotion base layer shows through during silence
- Removed emotion merge from LipSyncComponent (now handled by AnimBP node ordering)
- Removed verbose per-tick VISEME debug log
- Two-layer AnimBP chain: FacialExpression → LipSync → mh_arkit_mapping_pose
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Decouple viseme timing from 32ms audio chunks by introducing an independent
FVisemeTimelineEntry timeline evaluated at render framerate. Playback-time
envelope tracking from consumed queue frames replaces arrival-time-only
updates, with fast 40ms decay when queue is empty.
- Viseme subsampling caps at ~10/sec (100ms min) to prevent saccades
- Full-duration quintic smootherstep crossfade (C2 continuous, no hold phase)
- Dead zone lowered to 0.15 for cleaner silence transitions
- TotalActiveFramesSeen cumulative counter for accurate timeline scaling
- Absolute cursor preservation on timeline rebuild
- Moderate Lerp smoothing (attack 0.55, release 0.40)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix event_id filtering bug: reset LastInterruptEventId when new generation
starts, preventing all audio from being silently dropped after an interruption
- Match C++ sample API config: remove optimize_streaming_latency and
custom_llm_extra_body overrides, send empty conversation_config_override
in Server VAD mode (only send turn_timeout in Client mode)
- Instant audio stop on interruption: call ResetAudio() before Stop() to
flush USoundWaveProcedural's internal ring buffer
- Lip sync reset on interruption/stop: bind OnAgentInterrupted (snap to
neutral) and OnAgentStoppedSpeaking (clear queues) events
- Revert jitter buffer (replaced by pre-buffer approach, default 2000ms)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove silence padding accumulation bug: QueueAudio'd silence was
accumulating in USoundWaveProcedural's internal buffer during TTS gaps,
delaying real audio by ~800ms. USoundWaveProcedural with
INDEFINITELY_LOOPING_DURATION generates silence internally instead.
- Fix pre-buffer bypass: guard OnProceduralUnderflow with bPreBuffering
check — the audio component never stops (INDEFINITELY_LOOPING_DURATION)
so it was draining AudioQueue during pre-buffering, defeating it entirely.
- Audio pre-buffer default 2000ms (max 4000ms) to absorb ElevenLabs
server-side TTS inter-chunk gaps (~2s between chunks confirmed).
- Add diagnostic timestamps [T+Xs] in HandleAudioReceived and
AudioQueue DRY/recovered logs for debugging audio pipeline timing.
- Fix lip sync not returning to neutral: add snap-to-zero (< 0.01)
in blendshape smoothing pass and clean up PreviousBlendshapes to
prevent asymptotic Lerp residuals keeping mouth slightly open.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix text erasure between TTS audio chunks (bFullTextReceived guard):
partial text now persists across all chunks of the same utterance instead
of being erased after chunk 1's queue empties
- Add audio pre-buffering (AudioPreBufferMs, default 250ms) to absorb TTS
inter-chunk gaps and eliminate mid-sentence audio pauses
- Lip sync pauses viseme queue consumption during pre-buffer to stay in sync
- Inter-frame interpolation (lerp between consumed and next queued frame)
for smoother mouth transitions instead of 32ms step-wise jumps
- Reduce double-smoothing (blendshape smooth 0.8→0.4, release 0.5→0.65)
- Adjust duration weights (vowels 2.0/1.7, plosives 0.8, silence 1.0)
- UI range refinement (AmplitudeScale 0.5-1.0, SmoothingSpeed 35-65)
- Silence padding capped at 512 samples (32ms) to prevent buffer accumulation
- Audio playback restart on buffer underrun during speech
- Optimized log levels (most debug→Verbose, kept key diagnostics at Log)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Real-time lip sync component that performs client-side spectral analysis
on the agent's PCM audio stream (ElevenLabs doesn't provide viseme data).
Pipeline: 512-point FFT (16kHz) → 5 frequency bands → 15 OVR visemes
→ ARKit blendshapes (MetaHuman compatible) → auto-apply morph targets.
Currently uses SetMorphTarget() which may be overridden by MetaHuman's
Face AnimBP — face animation not yet working. Debug logs added to
diagnose: audio flow, spectrum energy, morph target name matching.
Next steps: verify debug output, fix MetaHuman morph target override
(likely needs AnimBP integration like Convai approach).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE was converting the raw ms value to seconds in the Details panel,
showing "0.1 s" instead of "100". Removing Units="ms" lets the slider
display the integer value directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All configuration and event properties in ConversationalAgentComponent and
MicrophoneCaptureComponent now have explicit ToolTip meta for clear descriptions
in the Unreal Editor Details panel.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Server VAD + interruption: mic stays open while agent speaks, server
detects user voice and triggers interruption automatically. Echo
suppression disabled in this mode so audio reaches the server.
- Fix agent_chat_response_part parsing: ElevenLabs API now uses
text_response_part.text instead of agent_chat_response_part_event.
Added fallback for legacy format.
- Expose MicChunkDurationMs as UPROPERTY (20-500ms, default 100ms)
instead of compile-time constant.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix regression from v1.7.0 where agent couldn't hear user speech:
- Restore AsyncTask game-thread dispatch for delegate broadcast (AddUObject
weak pointer checks are not thread-safe from WASAPI thread)
- Keep early echo suppression in WASAPI callback (before resampling)
- Keep MicChunkMinBytes at 3200 (100ms) for lower latency
- Add thread safety: std::atomic<bool> for bIsListening/bAgentSpeaking/bCapturing,
FCriticalSection for MicSendLock and WebSocketSendLock
- Add EchoSuppressFlag pointer from agent to mic component
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Skip mic buffer flush during collision avoidance (bAgentGenerating guard
in StopListening) to prevent sending audio to a mid-generation server
which caused both sides to stall permanently
- Add OnAgentPartialResponse event: streams LLM text fragments from
agent_chat_response_part in real-time (opt-in via bEnableAgentPartialResponse),
separate from the existing OnAgentTextResponse (full text at end)
- French agent server drop after 3 turns is a server-side issue, not client
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three bugs prevented voice input from working:
1. ResampleTo16000() treated NumFrames as total samples, dividing by
channel count again — losing half the audio data with stereo input.
The corrupted audio was unrecognizable to ElevenLabs VAD/STT.
2. Sent nonexistent "client_vad" turn mode in session init. The API has
no turn.mode field; replaced with turn_timeout parameter.
3. Sent user_activity with every audio chunk, which resets the turn
timeout timer and prevents the server from taking its turn.
Also: send audio chunks as compact JSON, add message type debug logging,
send conversation_initiation_client_data on connect.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
WASAPI fires mic callbacks every ~5ms (158 bytes at 16kHz 16-bit mono).
ElevenLabs VAD/STT requires a minimum of ~100ms (3200 bytes) per chunk.
Tiny fragments arrived at the server but were never processed, so the
agent never transcribed or responded to user speech.
Fix: OnMicrophoneDataCaptured now appends to MicAccumulationBuffer and
only calls SendAudioChunk once >= 3200 bytes are accumulated. StopListening
flushes any remaining bytes before sending UserTurnEnd so the final words
of an utterance are never discarded. HandleDisconnected also clears the
buffer to prevent stale data on reconnect.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
## Posture System — Diagonal Tilt Bug (à corriger)
### Problème
Le tilt diagonal (ear-to-shoulder) est PLUS VISIBLE avec la neck bone chain qu'avec le mono-bone.
### Piste de fix
Faire le swing-twist **PER-BONE** : pour chaque bone de la chaîne, composer le FractionalRot avec le bone's own rotation, swing-twist autour du bone's own tilt axis.
| Convai (reference) | `<repo_root>/ConvAI/Convai/` | gRPC + protobuf streaming to Convai API. Has ElevenLabs voice type enum in `ConvaiDefinitions.h`. Used as architectural reference. |
| **PS_AI_Agent_ElevenLabs** | `<repo_root>/Unreal/PS_AI_Agent/Plugins/PS_AI_Agent_ElevenLabs/` | Our ElevenLabs Conversational AI integration. See `.claude/elevenlabs_plugin.md` for full details. |
| Convai (reference) | `<repo_root>/ConvAI/Convai/` | gRPC + protobuf streaming to Convai API. Used as architectural reference. |
UE's libwebsockets backend fires **both**`OnMessage()` (text callback) **and**`OnRawMessage()` (binary callback) for the same incoming frame.
We had bound both `WebSocket->OnMessage()` and `WebSocket->OnRawMessage()` in `Connect()`.
Result: every audio frame was decoded and enqueued twice → played twice.
Fix: **Remove `OnMessage` binding entirely.**`OnRawMessage` now handles all frames (JSON control messages peeked via first byte, raw PCM otherwise).
**Bug 2 Root Cause — Mic Silent:**
ElevenLabs requires a `conversation_initiation_client_data` message sent **immediately** after the WebSocket handshake completes. Without it, the server never enters a state where it will accept and process client audio chunks. This is a required session negotiation step, not optional.
Fix: Send `conversation_initiation_client_data` in `OnWsConnected()` before any other message.
**Bug 2 Secondary — Delegate Stacking:**
`StartListening()` called `Mic->OnAudioCaptured.AddUObject(this, ...)` without first removing existing bindings. If called more than once (e.g. after reconnect), delegates stack up and audio is sent multiple times per frame.
Fix: Add `Mic->OnAudioCaptured.RemoveAll(this)` before `AddUObject` in `StartListening()`.
**Bug 3 — Latency:**
Added `"enable_intermediate_response": true` inside `custom_llm_extra_body` of the `conversation_initiation_client_data` message. Also added `optimize_streaming_latency: 3` in `conversation_config_override.tts`.
**Files changed:**
-`ElevenLabsWebSocketProxy.cpp`:
-`Connect()`: removed `OnMessage` binding
-`OnWsConnected()`: now sends full `conversation_initiation_client_data` JSON
-`ElevenLabsConversationalAgentComponent.cpp`:
-`StartListening()`: added `RemoveAll` guard before delegate binding
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.