- FacialExpressionComponent: add FCriticalSection around emotion curves
to prevent race between TickComponent (game thread) and anim worker
thread — root cause of EXCEPTION_ACCESS_VIOLATION at Evaluate_AnyThread
- Remove deprecated FBlendedCurve::IsValid() guards (UE 5.5: always true)
- Body tracking: replace AddActorWorldRotation() with animation-only
pelvis bone rotation via AnimNode — eliminates replication tug-of-war
that caused client-side saccades when server overwrote local rotation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create OpusEncoder on ALL machines (was Authority-only) — clients now
encode mic audio, server decodes it; server still encodes agent audio
- FeedExternalAudio / OnMicrophoneDataCaptured: Opus-encode accumulated
PCM buffer before sending via ServerRelayMicAudio RPC on client path
(~200 bytes/100ms instead of 3200 bytes = ~16 Kbits/s vs 256 Kbits/s)
- ServerRelayMicAudio_Implementation: auto-detect Opus (size < raw chunk)
and decode back to PCM before forwarding to WebSocket
- Add public DecompressMicAudio() helper for clean API access from
InteractionComponent relay without exposing private Opus members
- Graceful fallback: if Opus unavailable, raw PCM is sent/received as before
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Attach AudioPlaybackComponent to owner's root component for proper
3D world positioning (was unattached = stuck at origin)
- Enable default inline spatialization with 15m falloff distance when
no external SoundAttenuation asset is set
- External SoundAttenuation asset still overrides the default if set
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Auto-end conversation on deselection: when bAutoStartConversation is
true and the player walks out of range, EndConversation() is called
so the NPC becomes available for other players
- Server-side posture: add ApplyConversationPosture() helper called
from ServerRequestConversation, ServerReleaseConversation,
EndConversation (Authority path), and HandleDisconnected — fixes
NPC not tracking the client on the listen server (OnRep never fires
on Authority)
- Guard DetachPostureTarget: only clear TargetActor if it matches our
pawn, preventing the server IC from overwriting posture set by the
conversation system for a remote client
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Silence gate: skip sending silent mic audio over network RPCs on clients
(~256 Kbits/s saved when not speaking, fixes chaotic teleporting)
- Lazy init: defer InteractionComponent mic creation from BeginPlay to
TickComponent with IsLocallyControlled guard (fixes "No owning connection"
from server-side replicas of remote pawns)
- Body tracking: use bNetIsConversing as fallback for IsConnected() on
clients where WebSocket doesn't exist
- EvaluateBestAgent: null-check NetConversatingPawn before comparison
- MicCaptureComponent: use TWeakObjectPtr in AsyncTask lambda to prevent
FMRSWRecursiveAccessDetector race on component destruction
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
In a listen server, the server-side copy of a remote client's pawn also
has an InteractionComponent that ticks. This caused a race condition:
the server-side tick would start conversations using GetFirstPlayerController()
(= server's PC), setting NetConversatingPawn to the server's pawn instead
of the client's. The client's relay RPC arrived too late and was rejected
because bNetIsConversing was already true.
Fix: disable tick and skip mic creation in BeginPlay for non-locally-controlled
pawns. The client handles all interaction locally via relay RPCs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 clients cannot call Server RPCs on actors they don't own. NPC actors
are server-owned, causing "No owning connection" errors when remote clients
try to start conversations, send mic audio, or interrupt agents.
Solution: relay all client→NPC RPCs through the InteractionComponent on
the player's pawn (which IS owned by the client). The relay forwards
commands to the NPC's ElevenLabsComponent on the server side.
Changes:
- InteractionComponent: add Server relay RPCs (Start/End conversation,
mic audio, text message, interrupt) and Client relay RPCs
(ConversationStarted/Failed) with GetLifetimeReplicatedProps
- ElevenLabsComponent: implement FindLocalRelayComponent(), route all
client-side calls through relay (StartConversation, EndConversation,
SendTextMessage, InterruptAgent, FeedExternalAudio, mic capture)
- Fix HandleConnected/ServerRequestConversation to route Client RPCs
through the player pawn's relay instead of the NPC (no owning connection)
- Fix StartListening/FeedExternalAudio/StopListening to accept
bNetIsConversing on clients (WebSocket only exists on server)
- Fix EvaluateBestAgent to use NetConversatingPawn instead of
NetConversatingPlayer (NULL on remote clients due to bOnlyRelevantToOwner)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent starts with bActive=false and waits for
OnConversationConnected, which only fires on the server (WebSocket).
Remote clients never got bActive=true, so CurrentActiveAlpha stayed
at 0 and all head/eye rotation was zeroed out.
Now OnRep_ConversationState also sets Posture->bActive alongside
TargetActor, matching what was already done for FacialExpression
and LipSync components.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs NetConversatingPawn, PostureComponent availability, and TargetActor
assignment to diagnose why head tracking doesn't work on remote clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raw PCM is split into 32KB sub-chunks for network transmission.
On the client, these sub-chunks arrive nearly simultaneously,
triggering the "second chunk arrived" fast-path which cancelled
the pre-buffer after ~10ms instead of the intended 2000ms.
Now the fast-path only applies on Authority (server) where
chunks represent genuine separate TTS batches. Clients always
wait the full pre-buffer duration via TickComponent timer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PlayerControllers have bOnlyRelevantToOwner=true, so NetConversatingPlayer
was always nullptr on remote clients. Added NetConversatingPawn (APawn*)
which IS replicated to all clients. OnRep_ConversationState now uses
NetConversatingPawn as the posture TargetActor, enabling head/eye
tracking on all clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 limits replicated TArrays to 65535 elements. ElevenLabs sends
audio chunks up to ~72K bytes, exceeding this limit when sent as
raw PCM. Split into 32000-byte sub-chunks (1s of 16kHz 16-bit mono)
before calling MulticastReceiveAgentAudio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 5.5's FVoiceModule returns NULL encoder/decoder, breaking
Opus-based audio replication. This adds a transparent fallback:
- Server sends raw PCM when OpusEncoder is null (~32KB/s, fine for LAN)
- Client accepts raw PCM when OpusDecoder is null
- Changed MulticastReceiveAgentAudio from Unreliable to Reliable
to handle larger uncompressed payloads without packet loss
- Added OnlineSubsystemNull config for future Opus compatibility
- Removed premature bAgentSpeaking=true from MulticastAgentStartedSpeaking
to fix race condition with audio initialization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Log at InitOpusCodec: FVoiceModule availability, encoder/decoder creation, net role
- Log at HandleAudioReceived: warn once if encoder is null or role is not Authority
- These fire unconditionally (not behind bDebug) to catch the root cause
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs compressed audio size in HandleAudioReceived to diagnose
why MulticastReceiveAgentAudio (Unreliable) never reaches clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- OnRep_ConversationState now sets PostureComponent TargetActor from
replicated NetConversatingPlayer so remote clients see head/eyes tracking
- Activate FacialExpressionComponent and LipSyncComponent on remote clients
(OnAgentConnected never fires on clients since WebSocket is server-only)
- Fix audio race condition: MulticastAgentStartedSpeaking no longer sets
bAgentSpeaking prematurely, letting EnqueueAgentAudio handle the full
first-chunk initialization (pre-buffer, Play(), state reset)
- Add diagnostic logging to MulticastReceiveAgentAudio for silent failures
(OpusDecoder invalid, LOD culling, decode failure)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces UPS_AI_ConvAgent_AgentConfig_ElevenLabs data asset to encapsulate
full agent configuration (voice, LLM, prompt, language, emotions) with a
custom Detail Customization providing:
- Voice/TTS Model/LLM/Language pickers with Fetch buttons (ElevenLabs API)
- LLM latency hints in dropdown (~250ms, ~700ms, etc.)
- Create/Update/Fetch Agent buttons for REST API CRUD
- Auto-fetch on editor open, auto-select first voice for new assets
- Prompt fragment management (language, multilingual, emotion tool)
- Smart defaults: gemini-2.5-flash LLM, eleven_turbo_v2_5 TTS, English
- Speed range expanded to 0.7-1.95 (was 0.7-1.2)
- bAutoStartConversation + StartConversationWithSelectedAgent() on InteractionComponent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Saved/ is not staged in packaged builds, so Content/Certificates/ is the
only reliable location. Simplified code by removing Android-specific
writable fallback (Content/ works on all platforms with NonUFS staging).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Posture, FacialExpression, LipSync: bActive + ActivationBlendDuration for
smooth alpha blend in/out (linear interp via FInterpConstantTo).
- Auto-activation: components bind to OnAgentConnected/OnAgentDisconnected,
starting inactive and blending in when conversation begins.
- Without an agent component, bActive defaults to true (backward compatible).
- Add BlueprintFunctionLibrary with SetPostProcessAnimBlueprint helper
(wraps UE5.5 SetOverridePostProcessAnimBP for per-instance BP setup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Whitelist Android in .uplugin Runtime module and handle
read-only APK paths for SSL certificate copy on Android
(ProjectSavedDir fallback). No change to Win64 behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only init, deinit, errors and critical warnings remain unconditional.
All turn-by-turn logs (mic open/close, agent generating/speaking/stopped,
emotion changes, audio chunks, text/viseme processing, latency, bone
resolution, selection, registration) are now gated behind component
bDebug or Settings->bVerboseLogging.
10 files, ~125 lines removed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Packaged build fixes:
- Use EvaluateCurveData() for emotion and lip sync curves (works with
compressed/cooked data instead of raw FloatCurves)
- Add FMemMark wrapper for game-thread curve evaluation (FBlendedCurve
uses TMemStackAllocator)
- Lazy binding in AnimNodes and LipSyncComponent for packaged build
component initialization order
- SetIsReplicatedByDefault(true) instead of SetIsReplicated(true)
- Load AudioCapture module explicitly in plugin StartupModule
- Bundle cacert.pem for SSL in packaged builds
- Add DirectoriesToAlwaysStageAsNonUFS for certificates
- Add EmotionPoseMap and LipSyncPoseMap .cpp implementations
Body tracking:
- Body tracking now activates on conversation start (HandleAgentResponseStarted)
instead of on selection, creating a natural notice→engage two-step:
eyes+head track on selection, body turns when agent starts responding
- SendTextMessage also enables body tracking for text input
Cleanup:
- Remove all temporary [DIAG] and [BODY] debug logs
- Gate PostureComponent periodic debug log behind bDebug flag
- Remove obsolete editor-time curve caches (CachedCurveData, RebuildCurveCache,
FPS_AI_ConvAgent_CachedEmotionCurves, FPS_AI_ConvAgent_CachedPoseCurves)
from EmotionPoseMap and LipSyncPoseMap — no longer needed since
EvaluateCurveData() reads compressed curves directly at runtime
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replicate conversation state (bNetIsConversing, NetConversatingPlayer) for exclusive NPC locking
- Opus encode TTS audio on server, multicast to all clients for shared playback
- Replicate emotion state (OnRep) so clients compute facial expressions locally
- Multicast speaking/interrupted/text events so lip sync and posture run locally
- Route mic audio via Server RPC (client→server→ElevenLabs WebSocket)
- LOD: cull audio beyond 30m, skip lip sync beyond 15m for non-speaker clients
- Auto-detect player disconnection and release NPC on authority
- InteractionComponent: skip occupied NPCs, auto-start conversation on selection
- No changes to LipSync, Posture, FacialExpression, MicCapture or AnimNodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Body tracking is now activated by ElevenLabsComponent directly
in StartListening() and SendTextMessage(), instead of being
managed by InteractionComponent. This ensures the agent turns
its body toward the player on any form of conversation input.
InteractionComponent still disables body tracking on deselection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent gains bEnableBodyTracking flag. When false, only
head and eyes track the target — body stays frozen.
InteractionComponent now:
- On posture attach: sets TargetActor but disables body tracking
(agent notices player with eyes+head only)
- On StartListening: enables body tracking (agent fully engages)
- On StopListening: disables body tracking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Automatically calls StartListening/StopListening on the agent's
ElevenLabsComponent on selection/deselection. Enabled by default.
Disable for manual control (e.g. push-to-talk).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent: body no longer returns to original yaw when
TargetActor is cleared — only head and eyes return to neutral.
InteractionComponent: add AgentEyeLevelOffset (default 150cm) so
the view cone check targets chest height instead of feet, preventing
selection loss at close range.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
InteractionComponent now automatically sets/clears the agent's
PostureComponent TargetActor on selection/deselection, with
configurable attach/detach delays and a master toggle for manual control.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aligns all class/struct/enum/delegate prefixes with the module name
PS_AI_ConvAgent. Removes redundant Conv_ from ElevenLabsComponent
(PS_AI_Agent_Conv_ElevenLabsComponent → PS_AI_ConvAgent_ElevenLabsComponent).
UI strings now use "PS AI ConvAgent". CoreRedirects updated for both
ElevenLabs* and PS_AI_Agent_* legacy references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
