UE5 clients cannot call Server RPCs on actors they don't own. NPC actors
are server-owned, causing "No owning connection" errors when remote clients
try to start conversations, send mic audio, or interrupt agents.
Solution: relay all client→NPC RPCs through the InteractionComponent on
the player's pawn (which IS owned by the client). The relay forwards
commands to the NPC's ElevenLabsComponent on the server side.
Changes:
- InteractionComponent: add Server relay RPCs (Start/End conversation,
mic audio, text message, interrupt) and Client relay RPCs
(ConversationStarted/Failed) with GetLifetimeReplicatedProps
- ElevenLabsComponent: implement FindLocalRelayComponent(), route all
client-side calls through relay (StartConversation, EndConversation,
SendTextMessage, InterruptAgent, FeedExternalAudio, mic capture)
- Fix HandleConnected/ServerRequestConversation to route Client RPCs
through the player pawn's relay instead of the NPC (no owning connection)
- Fix StartListening/FeedExternalAudio/StopListening to accept
bNetIsConversing on clients (WebSocket only exists on server)
- Fix EvaluateBestAgent to use NetConversatingPawn instead of
NetConversatingPlayer (NULL on remote clients due to bOnlyRelevantToOwner)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent starts with bActive=false and waits for
OnConversationConnected, which only fires on the server (WebSocket).
Remote clients never got bActive=true, so CurrentActiveAlpha stayed
at 0 and all head/eye rotation was zeroed out.
Now OnRep_ConversationState also sets Posture->bActive alongside
TargetActor, matching what was already done for FacialExpression
and LipSync components.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs NetConversatingPawn, PostureComponent availability, and TargetActor
assignment to diagnose why head tracking doesn't work on remote clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Raw PCM is split into 32KB sub-chunks for network transmission.
On the client, these sub-chunks arrive nearly simultaneously,
triggering the "second chunk arrived" fast-path which cancelled
the pre-buffer after ~10ms instead of the intended 2000ms.
Now the fast-path only applies on Authority (server) where
chunks represent genuine separate TTS batches. Clients always
wait the full pre-buffer duration via TickComponent timer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PlayerControllers have bOnlyRelevantToOwner=true, so NetConversatingPlayer
was always nullptr on remote clients. Added NetConversatingPawn (APawn*)
which IS replicated to all clients. OnRep_ConversationState now uses
NetConversatingPawn as the posture TargetActor, enabling head/eye
tracking on all clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 limits replicated TArrays to 65535 elements. ElevenLabs sends
audio chunks up to ~72K bytes, exceeding this limit when sent as
raw PCM. Split into 32000-byte sub-chunks (1s of 16kHz 16-bit mono)
before calling MulticastReceiveAgentAudio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 5.5's FVoiceModule returns NULL encoder/decoder, breaking
Opus-based audio replication. This adds a transparent fallback:
- Server sends raw PCM when OpusEncoder is null (~32KB/s, fine for LAN)
- Client accepts raw PCM when OpusDecoder is null
- Changed MulticastReceiveAgentAudio from Unreliable to Reliable
to handle larger uncompressed payloads without packet loss
- Added OnlineSubsystemNull config for future Opus compatibility
- Removed premature bAgentSpeaking=true from MulticastAgentStartedSpeaking
to fix race condition with audio initialization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Log at InitOpusCodec: FVoiceModule availability, encoder/decoder creation, net role
- Log at HandleAudioReceived: warn once if encoder is null or role is not Authority
- These fire unconditionally (not behind bDebug) to catch the root cause
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs compressed audio size in HandleAudioReceived to diagnose
why MulticastReceiveAgentAudio (Unreliable) never reaches clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- OnRep_ConversationState now sets PostureComponent TargetActor from
replicated NetConversatingPlayer so remote clients see head/eyes tracking
- Activate FacialExpressionComponent and LipSyncComponent on remote clients
(OnAgentConnected never fires on clients since WebSocket is server-only)
- Fix audio race condition: MulticastAgentStartedSpeaking no longer sets
bAgentSpeaking prematurely, letting EnqueueAgentAudio handle the full
first-chunk initialization (pre-buffer, Play(), state reset)
- Add diagnostic logging to MulticastReceiveAgentAudio for silent failures
(OpusDecoder invalid, LOD culling, decode failure)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces UPS_AI_ConvAgent_AgentConfig_ElevenLabs data asset to encapsulate
full agent configuration (voice, LLM, prompt, language, emotions) with a
custom Detail Customization providing:
- Voice/TTS Model/LLM/Language pickers with Fetch buttons (ElevenLabs API)
- LLM latency hints in dropdown (~250ms, ~700ms, etc.)
- Create/Update/Fetch Agent buttons for REST API CRUD
- Auto-fetch on editor open, auto-select first voice for new assets
- Prompt fragment management (language, multilingual, emotion tool)
- Smart defaults: gemini-2.5-flash LLM, eleven_turbo_v2_5 TTS, English
- Speed range expanded to 0.7-1.95 (was 0.7-1.2)
- bAutoStartConversation + StartConversationWithSelectedAgent() on InteractionComponent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Saved/ is not staged in packaged builds, so Content/Certificates/ is the
only reliable location. Simplified code by removing Android-specific
writable fallback (Content/ works on all platforms with NonUFS staging).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Posture, FacialExpression, LipSync: bActive + ActivationBlendDuration for
smooth alpha blend in/out (linear interp via FInterpConstantTo).
- Auto-activation: components bind to OnAgentConnected/OnAgentDisconnected,
starting inactive and blending in when conversation begins.
- Without an agent component, bActive defaults to true (backward compatible).
- Add BlueprintFunctionLibrary with SetPostProcessAnimBlueprint helper
(wraps UE5.5 SetOverridePostProcessAnimBP for per-instance BP setup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Whitelist Android in .uplugin Runtime module and handle
read-only APK paths for SSL certificate copy on Android
(ProjectSavedDir fallback). No change to Win64 behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only init, deinit, errors and critical warnings remain unconditional.
All turn-by-turn logs (mic open/close, agent generating/speaking/stopped,
emotion changes, audio chunks, text/viseme processing, latency, bone
resolution, selection, registration) are now gated behind component
bDebug or Settings->bVerboseLogging.
10 files, ~125 lines removed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Packaged build fixes:
- Use EvaluateCurveData() for emotion and lip sync curves (works with
compressed/cooked data instead of raw FloatCurves)
- Add FMemMark wrapper for game-thread curve evaluation (FBlendedCurve
uses TMemStackAllocator)
- Lazy binding in AnimNodes and LipSyncComponent for packaged build
component initialization order
- SetIsReplicatedByDefault(true) instead of SetIsReplicated(true)
- Load AudioCapture module explicitly in plugin StartupModule
- Bundle cacert.pem for SSL in packaged builds
- Add DirectoriesToAlwaysStageAsNonUFS for certificates
- Add EmotionPoseMap and LipSyncPoseMap .cpp implementations
Body tracking:
- Body tracking now activates on conversation start (HandleAgentResponseStarted)
instead of on selection, creating a natural notice→engage two-step:
eyes+head track on selection, body turns when agent starts responding
- SendTextMessage also enables body tracking for text input
Cleanup:
- Remove all temporary [DIAG] and [BODY] debug logs
- Gate PostureComponent periodic debug log behind bDebug flag
- Remove obsolete editor-time curve caches (CachedCurveData, RebuildCurveCache,
FPS_AI_ConvAgent_CachedEmotionCurves, FPS_AI_ConvAgent_CachedPoseCurves)
from EmotionPoseMap and LipSyncPoseMap — no longer needed since
EvaluateCurveData() reads compressed curves directly at runtime
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replicate conversation state (bNetIsConversing, NetConversatingPlayer) for exclusive NPC locking
- Opus encode TTS audio on server, multicast to all clients for shared playback
- Replicate emotion state (OnRep) so clients compute facial expressions locally
- Multicast speaking/interrupted/text events so lip sync and posture run locally
- Route mic audio via Server RPC (client→server→ElevenLabs WebSocket)
- LOD: cull audio beyond 30m, skip lip sync beyond 15m for non-speaker clients
- Auto-detect player disconnection and release NPC on authority
- InteractionComponent: skip occupied NPCs, auto-start conversation on selection
- No changes to LipSync, Posture, FacialExpression, MicCapture or AnimNodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Body tracking is now activated by ElevenLabsComponent directly
in StartListening() and SendTextMessage(), instead of being
managed by InteractionComponent. This ensures the agent turns
its body toward the player on any form of conversation input.
InteractionComponent still disables body tracking on deselection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent gains bEnableBodyTracking flag. When false, only
head and eyes track the target — body stays frozen.
InteractionComponent now:
- On posture attach: sets TargetActor but disables body tracking
(agent notices player with eyes+head only)
- On StartListening: enables body tracking (agent fully engages)
- On StopListening: disables body tracking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Automatically calls StartListening/StopListening on the agent's
ElevenLabsComponent on selection/deselection. Enabled by default.
Disable for manual control (e.g. push-to-talk).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent: body no longer returns to original yaw when
TargetActor is cleared — only head and eyes return to neutral.
InteractionComponent: add AgentEyeLevelOffset (default 150cm) so
the view cone check targets chest height instead of feet, preventing
selection loss at close range.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
InteractionComponent now automatically sets/clears the agent's
PostureComponent TargetActor on selection/deselection, with
configurable attach/detach delays and a master toggle for manual control.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aligns all class/struct/enum/delegate prefixes with the module name
PS_AI_ConvAgent. Removes redundant Conv_ from ElevenLabsComponent
(PS_AI_Agent_Conv_ElevenLabsComponent → PS_AI_ConvAgent_ElevenLabsComponent).
UI strings now use "PS AI ConvAgent". CoreRedirects updated for both
ElevenLabs* and PS_AI_Agent_* legacy references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Separates generic systems (lip sync, posture, facial expression, mic)
from ElevenLabs-specific ones (conv agent, websocket, settings).
Generic classes now use PS_AI_Agent_ prefix; ElevenLabs-specific keep
_ElevenLabs suffix. CoreRedirects in DefaultEngine.ini ensure existing
.uasset references load correctly. Binaries/Intermediate deleted for
full rebuild.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add BodyDriftCompensation parameter (0→1) to counter-rotate head when
body animation moves the torso (bow, lean). Drift correction applied
AFTER swing-twist on the first chain bone only, preventing the
correction from being stripped by tilt decomposition or compounded
through the chain.
- Add bDrawDebugGaze toggle: draws per-eye debug lines from Face mesh
FACIAL_L/R_Eye bones (cyan = desired direction, green = actual bone
Z-axis gaze) to visually verify eye contact accuracy.
- Cache Face mesh separately from Body mesh for correct eye bone transforms.
- Use eye bone midpoint (Face mesh) as EyeOrigin for pitch calculation
instead of head bone, fixing vertical offset.
- Calibrate ARKit eye ranges: horizontal 30→40, vertical 20→35 to match
MetaHuman actual eye deflection per curve unit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix eye compensation to use smooth Lerp blend instead of binary switch.
Uses Curve.Get() to read animation's existing CTRL values, then
Lerp(animCTRL, postureCTRL, Comp) for proportional blending.
- Split AnimationCompensation into HeadAnimationCompensation and
EyeAnimationCompensation for independent control per layer.
- Fix CTRL curve naming: use correct MetaHuman format
(CTRL_expressions_eyeLook{Dir}{L/R}) with proper In/Out→Left/Right mapping.
- Add eye diagnostic modes (disabled) for pipeline debugging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add NeckBoneChain config (FElevenLabsNeckBoneEntry USTRUCT) to distribute
head rotation across multiple bones (e.g. neck_01=0.25, neck_02=0.35,
head=0.40) for more natural neck arc movement.
- Swing-twist now uses bone's actual tilt axis (BoneRot.RotateVector) instead
of hardcoded FVector::RightVector — accounts for MetaHuman ~11.5° reference
pose rotation on head bone.
- Clean up diagnostic logging: reduced to Log level, removed verbose per-frame
quaternion dumps.
- Backward compatible: empty NeckBoneChain falls back to single HeadBoneName.
KNOWN ISSUE: diagonal tilt is more visible with neck chain — needs investigation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
