UE5 limits replicated TArrays to 65535 elements. ElevenLabs sends
audio chunks up to ~72K bytes, exceeding this limit when sent as
raw PCM. Split into 32000-byte sub-chunks (1s of 16kHz 16-bit mono)
before calling MulticastReceiveAgentAudio.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE5 5.5's FVoiceModule returns NULL encoder/decoder, breaking
Opus-based audio replication. This adds a transparent fallback:
- Server sends raw PCM when OpusEncoder is null (~32KB/s, fine for LAN)
- Client accepts raw PCM when OpusDecoder is null
- Changed MulticastReceiveAgentAudio from Unreliable to Reliable
to handle larger uncompressed payloads without packet loss
- Added OnlineSubsystemNull config for future Opus compatibility
- Removed premature bAgentSpeaking=true from MulticastAgentStartedSpeaking
to fix race condition with audio initialization
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Log at InitOpusCodec: FVoiceModule availability, encoder/decoder creation, net role
- Log at HandleAudioReceived: warn once if encoder is null or role is not Authority
- These fire unconditionally (not behind bDebug) to catch the root cause
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Logs compressed audio size in HandleAudioReceived to diagnose
why MulticastReceiveAgentAudio (Unreliable) never reaches clients.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- OnRep_ConversationState now sets PostureComponent TargetActor from
replicated NetConversatingPlayer so remote clients see head/eyes tracking
- Activate FacialExpressionComponent and LipSyncComponent on remote clients
(OnAgentConnected never fires on clients since WebSocket is server-only)
- Fix audio race condition: MulticastAgentStartedSpeaking no longer sets
bAgentSpeaking prematurely, letting EnqueueAgentAudio handle the full
first-chunk initialization (pre-buffer, Play(), state reset)
- Add diagnostic logging to MulticastReceiveAgentAudio for silent failures
(OpusDecoder invalid, LOD culling, decode failure)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Introduces UPS_AI_ConvAgent_AgentConfig_ElevenLabs data asset to encapsulate
full agent configuration (voice, LLM, prompt, language, emotions) with a
custom Detail Customization providing:
- Voice/TTS Model/LLM/Language pickers with Fetch buttons (ElevenLabs API)
- LLM latency hints in dropdown (~250ms, ~700ms, etc.)
- Create/Update/Fetch Agent buttons for REST API CRUD
- Auto-fetch on editor open, auto-select first voice for new assets
- Prompt fragment management (language, multilingual, emotion tool)
- Smart defaults: gemini-2.5-flash LLM, eleven_turbo_v2_5 TTS, English
- Speed range expanded to 0.7-1.95 (was 0.7-1.2)
- bAutoStartConversation + StartConversationWithSelectedAgent() on InteractionComponent
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Saved/ is not staged in packaged builds, so Content/Certificates/ is the
only reliable location. Simplified code by removing Android-specific
writable fallback (Content/ works on all platforms with NonUFS staging).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Posture, FacialExpression, LipSync: bActive + ActivationBlendDuration for
smooth alpha blend in/out (linear interp via FInterpConstantTo).
- Auto-activation: components bind to OnAgentConnected/OnAgentDisconnected,
starting inactive and blending in when conversation begins.
- Without an agent component, bActive defaults to true (backward compatible).
- Add BlueprintFunctionLibrary with SetPostProcessAnimBlueprint helper
(wraps UE5.5 SetOverridePostProcessAnimBP for per-instance BP setup).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Whitelist Android in .uplugin Runtime module and handle
read-only APK paths for SSL certificate copy on Android
(ProjectSavedDir fallback). No change to Win64 behavior.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Only init, deinit, errors and critical warnings remain unconditional.
All turn-by-turn logs (mic open/close, agent generating/speaking/stopped,
emotion changes, audio chunks, text/viseme processing, latency, bone
resolution, selection, registration) are now gated behind component
bDebug or Settings->bVerboseLogging.
10 files, ~125 lines removed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Packaged build fixes:
- Use EvaluateCurveData() for emotion and lip sync curves (works with
compressed/cooked data instead of raw FloatCurves)
- Add FMemMark wrapper for game-thread curve evaluation (FBlendedCurve
uses TMemStackAllocator)
- Lazy binding in AnimNodes and LipSyncComponent for packaged build
component initialization order
- SetIsReplicatedByDefault(true) instead of SetIsReplicated(true)
- Load AudioCapture module explicitly in plugin StartupModule
- Bundle cacert.pem for SSL in packaged builds
- Add DirectoriesToAlwaysStageAsNonUFS for certificates
- Add EmotionPoseMap and LipSyncPoseMap .cpp implementations
Body tracking:
- Body tracking now activates on conversation start (HandleAgentResponseStarted)
instead of on selection, creating a natural notice→engage two-step:
eyes+head track on selection, body turns when agent starts responding
- SendTextMessage also enables body tracking for text input
Cleanup:
- Remove all temporary [DIAG] and [BODY] debug logs
- Gate PostureComponent periodic debug log behind bDebug flag
- Remove obsolete editor-time curve caches (CachedCurveData, RebuildCurveCache,
FPS_AI_ConvAgent_CachedEmotionCurves, FPS_AI_ConvAgent_CachedPoseCurves)
from EmotionPoseMap and LipSyncPoseMap — no longer needed since
EvaluateCurveData() reads compressed curves directly at runtime
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replicate conversation state (bNetIsConversing, NetConversatingPlayer) for exclusive NPC locking
- Opus encode TTS audio on server, multicast to all clients for shared playback
- Replicate emotion state (OnRep) so clients compute facial expressions locally
- Multicast speaking/interrupted/text events so lip sync and posture run locally
- Route mic audio via Server RPC (client→server→ElevenLabs WebSocket)
- LOD: cull audio beyond 30m, skip lip sync beyond 15m for non-speaker clients
- Auto-detect player disconnection and release NPC on authority
- InteractionComponent: skip occupied NPCs, auto-start conversation on selection
- No changes to LipSync, Posture, FacialExpression, MicCapture or AnimNodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Body tracking is now activated by ElevenLabsComponent directly
in StartListening() and SendTextMessage(), instead of being
managed by InteractionComponent. This ensures the agent turns
its body toward the player on any form of conversation input.
InteractionComponent still disables body tracking on deselection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent gains bEnableBodyTracking flag. When false, only
head and eyes track the target — body stays frozen.
InteractionComponent now:
- On posture attach: sets TargetActor but disables body tracking
(agent notices player with eyes+head only)
- On StartListening: enables body tracking (agent fully engages)
- On StopListening: disables body tracking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Automatically calls StartListening/StopListening on the agent's
ElevenLabsComponent on selection/deselection. Enabled by default.
Disable for manual control (e.g. push-to-talk).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent: body no longer returns to original yaw when
TargetActor is cleared — only head and eyes return to neutral.
InteractionComponent: add AgentEyeLevelOffset (default 150cm) so
the view cone check targets chest height instead of feet, preventing
selection loss at close range.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
InteractionComponent now automatically sets/clears the agent's
PostureComponent TargetActor on selection/deselection, with
configurable attach/detach delays and a master toggle for manual control.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aligns all class/struct/enum/delegate prefixes with the module name
PS_AI_ConvAgent. Removes redundant Conv_ from ElevenLabsComponent
(PS_AI_Agent_Conv_ElevenLabsComponent → PS_AI_ConvAgent_ElevenLabsComponent).
UI strings now use "PS AI ConvAgent". CoreRedirects updated for both
ElevenLabs* and PS_AI_Agent_* legacy references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Separates generic systems (lip sync, posture, facial expression, mic)
from ElevenLabs-specific ones (conv agent, websocket, settings).
Generic classes now use PS_AI_Agent_ prefix; ElevenLabs-specific keep
_ElevenLabs suffix. CoreRedirects in DefaultEngine.ini ensure existing
.uasset references load correctly. Binaries/Intermediate deleted for
full rebuild.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add BodyDriftCompensation parameter (0→1) to counter-rotate head when
body animation moves the torso (bow, lean). Drift correction applied
AFTER swing-twist on the first chain bone only, preventing the
correction from being stripped by tilt decomposition or compounded
through the chain.
- Add bDrawDebugGaze toggle: draws per-eye debug lines from Face mesh
FACIAL_L/R_Eye bones (cyan = desired direction, green = actual bone
Z-axis gaze) to visually verify eye contact accuracy.
- Cache Face mesh separately from Body mesh for correct eye bone transforms.
- Use eye bone midpoint (Face mesh) as EyeOrigin for pitch calculation
instead of head bone, fixing vertical offset.
- Calibrate ARKit eye ranges: horizontal 30→40, vertical 20→35 to match
MetaHuman actual eye deflection per curve unit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix eye compensation to use smooth Lerp blend instead of binary switch.
Uses Curve.Get() to read animation's existing CTRL values, then
Lerp(animCTRL, postureCTRL, Comp) for proportional blending.
- Split AnimationCompensation into HeadAnimationCompensation and
EyeAnimationCompensation for independent control per layer.
- Fix CTRL curve naming: use correct MetaHuman format
(CTRL_expressions_eyeLook{Dir}{L/R}) with proper In/Out→Left/Right mapping.
- Add eye diagnostic modes (disabled) for pipeline debugging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add NeckBoneChain config (FElevenLabsNeckBoneEntry USTRUCT) to distribute
head rotation across multiple bones (e.g. neck_01=0.25, neck_02=0.35,
head=0.40) for more natural neck arc movement.
- Swing-twist now uses bone's actual tilt axis (BoneRot.RotateVector) instead
of hardcoded FVector::RightVector — accounts for MetaHuman ~11.5° reference
pose rotation on head bone.
- Clean up diagnostic logging: reduced to Log level, removed verbose per-frame
quaternion dumps.
- Backward compatible: empty NeckBoneChain falls back to single HeadBoneName.
KNOWN ISSUE: diagonal tilt is more visible with neck chain — needs investigation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix two-step body animation: head overflow now checked against
TargetBodyWorldYaw (body TARGET) instead of current body position,
preventing head overcompensation during body interpolation.
- Fix ARKit eye curve normalization: curves now normalized by fixed
physical range (30°/20°) instead of configurable MaxEye thresholds.
MaxEye controls cascade threshold only, not visual deflection.
- Fix diagonal head tilt: full FQuat pipeline (no FRotator round-trip)
+ swing-twist decomposition using bone's actual tilt axis (accounts
for MetaHuman ~11.5° reference pose rotation on head bone).
- Clean up diagnostic logging: reduced to Log level, removed verbose
per-frame quaternion dumps from investigation phase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite posture system as a relative cascade (Eyes → Head → Body) with
persistent sticky targets. Each layer stays put until the layer below
overflows its max angle, then realigns fully toward the target.
Key changes:
- Thread safety: FCriticalSection protects AnimNode shared data
- Persistent TargetHeadYaw/Pitch: overflow checked against target (not
interpolating current) so head completes full realignment
- Persistent TargetBodyWorldYaw: body only moves when head+eyes combined
range is exceeded (sticky, same pattern as head)
- Quaternion head rotation: compose independent NodQuat × TurnQuat to
avoid diagonal coupling that FRotator causes
- Eye curves: negate CurrentEyeYaw for correct ARKit convention
- AnimNode: enhanced logging, axis diagnostic define (disabled)
- Remove old percentage/degree activation thresholds (max angles serve
as natural thresholds in the relative cascade)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New ElevenLabsPostureComponent with 3-layer rotation distribution:
- Body (60%): rotates entire actor (yaw only) via SetActorRotation
- Head (20%): direct bone transform in AnimNode
- Eyes (10%): 8 ARKit eye look curves (eyeLookUp/Down/In/Out L/R)
Features:
- Configurable rotation percentages and angle limits per layer
- Smooth FInterpTo interpolation with per-layer speeds
- TargetActor + TargetOffset for any actor type (no skeleton required)
- Smooth return to neutral when TargetActor is cleared
- Blue AnimGraph node in ElevenLabs category
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create dedicated UElevenLabsEmotionPoseMap data asset for emotions
- FacialExpressionComponent now uses EmotionPoseMap (not LipSyncPoseMap)
- Remove emotion data (FElevenLabsEmotionPoseSet, EmotionPoses TMap) from LipSyncPoseMap
- Add ElevenLabsEmotionPoseMapFactory for Content Browser asset creation
- Standardize log format to [T+Xs] [Turn N] in ConversationalAgentComponent
- Remove unused #include "Engine/World.h"
- Simplify collision log to single line
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New AnimNode_ElevenLabsFacialExpression: independent AnimBP node for emotion expressions
- New AnimGraphNode (amber color) in ElevenLabs category for AnimBP editor
- Emotion AnimSequences now play in real-time (looping) instead of static pose at t=0
- Smooth crossfade between emotions with configurable duration
- LipSync AnimNode skips near-zero curves so emotion base layer shows through during silence
- Removed emotion merge from LipSyncComponent (now handled by AnimBP node ordering)
- Removed verbose per-tick VISEME debug log
- Two-layer AnimBP chain: FacialExpression → LipSync → mh_arkit_mapping_pose
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
