Packaged build fixes:
- Use EvaluateCurveData() for emotion and lip sync curves (works with
compressed/cooked data instead of raw FloatCurves)
- Add FMemMark wrapper for game-thread curve evaluation (FBlendedCurve
uses TMemStackAllocator)
- Lazy binding in AnimNodes and LipSyncComponent for packaged build
component initialization order
- SetIsReplicatedByDefault(true) instead of SetIsReplicated(true)
- Load AudioCapture module explicitly in plugin StartupModule
- Bundle cacert.pem for SSL in packaged builds
- Add DirectoriesToAlwaysStageAsNonUFS for certificates
- Add EmotionPoseMap and LipSyncPoseMap .cpp implementations
Body tracking:
- Body tracking now activates on conversation start (HandleAgentResponseStarted)
instead of on selection, creating a natural notice→engage two-step:
eyes+head track on selection, body turns when agent starts responding
- SendTextMessage also enables body tracking for text input
Cleanup:
- Remove all temporary [DIAG] and [BODY] debug logs
- Gate PostureComponent periodic debug log behind bDebug flag
- Remove obsolete editor-time curve caches (CachedCurveData, RebuildCurveCache,
FPS_AI_ConvAgent_CachedEmotionCurves, FPS_AI_ConvAgent_CachedPoseCurves)
from EmotionPoseMap and LipSyncPoseMap — no longer needed since
EvaluateCurveData() reads compressed curves directly at runtime
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replicate conversation state (bNetIsConversing, NetConversatingPlayer) for exclusive NPC locking
- Opus encode TTS audio on server, multicast to all clients for shared playback
- Replicate emotion state (OnRep) so clients compute facial expressions locally
- Multicast speaking/interrupted/text events so lip sync and posture run locally
- Route mic audio via Server RPC (client→server→ElevenLabs WebSocket)
- LOD: cull audio beyond 30m, skip lip sync beyond 15m for non-speaker clients
- Auto-detect player disconnection and release NPC on authority
- InteractionComponent: skip occupied NPCs, auto-start conversation on selection
- No changes to LipSync, Posture, FacialExpression, MicCapture or AnimNodes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Body tracking is now activated by ElevenLabsComponent directly
in StartListening() and SendTextMessage(), instead of being
managed by InteractionComponent. This ensures the agent turns
its body toward the player on any form of conversation input.
InteractionComponent still disables body tracking on deselection.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent gains bEnableBodyTracking flag. When false, only
head and eyes track the target — body stays frozen.
InteractionComponent now:
- On posture attach: sets TargetActor but disables body tracking
(agent notices player with eyes+head only)
- On StartListening: enables body tracking (agent fully engages)
- On StopListening: disables body tracking
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Automatically calls StartListening/StopListening on the agent's
ElevenLabsComponent on selection/deselection. Enabled by default.
Disable for manual control (e.g. push-to-talk).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
PostureComponent: body no longer returns to original yaw when
TargetActor is cleared — only head and eyes return to neutral.
InteractionComponent: add AgentEyeLevelOffset (default 150cm) so
the view cone check targets chest height instead of feet, preventing
selection loss at close range.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
InteractionComponent now automatically sets/clears the agent's
PostureComponent TargetActor on selection/deselection, with
configurable attach/detach delays and a master toggle for manual control.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Aligns all class/struct/enum/delegate prefixes with the module name
PS_AI_ConvAgent. Removes redundant Conv_ from ElevenLabsComponent
(PS_AI_Agent_Conv_ElevenLabsComponent → PS_AI_ConvAgent_ElevenLabsComponent).
UI strings now use "PS AI ConvAgent". CoreRedirects updated for both
ElevenLabs* and PS_AI_Agent_* legacy references.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Separates generic systems (lip sync, posture, facial expression, mic)
from ElevenLabs-specific ones (conv agent, websocket, settings).
Generic classes now use PS_AI_Agent_ prefix; ElevenLabs-specific keep
_ElevenLabs suffix. CoreRedirects in DefaultEngine.ini ensure existing
.uasset references load correctly. Binaries/Intermediate deleted for
full rebuild.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add BodyDriftCompensation parameter (0→1) to counter-rotate head when
body animation moves the torso (bow, lean). Drift correction applied
AFTER swing-twist on the first chain bone only, preventing the
correction from being stripped by tilt decomposition or compounded
through the chain.
- Add bDrawDebugGaze toggle: draws per-eye debug lines from Face mesh
FACIAL_L/R_Eye bones (cyan = desired direction, green = actual bone
Z-axis gaze) to visually verify eye contact accuracy.
- Cache Face mesh separately from Body mesh for correct eye bone transforms.
- Use eye bone midpoint (Face mesh) as EyeOrigin for pitch calculation
instead of head bone, fixing vertical offset.
- Calibrate ARKit eye ranges: horizontal 30→40, vertical 20→35 to match
MetaHuman actual eye deflection per curve unit.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix eye compensation to use smooth Lerp blend instead of binary switch.
Uses Curve.Get() to read animation's existing CTRL values, then
Lerp(animCTRL, postureCTRL, Comp) for proportional blending.
- Split AnimationCompensation into HeadAnimationCompensation and
EyeAnimationCompensation for independent control per layer.
- Fix CTRL curve naming: use correct MetaHuman format
(CTRL_expressions_eyeLook{Dir}{L/R}) with proper In/Out→Left/Right mapping.
- Add eye diagnostic modes (disabled) for pipeline debugging.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add NeckBoneChain config (FElevenLabsNeckBoneEntry USTRUCT) to distribute
head rotation across multiple bones (e.g. neck_01=0.25, neck_02=0.35,
head=0.40) for more natural neck arc movement.
- Swing-twist now uses bone's actual tilt axis (BoneRot.RotateVector) instead
of hardcoded FVector::RightVector — accounts for MetaHuman ~11.5° reference
pose rotation on head bone.
- Clean up diagnostic logging: reduced to Log level, removed verbose per-frame
quaternion dumps.
- Backward compatible: empty NeckBoneChain falls back to single HeadBoneName.
KNOWN ISSUE: diagonal tilt is more visible with neck chain — needs investigation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix two-step body animation: head overflow now checked against
TargetBodyWorldYaw (body TARGET) instead of current body position,
preventing head overcompensation during body interpolation.
- Fix ARKit eye curve normalization: curves now normalized by fixed
physical range (30°/20°) instead of configurable MaxEye thresholds.
MaxEye controls cascade threshold only, not visual deflection.
- Fix diagonal head tilt: full FQuat pipeline (no FRotator round-trip)
+ swing-twist decomposition using bone's actual tilt axis (accounts
for MetaHuman ~11.5° reference pose rotation on head bone).
- Clean up diagnostic logging: reduced to Log level, removed verbose
per-frame quaternion dumps from investigation phase.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Rewrite posture system as a relative cascade (Eyes → Head → Body) with
persistent sticky targets. Each layer stays put until the layer below
overflows its max angle, then realigns fully toward the target.
Key changes:
- Thread safety: FCriticalSection protects AnimNode shared data
- Persistent TargetHeadYaw/Pitch: overflow checked against target (not
interpolating current) so head completes full realignment
- Persistent TargetBodyWorldYaw: body only moves when head+eyes combined
range is exceeded (sticky, same pattern as head)
- Quaternion head rotation: compose independent NodQuat × TurnQuat to
avoid diagonal coupling that FRotator causes
- Eye curves: negate CurrentEyeYaw for correct ARKit convention
- AnimNode: enhanced logging, axis diagnostic define (disabled)
- Remove old percentage/degree activation thresholds (max angles serve
as natural thresholds in the relative cascade)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
New ElevenLabsPostureComponent with 3-layer rotation distribution:
- Body (60%): rotates entire actor (yaw only) via SetActorRotation
- Head (20%): direct bone transform in AnimNode
- Eyes (10%): 8 ARKit eye look curves (eyeLookUp/Down/In/Out L/R)
Features:
- Configurable rotation percentages and angle limits per layer
- Smooth FInterpTo interpolation with per-layer speeds
- TargetActor + TargetOffset for any actor type (no skeleton required)
- Smooth return to neutral when TargetActor is cleared
- Blue AnimGraph node in ElevenLabs category
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Create dedicated UElevenLabsEmotionPoseMap data asset for emotions
- FacialExpressionComponent now uses EmotionPoseMap (not LipSyncPoseMap)
- Remove emotion data (FElevenLabsEmotionPoseSet, EmotionPoses TMap) from LipSyncPoseMap
- Add ElevenLabsEmotionPoseMapFactory for Content Browser asset creation
- Standardize log format to [T+Xs] [Turn N] in ConversationalAgentComponent
- Remove unused #include "Engine/World.h"
- Simplify collision log to single line
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- New AnimNode_ElevenLabsFacialExpression: independent AnimBP node for emotion expressions
- New AnimGraphNode (amber color) in ElevenLabs category for AnimBP editor
- Emotion AnimSequences now play in real-time (looping) instead of static pose at t=0
- Smooth crossfade between emotions with configurable duration
- LipSync AnimNode skips near-zero curves so emotion base layer shows through during silence
- Removed emotion merge from LipSyncComponent (now handled by AnimBP node ordering)
- Removed verbose per-tick VISEME debug log
- Two-layer AnimBP chain: FacialExpression → LipSync → mh_arkit_mapping_pose
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Decouple viseme timing from 32ms audio chunks by introducing an independent
FVisemeTimelineEntry timeline evaluated at render framerate. Playback-time
envelope tracking from consumed queue frames replaces arrival-time-only
updates, with fast 40ms decay when queue is empty.
- Viseme subsampling caps at ~10/sec (100ms min) to prevent saccades
- Full-duration quintic smootherstep crossfade (C2 continuous, no hold phase)
- Dead zone lowered to 0.15 for cleaner silence transitions
- TotalActiveFramesSeen cumulative counter for accurate timeline scaling
- Absolute cursor preservation on timeline rebuild
- Moderate Lerp smoothing (attack 0.55, release 0.40)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix event_id filtering bug: reset LastInterruptEventId when new generation
starts, preventing all audio from being silently dropped after an interruption
- Match C++ sample API config: remove optimize_streaming_latency and
custom_llm_extra_body overrides, send empty conversation_config_override
in Server VAD mode (only send turn_timeout in Client mode)
- Instant audio stop on interruption: call ResetAudio() before Stop() to
flush USoundWaveProcedural's internal ring buffer
- Lip sync reset on interruption/stop: bind OnAgentInterrupted (snap to
neutral) and OnAgentStoppedSpeaking (clear queues) events
- Revert jitter buffer (replaced by pre-buffer approach, default 2000ms)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Remove silence padding accumulation bug: QueueAudio'd silence was
accumulating in USoundWaveProcedural's internal buffer during TTS gaps,
delaying real audio by ~800ms. USoundWaveProcedural with
INDEFINITELY_LOOPING_DURATION generates silence internally instead.
- Fix pre-buffer bypass: guard OnProceduralUnderflow with bPreBuffering
check — the audio component never stops (INDEFINITELY_LOOPING_DURATION)
so it was draining AudioQueue during pre-buffering, defeating it entirely.
- Audio pre-buffer default 2000ms (max 4000ms) to absorb ElevenLabs
server-side TTS inter-chunk gaps (~2s between chunks confirmed).
- Add diagnostic timestamps [T+Xs] in HandleAudioReceived and
AudioQueue DRY/recovered logs for debugging audio pipeline timing.
- Fix lip sync not returning to neutral: add snap-to-zero (< 0.01)
in blendshape smoothing pass and clean up PreviousBlendshapes to
prevent asymptotic Lerp residuals keeping mouth slightly open.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix text erasure between TTS audio chunks (bFullTextReceived guard):
partial text now persists across all chunks of the same utterance instead
of being erased after chunk 1's queue empties
- Add audio pre-buffering (AudioPreBufferMs, default 250ms) to absorb TTS
inter-chunk gaps and eliminate mid-sentence audio pauses
- Lip sync pauses viseme queue consumption during pre-buffer to stay in sync
- Inter-frame interpolation (lerp between consumed and next queued frame)
for smoother mouth transitions instead of 32ms step-wise jumps
- Reduce double-smoothing (blendshape smooth 0.8→0.4, release 0.5→0.65)
- Adjust duration weights (vowels 2.0/1.7, plosives 0.8, silence 1.0)
- UI range refinement (AmplitudeScale 0.5-1.0, SmoothingSpeed 35-65)
- Silence padding capped at 512 samples (32ms) to prevent buffer accumulation
- Audio playback restart on buffer underrun during speech
- Optimized log levels (most debug→Verbose, kept key diagnostics at Log)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Real-time lip sync component that performs client-side spectral analysis
on the agent's PCM audio stream (ElevenLabs doesn't provide viseme data).
Pipeline: 512-point FFT (16kHz) → 5 frequency bands → 15 OVR visemes
→ ARKit blendshapes (MetaHuman compatible) → auto-apply morph targets.
Currently uses SetMorphTarget() which may be overridden by MetaHuman's
Face AnimBP — face animation not yet working. Debug logs added to
diagnose: audio flow, spectrum energy, morph target name matching.
Next steps: verify debug output, fix MetaHuman morph target override
(likely needs AnimBP integration like Convai approach).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
UE was converting the raw ms value to seconds in the Details panel,
showing "0.1 s" instead of "100". Removing Units="ms" lets the slider
display the integer value directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
All configuration and event properties in ConversationalAgentComponent and
MicrophoneCaptureComponent now have explicit ToolTip meta for clear descriptions
in the Unreal Editor Details panel.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Server VAD + interruption: mic stays open while agent speaks, server
detects user voice and triggers interruption automatically. Echo
suppression disabled in this mode so audio reaches the server.
- Fix agent_chat_response_part parsing: ElevenLabs API now uses
text_response_part.text instead of agent_chat_response_part_event.
Added fallback for legacy format.
- Expose MicChunkDurationMs as UPROPERTY (20-500ms, default 100ms)
instead of compile-time constant.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fix regression from v1.7.0 where agent couldn't hear user speech:
- Restore AsyncTask game-thread dispatch for delegate broadcast (AddUObject
weak pointer checks are not thread-safe from WASAPI thread)
- Keep early echo suppression in WASAPI callback (before resampling)
- Keep MicChunkMinBytes at 3200 (100ms) for lower latency
- Add thread safety: std::atomic<bool> for bIsListening/bAgentSpeaking/bCapturing,
FCriticalSection for MicSendLock and WebSocketSendLock
- Add EchoSuppressFlag pointer from agent to mic component
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Skip mic buffer flush during collision avoidance (bAgentGenerating guard
in StopListening) to prevent sending audio to a mid-generation server
which caused both sides to stall permanently
- Add OnAgentPartialResponse event: streams LLM text fragments from
agent_chat_response_part in real-time (opt-in via bEnableAgentPartialResponse),
separate from the existing OnAgentTextResponse (full text at end)
- French agent server drop after 3 turns is a server-side issue, not client
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
