Compare commits

...

8 Commits

Author SHA1 Message Date
7373959d8b v2.4.0: Posture component — multi-layer look-at (body/head/eyes)
New ElevenLabsPostureComponent with 3-layer rotation distribution:
- Body (60%): rotates entire actor (yaw only) via SetActorRotation
- Head (20%): direct bone transform in AnimNode
- Eyes (10%): 8 ARKit eye look curves (eyeLookUp/Down/In/Out L/R)

Features:
- Configurable rotation percentages and angle limits per layer
- Smooth FInterpTo interpolation with per-layer speeds
- TargetActor + TargetOffset for any actor type (no skeleton required)
- Smooth return to neutral when TargetActor is cleared
- Blue AnimGraph node in ElevenLabs category

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 21:16:57 +01:00
88c175909e v2.3.0: Separate emotion data asset + code cleanup
- Create dedicated UElevenLabsEmotionPoseMap data asset for emotions
- FacialExpressionComponent now uses EmotionPoseMap (not LipSyncPoseMap)
- Remove emotion data (FElevenLabsEmotionPoseSet, EmotionPoses TMap) from LipSyncPoseMap
- Add ElevenLabsEmotionPoseMapFactory for Content Browser asset creation
- Standardize log format to [T+Xs] [Turn N] in ConversationalAgentComponent
- Remove unused #include "Engine/World.h"
- Simplify collision log to single line

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 20:17:04 +01:00
f6541bd7e2 v2.2.0: Separate AnimNode for facial expressions + real-time emotion anim playback
- New AnimNode_ElevenLabsFacialExpression: independent AnimBP node for emotion expressions
- New AnimGraphNode (amber color) in ElevenLabs category for AnimBP editor
- Emotion AnimSequences now play in real-time (looping) instead of static pose at t=0
- Smooth crossfade between emotions with configurable duration
- LipSync AnimNode skips near-zero curves so emotion base layer shows through during silence
- Removed emotion merge from LipSyncComponent (now handled by AnimBP node ordering)
- Removed verbose per-tick VISEME debug log
- Two-layer AnimBP chain: FacialExpression → LipSync → mh_arkit_mapping_pose

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 19:13:18 +01:00
949efd5578 metahuman datas 2026-02-24 18:13:56 +01:00
f57bb65297 WIP: Emotion facial expressions + client tool support
- ElevenLabs client tool call/result WebSocket support (set_emotion)
- EElevenLabsEmotion + EElevenLabsEmotionIntensity enums
- Emotion poses in PoseMap data asset (Normal/Medium/Extreme per emotion)
- Standalone ElevenLabsFacialExpressionComponent (separated from LipSync)
- Two-layer architecture: emotion base (eyes/brows/cheeks) + lip sync on top (mouth)
- Smooth emotion blending (~500ms configurable transitions)
- LipSync reads emotion curves from FacialExpressionComponent via GetCurrentEmotionCurves()

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 18:08:23 +01:00
e57be0a1d9 v2.1.0: Decoupled viseme timeline with quintic crossfade and tuned dead zone
Decouple viseme timing from 32ms audio chunks by introducing an independent
FVisemeTimelineEntry timeline evaluated at render framerate. Playback-time
envelope tracking from consumed queue frames replaces arrival-time-only
updates, with fast 40ms decay when queue is empty.

- Viseme subsampling caps at ~10/sec (100ms min) to prevent saccades
- Full-duration quintic smootherstep crossfade (C2 continuous, no hold phase)
- Dead zone lowered to 0.15 for cleaner silence transitions
- TotalActiveFramesSeen cumulative counter for accurate timeline scaling
- Absolute cursor preservation on timeline rebuild
- Moderate Lerp smoothing (attack 0.55, release 0.40)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 16:46:36 +01:00
aa3010bae8 v2.0.0: Syllable-core lip sync + pose-based phoneme mapping
Major rewrite of the lip sync system for natural, syllable-level
mouth animation using artist-crafted phoneme pose AnimSequences.

Pose system:
- ElevenLabsLipSyncPoseMap UDataAsset maps OVR visemes to AnimSequences
- Curve extraction at BeginPlay from MHF_AA, MHF_PBM, etc. poses
- 50+ CTRL_expressions curves per viseme for MetaHuman fidelity
- Editor factory for creating PoseMap assets via right-click menu

Syllable-core viseme filter:
- Only emit vowels (aa, E, ih, oh, ou) and lip-visible consonants
  (PP=P/B/M, FF=F/V, CH=CH/SH/J) — ~1-2 visemes per syllable
- Skip invisible tongue/throat consonants (DD, kk, SS, nn, RR, TH)
- Audio amplitude handles pauses naturally (no text-driven sil)
- Removes "frantic mouth" look from articulating every phoneme

French pronunciation fixes:
- Apostrophe handling: d'accord, s'il — char before ' not word-final
- E-muet fix for short words: je, de, le (≤2 chars) keep vowel
- Double consonant rules: cc, pp, ff, rr, bb, gg, dd

Silence detection (pose mode):
- Hybrid amplitude: per-window RMS for silence gating (32ms),
  chunk-level RMS for shape intensity (no jitter on 50+ curves)
- Intra-chunk pauses now visible (mouth properly closes during gaps)

Pose-mode optimizations:
- Skip spectral analysis (text visemes provide shape)
- Gentler viseme smoothing (attack ×0.7, release ×0.45)
- Skip blendshape double-smoothing (avoids vibration)
- Higher viseme weight threshold (0.05 vs 0.001)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 13:47:10 +01:00
6543bc6785 v1.9.1: Fix audio loss after interruption, instant audio stop, lip sync reset
- Fix event_id filtering bug: reset LastInterruptEventId when new generation
  starts, preventing all audio from being silently dropped after an interruption
- Match C++ sample API config: remove optimize_streaming_latency and
  custom_llm_extra_body overrides, send empty conversation_config_override
  in Server VAD mode (only send turn_timeout in Client mode)
- Instant audio stop on interruption: call ResetAudio() before Stop() to
  flush USoundWaveProcedural's internal ring buffer
- Lip sync reset on interruption/stop: bind OnAgentInterrupted (snap to
  neutral) and OnAgentStoppedSpeaking (clear queues) events
- Revert jitter buffer (replaced by pre-buffer approach, default 2000ms)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:48:56 +01:00
322 changed files with 2881 additions and 254 deletions

Some files were not shown because too many files have changed in this diff Show More