Compare commits

...

4 Commits

Author SHA1 Message Date
c2142f3e6b v1.9.0: Fix audio gaps, pre-buffer, and lip sync neutral pose
- Remove silence padding accumulation bug: QueueAudio'd silence was
  accumulating in USoundWaveProcedural's internal buffer during TTS gaps,
  delaying real audio by ~800ms. USoundWaveProcedural with
  INDEFINITELY_LOOPING_DURATION generates silence internally instead.
- Fix pre-buffer bypass: guard OnProceduralUnderflow with bPreBuffering
  check — the audio component never stops (INDEFINITELY_LOOPING_DURATION)
  so it was draining AudioQueue during pre-buffering, defeating it entirely.
- Audio pre-buffer default 2000ms (max 4000ms) to absorb ElevenLabs
  server-side TTS inter-chunk gaps (~2s between chunks confirmed).
- Add diagnostic timestamps [T+Xs] in HandleAudioReceived and
  AudioQueue DRY/recovered logs for debugging audio pipeline timing.
- Fix lip sync not returning to neutral: add snap-to-zero (< 0.01)
  in blendshape smoothing pass and clean up PreviousBlendshapes to
  prevent asymptotic Lerp residuals keeping mouth slightly open.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 20:37:23 +01:00
7dfffdbad8 Lip sync v2: text persistence across TTS chunks, audio pre-buffering, smoothing fixes
- Fix text erasure between TTS audio chunks (bFullTextReceived guard):
  partial text now persists across all chunks of the same utterance instead
  of being erased after chunk 1's queue empties
- Add audio pre-buffering (AudioPreBufferMs, default 250ms) to absorb TTS
  inter-chunk gaps and eliminate mid-sentence audio pauses
- Lip sync pauses viseme queue consumption during pre-buffer to stay in sync
- Inter-frame interpolation (lerp between consumed and next queued frame)
  for smoother mouth transitions instead of 32ms step-wise jumps
- Reduce double-smoothing (blendshape smooth 0.8→0.4, release 0.5→0.65)
- Adjust duration weights (vowels 2.0/1.7, plosives 0.8, silence 1.0)
- UI range refinement (AmplitudeScale 0.5-1.0, SmoothingSpeed 35-65)
- Silence padding capped at 512 samples (32ms) to prevent buffer accumulation
- Audio playback restart on buffer underrun during speech
- Optimized log levels (most debug→Verbose, kept key diagnostics at Log)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 19:34:36 +01:00
ce7a146ce9 Commit Lipsync WIP 2026-02-22 15:54:53 +01:00
224af6a27b WIP: Add ElevenLabsLipSyncComponent with spectral analysis lip sync
Real-time lip sync component that performs client-side spectral analysis
on the agent's PCM audio stream (ElevenLabs doesn't provide viseme data).

Pipeline: 512-point FFT (16kHz) → 5 frequency bands → 15 OVR visemes
→ ARKit blendshapes (MetaHuman compatible) → auto-apply morph targets.

Currently uses SetMorphTarget() which may be overridden by MetaHuman's
Face AnimBP — face animation not yet working. Debug logs added to
diagnose: audio flow, spectrum energy, morph target name matching.

Next steps: verify debug output, fix MetaHuman morph target override
(likely needs AnimBP integration like Convai approach).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-22 11:23:34 +01:00
329 changed files with 2381 additions and 8 deletions

File diff suppressed because one or more lines are too long

View File

@ -1,8 +1,8 @@
[/Script/EngineSettings.GameMapsSettings]
GameDefaultMap=/Game/TestMap.TestMap
EditorStartupMap=/Game/TestMap.TestMap
GameDefaultMap=/Game/Demo_VoiceOnly.Demo_VoiceOnly
EditorStartupMap=/Game/Demo_VoiceOnly.Demo_VoiceOnly
[/Script/Engine.RendererSettings]
r.AllowStaticLighting=False
@ -24,6 +24,9 @@ r.DefaultFeature.AutoExposure.ExtendDefaultLuminanceRange=True
r.DefaultFeature.LocalExposure.HighlightContrastScale=0.8
r.DefaultFeature.LocalExposure.ShadowContrastScale=0.8
r.GPUSkin.Support16BitBoneIndex=True
r.GPUSkin.UnlimitedBoneInfluences=True
SkeletalMesh.UseExperimentalChunking=1
[/Script/WindowsTargetPlatform.WindowsTargetSettings]
DefaultGraphicsRHI=DefaultGraphicsRHI_DX12
@ -91,4 +94,3 @@ ConnectionType=USBOnly
bUseManualIPAddress=False
ManualIPAddress=

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More