v2.0.0: Syllable-core lip sync + pose-based phoneme mapping

Browse Source

Major rewrite of the lip sync system for natural, syllable-level
mouth animation using artist-crafted phoneme pose AnimSequences.

Pose system:
- ElevenLabsLipSyncPoseMap UDataAsset maps OVR visemes to AnimSequences
- Curve extraction at BeginPlay from MHF_AA, MHF_PBM, etc. poses
- 50+ CTRL_expressions curves per viseme for MetaHuman fidelity
- Editor factory for creating PoseMap assets via right-click menu

Syllable-core viseme filter:
- Only emit vowels (aa, E, ih, oh, ou) and lip-visible consonants
  (PP=P/B/M, FF=F/V, CH=CH/SH/J) — ~1-2 visemes per syllable
- Skip invisible tongue/throat consonants (DD, kk, SS, nn, RR, TH)
- Audio amplitude handles pauses naturally (no text-driven sil)
- Removes "frantic mouth" look from articulating every phoneme

French pronunciation fixes:
- Apostrophe handling: d'accord, s'il — char before ' not word-final
- E-muet fix for short words: je, de, le (≤2 chars) keep vowel
- Double consonant rules: cc, pp, ff, rr, bb, gg, dd

Silence detection (pose mode):
- Hybrid amplitude: per-window RMS for silence gating (32ms),
  chunk-level RMS for shape intensity (no jitter on 50+ curves)
- Intra-chunk pauses now visible (mouth properly closes during gaps)

Pose-mode optimizations:
- Skip spectral analysis (text visemes provide shape)
- Gentler viseme smoothing (attack ×0.7, release ×0.45)
- Skip blendshape double-smoothing (avoids vibration)
- Higher viseme weight threshold (0.05 vs 0.001)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

...

This commit is contained in:

j.foucher 2026-02-24 13:47:10 +01:00

parent 6543bc6785

commit aa3010bae8

232 changed files with 645 additions and 107 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN

Unreal/PS_AI_Agent/Content/ElevenLabsLipSyncPoseMap.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/AnimBP/A1D_MH_BEye.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/AnimBP/A2D_MH_EyeLook.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/AnimBP/B1D_MH_Movement.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/AnimBP/B1D_MH_MovementV2.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/AnimBP/B2D_MH_HeadLook.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/AnimBP/B2D_MH_HeadLookV2.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Convai_MetaHuman_BodyAnim.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Convai_MetaHuman_FaceAnim.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Instruments/Enm/E_Emotions.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Instruments/Enm/E_InConversation.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Instruments/Enm/E_Lips.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Instruments/Enm/E_Movement.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Instruments/Enm/E_TurnDirection.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Instruments/Functions/F_AIUtilities.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_BEye_Close.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_BEye_CloseExra.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_BEye_Open.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_Base.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_LooKEye_CC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_LooKEye_CD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_LooKEye_CU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_LooKEye_LC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_LooKEye_LD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_LooKEye_LU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_LooKEye_RC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_LooKEye_RD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Eyes/MHF_LooKEye_RU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_Base.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_HeadLooK_CC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_HeadLook_CD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_HeadLook_CU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_HeadLook_LC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_HeadLook_LD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_HeadLook_LU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_HeadLook_RC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_HeadLook_RD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/HeadLook/MHB_HeadLook_RU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Idle/MH_Idle01_UE.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Idle/MH_Idle02_UE.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Idle/MH_Idle03_UE.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_AA.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_AAn.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_CH.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_CHn.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_EE.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_EEn.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_EEn2.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_FV.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_II.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_IIn.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_KG.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_KGn.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_KGn2.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_NL.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_NLn.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_None.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_OH.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_OHn.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_OU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_PBM.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_PBMn.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_RR.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_SZ.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_SZn.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_TD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Lips/MHF_TH.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Movement/MM_Run_Fwd.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Movement/MM_Walk_Fwd.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Movement/MM_Walk_InPlace.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Talk/MH_Talk.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Talk/MH_Talk01.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Talk/MH_Talk02.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Talk/MH_Talk03.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Turn/MH_Turn_L180_UE.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Turn/MH_Turn_L180_UE_Montage.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Turn/MH_Turn_L90_UE.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Turn/MH_Turn_L90_UE_Montage.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Turn/MH_Turn_R180_UE.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Turn/MH_Turn_R180_UE_Montage.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Turn/MH_Turn_R90_UE.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions/Turn/MH_Turn_R90_UE_Montage.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_CC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_CD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_CU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_LBC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_LBD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_LBU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_LC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_LD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_LU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_RBC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_RBD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_RBU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_RC.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_RD.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/FP_Idle_AO_RU.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/AO/female_neck_01_neck_02_head_anim.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/Breaks/FP_Breaks_AdjustGlasses.uasset Normal file

Binary file not shown.

BIN

Unreal/PS_AI_Agent/Content/MetaHumans/Animations/Motions2/FP/Breaks/FP_Breaks_LookAround_RightToLeft.uasset Normal file

Binary file not shown.

Some files were not shown because too many files have changed in this diff Show More