j.foucher/PS_AI_Agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rename: unify prefix PS_AI_Agent_ → PS_AI_ConvAgent_

Browse Source 
Aligns all class/struct/enum/delegate prefixes with the module name
PS_AI_ConvAgent. Removes redundant Conv_ from ElevenLabsComponent
(PS_AI_Agent_Conv_ElevenLabsComponent → PS_AI_ConvAgent_ElevenLabsComponent).
UI strings now use "PS AI ConvAgent". CoreRedirects updated for both
ElevenLabs* and PS_AI_Agent_* legacy references.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
j.foucher 2026-02-26 17:33:47 +01:00
parent eeb0b1172b
commit b3bfabc52d
42 changed files with 909 additions and 879 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

20
.claude/CLAUDE.md
View File

@ -1,10 +1,12 @@
# PS_AI_Agent — Notes pour Claude
# PS_AI_ConvAgent — Notes pour Claude
## Plugin Rename (completed)
- Plugin: `PS_AI_Agent_ElevenLabs` → `PS_AI_ConvAgent`
- Generic classes: `ElevenLabs*` → `PS_AI_Agent_*` (e.g. `UPS_AI_Agent_PostureComponent`)
- ElevenLabs-specific classes keep `_ElevenLabs` suffix (e.g. `UPS_AI_Agent_Conv_ElevenLabsComponent`)
- CoreRedirects in DefaultEngine.ini handle .uasset references
## Naming Convention
- **Module**: `PS_AI_ConvAgent` / `PS_AI_ConvAgentEditor`
- **API macro**: `PS_AI_CONVAGENT_API`
- **Class prefix**: `PS_AI_ConvAgent_` (e.g. `UPS_AI_ConvAgent_PostureComponent`)
- **ElevenLabs-specific**: suffix `_ElevenLabs` (e.g. `UPS_AI_ConvAgent_ElevenLabsComponent`)
- **UI display**: `"PS AI ConvAgent"` (categories, DisplayName)
- **CoreRedirects**: DefaultEngine.ini handles both ElevenLabs* and PS_AI_Agent_* → PS_AI_ConvAgent_*
## Posture System — Diagonal Tilt Bug (à corriger)
@ -18,15 +20,15 @@ Le swing-twist est fait une seule fois sur le tip bone (head), puis le CleanOffs
Faire le swing-twist **PER-BONE** : pour chaque bone de la chaîne, composer le FractionalRot avec le bone's own rotation, swing-twist autour du bone's own tilt axis, puis appliquer le swing seulement.
### Fichiers concernés
- `AnimNode_PS_AI_Agent_Posture.cpp` — Evaluate_AnyThread, section multi-bone chain
- `AnimNode_PS_AI_ConvAgent_Posture.cpp` — Evaluate_AnyThread, section multi-bone chain
- Commit actuel: `8df6967` sur main
### Config actuelle neck chain (BP_Taro)
neck_01=0.25, neck_02=0.35, head=0.40
## Architecture Posture
- **PS_AI_Agent_PostureComponent** (game thread) : cascade Eyes→Head→Body, produit FQuat + eye curves
- **AnimNode_PS_AI_Agent_Posture** (anim thread) : applique rotation sur bone(s) + injecte curves
- **PS_AI_ConvAgent_PostureComponent** (game thread) : cascade Eyes→Head→Body, produit FQuat + eye curves
- **AnimNode_PS_AI_ConvAgent_Posture** (anim thread) : applique rotation sur bone(s) + injecte curves
- Pipeline 100% FQuat (pas de round-trip FRotator)
- Thread safety via FCriticalSection
- ARKit eye curves normalisées par range fixe (40°/35°), pas par MaxEye threshold

82
Unreal/PS_AI_Agent/Config/DefaultEngine.ini
View File

@ -85,45 +85,73 @@ FontDPI=72
+PackageRedirects=(OldName="/Script/PS_AI_Agent_ElevenLabs", NewName="/Script/PS_AI_ConvAgent")
+PackageRedirects=(OldName="/Script/PS_AI_Agent_ElevenLabsEditor", NewName="/Script/PS_AI_ConvAgentEditor")
; ── Generic classes (ElevenLabs → PS_AI_Agent) ──
+ClassRedirects=(OldName="ElevenLabsLipSyncComponent", NewName="PS_AI_Agent_LipSyncComponent")
+ClassRedirects=(OldName="ElevenLabsFacialExpressionComponent", NewName="PS_AI_Agent_FacialExpressionComponent")
+ClassRedirects=(OldName="ElevenLabsPostureComponent", NewName="PS_AI_Agent_PostureComponent")
+ClassRedirects=(OldName="ElevenLabsMicrophoneCaptureComponent", NewName="PS_AI_Agent_MicrophoneCaptureComponent")
+ClassRedirects=(OldName="ElevenLabsLipSyncPoseMap", NewName="PS_AI_Agent_LipSyncPoseMap")
+ClassRedirects=(OldName="ElevenLabsEmotionPoseMap", NewName="PS_AI_Agent_EmotionPoseMap")
; ── Generic classes: ElevenLabs → PS_AI_ConvAgent ──
+ClassRedirects=(OldName="ElevenLabsLipSyncComponent", NewName="PS_AI_ConvAgent_LipSyncComponent")
+ClassRedirects=(OldName="ElevenLabsFacialExpressionComponent", NewName="PS_AI_ConvAgent_FacialExpressionComponent")
+ClassRedirects=(OldName="ElevenLabsPostureComponent", NewName="PS_AI_ConvAgent_PostureComponent")
+ClassRedirects=(OldName="ElevenLabsMicrophoneCaptureComponent", NewName="PS_AI_ConvAgent_MicrophoneCaptureComponent")
+ClassRedirects=(OldName="ElevenLabsLipSyncPoseMap", NewName="PS_AI_ConvAgent_LipSyncPoseMap")
+ClassRedirects=(OldName="ElevenLabsEmotionPoseMap", NewName="PS_AI_ConvAgent_EmotionPoseMap")
; ── Generic classes: PS_AI_Agent → PS_AI_ConvAgent (intermediate rename) ──
+ClassRedirects=(OldName="PS_AI_Agent_LipSyncComponent", NewName="PS_AI_ConvAgent_LipSyncComponent")
+ClassRedirects=(OldName="PS_AI_Agent_FacialExpressionComponent", NewName="PS_AI_ConvAgent_FacialExpressionComponent")
+ClassRedirects=(OldName="PS_AI_Agent_PostureComponent", NewName="PS_AI_ConvAgent_PostureComponent")
+ClassRedirects=(OldName="PS_AI_Agent_MicrophoneCaptureComponent", NewName="PS_AI_ConvAgent_MicrophoneCaptureComponent")
+ClassRedirects=(OldName="PS_AI_Agent_LipSyncPoseMap", NewName="PS_AI_ConvAgent_LipSyncPoseMap")
+ClassRedirects=(OldName="PS_AI_Agent_EmotionPoseMap", NewName="PS_AI_ConvAgent_EmotionPoseMap")
; ── ElevenLabs-specific classes ──
+ClassRedirects=(OldName="ElevenLabsConversationalAgentComponent", NewName="PS_AI_Agent_Conv_ElevenLabsComponent")
+ClassRedirects=(OldName="ElevenLabsWebSocketProxy", NewName="PS_AI_Agent_WebSocket_ElevenLabsProxy")
+ClassRedirects=(OldName="ElevenLabsSettings", NewName="PS_AI_Agent_Settings_ElevenLabs")
+ClassRedirects=(OldName="ElevenLabsConversationalAgentComponent", NewName="PS_AI_ConvAgent_ElevenLabsComponent")
+ClassRedirects=(OldName="ElevenLabsWebSocketProxy", NewName="PS_AI_ConvAgent_WebSocket_ElevenLabsProxy")
+ClassRedirects=(OldName="ElevenLabsSettings", NewName="PS_AI_ConvAgent_Settings_ElevenLabs")
+ClassRedirects=(OldName="PS_AI_Agent_Conv_ElevenLabsComponent", NewName="PS_AI_ConvAgent_ElevenLabsComponent")
+ClassRedirects=(OldName="PS_AI_Agent_WebSocket_ElevenLabsProxy", NewName="PS_AI_ConvAgent_WebSocket_ElevenLabsProxy")
+ClassRedirects=(OldName="PS_AI_Agent_Settings_ElevenLabs", NewName="PS_AI_ConvAgent_Settings_ElevenLabs")
; ── AnimNode structs ──
+StructRedirects=(OldName="AnimNode_ElevenLabsLipSync", NewName="AnimNode_PS_AI_Agent_LipSync")
+StructRedirects=(OldName="AnimNode_ElevenLabsFacialExpression", NewName="AnimNode_PS_AI_Agent_FacialExpression")
+StructRedirects=(OldName="AnimNode_ElevenLabsPosture", NewName="AnimNode_PS_AI_Agent_Posture")
+StructRedirects=(OldName="AnimNode_ElevenLabsLipSync", NewName="AnimNode_PS_AI_ConvAgent_LipSync")
+StructRedirects=(OldName="AnimNode_ElevenLabsFacialExpression", NewName="AnimNode_PS_AI_ConvAgent_FacialExpression")
+StructRedirects=(OldName="AnimNode_ElevenLabsPosture", NewName="AnimNode_PS_AI_ConvAgent_Posture")
+StructRedirects=(OldName="AnimNode_PS_AI_Agent_LipSync", NewName="AnimNode_PS_AI_ConvAgent_LipSync")
+StructRedirects=(OldName="AnimNode_PS_AI_Agent_FacialExpression", NewName="AnimNode_PS_AI_ConvAgent_FacialExpression")
+StructRedirects=(OldName="AnimNode_PS_AI_Agent_Posture", NewName="AnimNode_PS_AI_ConvAgent_Posture")
; ── AnimGraphNode classes ──
+ClassRedirects=(OldName="AnimGraphNode_ElevenLabsLipSync", NewName="AnimGraphNode_PS_AI_Agent_LipSync")
+ClassRedirects=(OldName="AnimGraphNode_ElevenLabsFacialExpression", NewName="AnimGraphNode_PS_AI_Agent_FacialExpression")
+ClassRedirects=(OldName="AnimGraphNode_ElevenLabsPosture", NewName="AnimGraphNode_PS_AI_Agent_Posture")
+ClassRedirects=(OldName="AnimGraphNode_ElevenLabsLipSync", NewName="AnimGraphNode_PS_AI_ConvAgent_LipSync")
+ClassRedirects=(OldName="AnimGraphNode_ElevenLabsFacialExpression", NewName="AnimGraphNode_PS_AI_ConvAgent_FacialExpression")
+ClassRedirects=(OldName="AnimGraphNode_ElevenLabsPosture", NewName="AnimGraphNode_PS_AI_ConvAgent_Posture")
+ClassRedirects=(OldName="AnimGraphNode_PS_AI_Agent_LipSync", NewName="AnimGraphNode_PS_AI_ConvAgent_LipSync")
+ClassRedirects=(OldName="AnimGraphNode_PS_AI_Agent_FacialExpression", NewName="AnimGraphNode_PS_AI_ConvAgent_FacialExpression")
+ClassRedirects=(OldName="AnimGraphNode_PS_AI_Agent_Posture", NewName="AnimGraphNode_PS_AI_ConvAgent_Posture")
; ── Factory classes ──
+ClassRedirects=(OldName="ElevenLabsLipSyncPoseMapFactory", NewName="PS_AI_Agent_LipSyncPoseMapFactory")
+ClassRedirects=(OldName="ElevenLabsEmotionPoseMapFactory", NewName="PS_AI_Agent_EmotionPoseMapFactory")
+ClassRedirects=(OldName="ElevenLabsLipSyncPoseMapFactory", NewName="PS_AI_ConvAgent_LipSyncPoseMapFactory")
+ClassRedirects=(OldName="ElevenLabsEmotionPoseMapFactory", NewName="PS_AI_ConvAgent_EmotionPoseMapFactory")
+ClassRedirects=(OldName="PS_AI_Agent_LipSyncPoseMapFactory", NewName="PS_AI_ConvAgent_LipSyncPoseMapFactory")
+ClassRedirects=(OldName="PS_AI_Agent_EmotionPoseMapFactory", NewName="PS_AI_ConvAgent_EmotionPoseMapFactory")
; ── Structs ──
+StructRedirects=(OldName="ElevenLabsEmotionPoseSet", NewName="PS_AI_Agent_EmotionPoseSet")
+StructRedirects=(OldName="ElevenLabsNeckBoneEntry", NewName="PS_AI_Agent_NeckBoneEntry")
+StructRedirects=(OldName="ElevenLabsConversationInfo", NewName="PS_AI_Agent_ConversationInfo_ElevenLabs")
+StructRedirects=(OldName="ElevenLabsTranscriptSegment", NewName="PS_AI_Agent_TranscriptSegment_ElevenLabs")
+StructRedirects=(OldName="ElevenLabsClientToolCall", NewName="PS_AI_Agent_ClientToolCall_ElevenLabs")
+StructRedirects=(OldName="ElevenLabsEmotionPoseSet", NewName="PS_AI_ConvAgent_EmotionPoseSet")
+StructRedirects=(OldName="ElevenLabsNeckBoneEntry", NewName="PS_AI_ConvAgent_NeckBoneEntry")
+StructRedirects=(OldName="ElevenLabsConversationInfo", NewName="PS_AI_ConvAgent_ConversationInfo_ElevenLabs")
+StructRedirects=(OldName="ElevenLabsTranscriptSegment", NewName="PS_AI_ConvAgent_TranscriptSegment_ElevenLabs")
+StructRedirects=(OldName="ElevenLabsClientToolCall", NewName="PS_AI_ConvAgent_ClientToolCall_ElevenLabs")
+StructRedirects=(OldName="PS_AI_Agent_EmotionPoseSet", NewName="PS_AI_ConvAgent_EmotionPoseSet")
+StructRedirects=(OldName="PS_AI_Agent_NeckBoneEntry", NewName="PS_AI_ConvAgent_NeckBoneEntry")
+StructRedirects=(OldName="PS_AI_Agent_ConversationInfo_ElevenLabs", NewName="PS_AI_ConvAgent_ConversationInfo_ElevenLabs")
+StructRedirects=(OldName="PS_AI_Agent_TranscriptSegment_ElevenLabs", NewName="PS_AI_ConvAgent_TranscriptSegment_ElevenLabs")
+StructRedirects=(OldName="PS_AI_Agent_ClientToolCall_ElevenLabs", NewName="PS_AI_ConvAgent_ClientToolCall_ElevenLabs")
; ── Enums ──
+EnumRedirects=(OldName="EElevenLabsConnectionState", NewName="EPS_AI_Agent_ConnectionState_ElevenLabs")
+EnumRedirects=(OldName="EElevenLabsTurnMode", NewName="EPS_AI_Agent_TurnMode_ElevenLabs")
+EnumRedirects=(OldName="EElevenLabsEmotion", NewName="EPS_AI_Agent_Emotion")
+EnumRedirects=(OldName="EElevenLabsEmotionIntensity", NewName="EPS_AI_Agent_EmotionIntensity")
+EnumRedirects=(OldName="EElevenLabsConnectionState", NewName="EPS_AI_ConvAgent_ConnectionState_ElevenLabs")
+EnumRedirects=(OldName="EElevenLabsTurnMode", NewName="EPS_AI_ConvAgent_TurnMode_ElevenLabs")
+EnumRedirects=(OldName="EElevenLabsEmotion", NewName="EPS_AI_ConvAgent_Emotion")
+EnumRedirects=(OldName="EElevenLabsEmotionIntensity", NewName="EPS_AI_ConvAgent_EmotionIntensity")
+EnumRedirects=(OldName="EPS_AI_Agent_ConnectionState_ElevenLabs", NewName="EPS_AI_ConvAgent_ConnectionState_ElevenLabs")
+EnumRedirects=(OldName="EPS_AI_Agent_TurnMode_ElevenLabs", NewName="EPS_AI_ConvAgent_TurnMode_ElevenLabs")
+EnumRedirects=(OldName="EPS_AI_Agent_Emotion", NewName="EPS_AI_ConvAgent_Emotion")
+EnumRedirects=(OldName="EPS_AI_Agent_EmotionIntensity", NewName="EPS_AI_ConvAgent_EmotionIntensity")
[/Script/AndroidFileServerEditor.AndroidFileServerRuntimeSettings]
bEnablePlugin=True

32
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/AnimNode_PS_AI_Agent_FacialExpression.cpp → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/AnimNode_PS_AI_ConvAgent_FacialExpression.cpp
View File

@ -1,22 +1,22 @@
// Copyright ASTERION. All Rights Reserved.
#include "AnimNode_PS_AI_Agent_FacialExpression.h"
#include "PS_AI_Agent_FacialExpressionComponent.h"
#include "AnimNode_PS_AI_ConvAgent_FacialExpression.h"
#include "PS_AI_ConvAgent_FacialExpressionComponent.h"
#include "Components/SkeletalMeshComponent.h"
#include "Animation/AnimInstanceProxy.h"
#include "GameFramework/Actor.h"
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_Agent_FacialExprAnimNode, Log, All);
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_ConvAgent_FacialExprAnimNode, Log, All);
// ─────────────────────────────────────────────────────────────────────────────
// FAnimNode_Base interface
// ─────────────────────────────────────────────────────────────────────────────
void FAnimNode_PS_AI_Agent_FacialExpression::Initialize_AnyThread(const FAnimationInitializeContext& Context)
void FAnimNode_PS_AI_ConvAgent_FacialExpression::Initialize_AnyThread(const FAnimationInitializeContext& Context)
{
BasePose.Initialize(Context);
// Find the PS_AI_Agent_FacialExpressionComponent on the owning actor.
// Find the PS_AI_ConvAgent_FacialExpressionComponent on the owning actor.
// This runs during initialization (game thread) so actor access is safe.
FacialExpressionComponent.Reset();
CachedEmotionCurves.Reset();
@ -27,19 +27,19 @@ void FAnimNode_PS_AI_Agent_FacialExpression::Initialize_AnyThread(const FAnimati
{
if (AActor* Owner = SkelMesh->GetOwner())
{
UPS_AI_Agent_FacialExpressionComponent* Comp =
Owner->FindComponentByClass<UPS_AI_Agent_FacialExpressionComponent>();
UPS_AI_ConvAgent_FacialExpressionComponent* Comp =
Owner->FindComponentByClass<UPS_AI_ConvAgent_FacialExpressionComponent>();
if (Comp)
{
FacialExpressionComponent = Comp;
UE_LOG(LogPS_AI_Agent_FacialExprAnimNode, Log,
TEXT("PS AI Agent Facial Expression AnimNode bound to component on %s."),
UE_LOG(LogPS_AI_ConvAgent_FacialExprAnimNode, Log,
TEXT("PS AI ConvAgent Facial Expression AnimNode bound to component on %s."),
*Owner->GetName());
}
else
{
UE_LOG(LogPS_AI_Agent_FacialExprAnimNode, Warning,
TEXT("No PS_AI_Agent_FacialExpressionComponent found on %s. "
UE_LOG(LogPS_AI_ConvAgent_FacialExprAnimNode, Warning,
TEXT("No PS_AI_ConvAgent_FacialExpressionComponent found on %s. "
"Add the component alongside the Conversational Agent."),
*Owner->GetName());
}
@ -48,12 +48,12 @@ void FAnimNode_PS_AI_Agent_FacialExpression::Initialize_AnyThread(const FAnimati
}
}
void FAnimNode_PS_AI_Agent_FacialExpression::CacheBones_AnyThread(const FAnimationCacheBonesContext& Context)
void FAnimNode_PS_AI_ConvAgent_FacialExpression::CacheBones_AnyThread(const FAnimationCacheBonesContext& Context)
{
BasePose.CacheBones(Context);
}
void FAnimNode_PS_AI_Agent_FacialExpression::Update_AnyThread(const FAnimationUpdateContext& Context)
void FAnimNode_PS_AI_ConvAgent_FacialExpression::Update_AnyThread(const FAnimationUpdateContext& Context)
{
BasePose.Update(Context);
@ -68,7 +68,7 @@ void FAnimNode_PS_AI_Agent_FacialExpression::Update_AnyThread(const FAnimationUp
}
}
void FAnimNode_PS_AI_Agent_FacialExpression::Evaluate_AnyThread(FPoseContext& Output)
void FAnimNode_PS_AI_ConvAgent_FacialExpression::Evaluate_AnyThread(FPoseContext& Output)
{
// Evaluate the upstream pose (pass-through)
BasePose.Evaluate(Output);
@ -84,9 +84,9 @@ void FAnimNode_PS_AI_Agent_FacialExpression::Evaluate_AnyThread(FPoseContext& Ou
}
}
void FAnimNode_PS_AI_Agent_FacialExpression::GatherDebugData(FNodeDebugData& DebugData)
void FAnimNode_PS_AI_ConvAgent_FacialExpression::GatherDebugData(FNodeDebugData& DebugData)
{
FString DebugLine = FString::Printf(TEXT("PS AI Agent Facial Expression (%d curves)"), CachedEmotionCurves.Num());
FString DebugLine = FString::Printf(TEXT("PS AI ConvAgent Facial Expression (%d curves)"), CachedEmotionCurves.Num());
DebugData.AddDebugItem(DebugLine);
BasePose.GatherDebugData(DebugData);
}

32
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/AnimNode_PS_AI_Agent_LipSync.cpp → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/AnimNode_PS_AI_ConvAgent_LipSync.cpp
View File

@ -1,22 +1,22 @@
// Copyright ASTERION. All Rights Reserved.
#include "AnimNode_PS_AI_Agent_LipSync.h"
#include "PS_AI_Agent_LipSyncComponent.h"
#include "AnimNode_PS_AI_ConvAgent_LipSync.h"
#include "PS_AI_ConvAgent_LipSyncComponent.h"
#include "Components/SkeletalMeshComponent.h"
#include "Animation/AnimInstanceProxy.h"
#include "GameFramework/Actor.h"
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_Agent_LipSyncAnimNode, Log, All);
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_ConvAgent_LipSyncAnimNode, Log, All);
// ─────────────────────────────────────────────────────────────────────────────
// FAnimNode_Base interface
// ─────────────────────────────────────────────────────────────────────────────
void FAnimNode_PS_AI_Agent_LipSync::Initialize_AnyThread(const FAnimationInitializeContext& Context)
void FAnimNode_PS_AI_ConvAgent_LipSync::Initialize_AnyThread(const FAnimationInitializeContext& Context)
{
BasePose.Initialize(Context);
// Find the PS_AI_Agent_LipSyncComponent on the owning actor.
// Find the PS_AI_ConvAgent_LipSyncComponent on the owning actor.
// This runs during initialization (game thread) so actor access is safe.
LipSyncComponent.Reset();
CachedCurves.Reset();
@ -27,19 +27,19 @@ void FAnimNode_PS_AI_Agent_LipSync::Initialize_AnyThread(const FAnimationInitial
{
if (AActor* Owner = SkelMesh->GetOwner())
{
UPS_AI_Agent_LipSyncComponent* Comp =
Owner->FindComponentByClass<UPS_AI_Agent_LipSyncComponent>();
UPS_AI_ConvAgent_LipSyncComponent* Comp =
Owner->FindComponentByClass<UPS_AI_ConvAgent_LipSyncComponent>();
if (Comp)
{
LipSyncComponent = Comp;
UE_LOG(LogPS_AI_Agent_LipSyncAnimNode, Log,
TEXT("PS AI Agent Lip Sync AnimNode bound to component on %s."),
UE_LOG(LogPS_AI_ConvAgent_LipSyncAnimNode, Log,
TEXT("PS AI ConvAgent Lip Sync AnimNode bound to component on %s."),
*Owner->GetName());
}
else
{
UE_LOG(LogPS_AI_Agent_LipSyncAnimNode, Warning,
TEXT("No PS_AI_Agent_LipSyncComponent found on %s. "
UE_LOG(LogPS_AI_ConvAgent_LipSyncAnimNode, Warning,
TEXT("No PS_AI_ConvAgent_LipSyncComponent found on %s. "
"Add the component alongside the Conversational Agent."),
*Owner->GetName());
}
@ -48,12 +48,12 @@ void FAnimNode_PS_AI_Agent_LipSync::Initialize_AnyThread(const FAnimationInitial
}
}
void FAnimNode_PS_AI_Agent_LipSync::CacheBones_AnyThread(const FAnimationCacheBonesContext& Context)
void FAnimNode_PS_AI_ConvAgent_LipSync::CacheBones_AnyThread(const FAnimationCacheBonesContext& Context)
{
BasePose.CacheBones(Context);
}
void FAnimNode_PS_AI_Agent_LipSync::Update_AnyThread(const FAnimationUpdateContext& Context)
void FAnimNode_PS_AI_ConvAgent_LipSync::Update_AnyThread(const FAnimationUpdateContext& Context)
{
BasePose.Update(Context);
@ -69,7 +69,7 @@ void FAnimNode_PS_AI_Agent_LipSync::Update_AnyThread(const FAnimationUpdateConte
}
}
void FAnimNode_PS_AI_Agent_LipSync::Evaluate_AnyThread(FPoseContext& Output)
void FAnimNode_PS_AI_ConvAgent_LipSync::Evaluate_AnyThread(FPoseContext& Output)
{
// Evaluate the upstream pose (pass-through)
BasePose.Evaluate(Output);
@ -87,9 +87,9 @@ void FAnimNode_PS_AI_Agent_LipSync::Evaluate_AnyThread(FPoseContext& Output)
}
}
void FAnimNode_PS_AI_Agent_LipSync::GatherDebugData(FNodeDebugData& DebugData)
void FAnimNode_PS_AI_ConvAgent_LipSync::GatherDebugData(FNodeDebugData& DebugData)
{
FString DebugLine = FString::Printf(TEXT("PS AI Agent Lip Sync (%d curves)"), CachedCurves.Num());
FString DebugLine = FString::Printf(TEXT("PS AI ConvAgent Lip Sync (%d curves)"), CachedCurves.Num());
DebugData.AddDebugItem(DebugLine);
BasePose.GatherDebugData(DebugData);
}

66
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/AnimNode_PS_AI_Agent_Posture.cpp → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/AnimNode_PS_AI_ConvAgent_Posture.cpp
View File

@ -1,12 +1,12 @@
// Copyright ASTERION. All Rights Reserved.
#include "AnimNode_PS_AI_Agent_Posture.h"
#include "PS_AI_Agent_PostureComponent.h"
#include "AnimNode_PS_AI_ConvAgent_Posture.h"
#include "PS_AI_ConvAgent_PostureComponent.h"
#include "Components/SkeletalMeshComponent.h"
#include "Animation/AnimInstanceProxy.h"
#include "GameFramework/Actor.h"
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_Agent_PostureAnimNode, Log, All);
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_ConvAgent_PostureAnimNode, Log, All);
// ─────────────────────────────────────────────────────────────────────────────
// ARKit → MetaHuman CTRL eye curve mapping.
@ -77,7 +77,7 @@ static const TMap<FName, FName>& GetARKitToCTRLEyeMap()
// FAnimNode_Base interface
// ─────────────────────────────────────────────────────────────────────────────
void FAnimNode_PS_AI_Agent_Posture::Initialize_AnyThread(const FAnimationInitializeContext& Context)
void FAnimNode_PS_AI_ConvAgent_Posture::Initialize_AnyThread(const FAnimationInitializeContext& Context)
{
BasePose.Initialize(Context);
@ -103,20 +103,20 @@ void FAnimNode_PS_AI_Agent_Posture::Initialize_AnyThread(const FAnimationInitial
{
if (AActor* Owner = SkelMesh->GetOwner())
{
UPS_AI_Agent_PostureComponent* Comp =
Owner->FindComponentByClass<UPS_AI_Agent_PostureComponent>();
UPS_AI_ConvAgent_PostureComponent* Comp =
Owner->FindComponentByClass<UPS_AI_ConvAgent_PostureComponent>();
if (Comp)
{
PostureComponent = Comp;
HeadBoneName = Comp->GetHeadBoneName();
// Cache neck bone chain configuration
const TArray<FPS_AI_Agent_NeckBoneEntry>& Chain = Comp->GetNeckBoneChain();
const TArray<FPS_AI_ConvAgent_NeckBoneEntry>& Chain = Comp->GetNeckBoneChain();
if (Chain.Num() > 0)
{
ChainBoneNames.Reserve(Chain.Num());
ChainBoneWeights.Reserve(Chain.Num());
for (const FPS_AI_Agent_NeckBoneEntry& Entry : Chain)
for (const FPS_AI_ConvAgent_NeckBoneEntry& Entry : Chain)
{
if (!Entry.BoneName.IsNone() && Entry.Weight > 0.0f)
{
@ -126,8 +126,8 @@ void FAnimNode_PS_AI_Agent_Posture::Initialize_AnyThread(const FAnimationInitial
}
}
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Log,
TEXT("PS AI Agent Posture AnimNode: Owner=%s Mesh=%s Head=%s Eyes=%s Chain=%d HeadComp=%.1f EyeComp=%.1f DriftComp=%.1f"),
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Log,
TEXT("PS AI ConvAgent Posture AnimNode: Owner=%s Mesh=%s Head=%s Eyes=%s Chain=%d HeadComp=%.1f EyeComp=%.1f DriftComp=%.1f"),
*Owner->GetName(), *SkelMesh->GetName(),
bApplyHeadRotation ? TEXT("ON") : TEXT("OFF"),
bApplyEyeCurves ? TEXT("ON") : TEXT("OFF"),
@ -138,8 +138,8 @@ void FAnimNode_PS_AI_Agent_Posture::Initialize_AnyThread(const FAnimationInitial
}
else
{
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
TEXT("No PS_AI_Agent_PostureComponent found on %s."),
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT("No PS_AI_ConvAgent_PostureComponent found on %s."),
*Owner->GetName());
}
}
@ -147,7 +147,7 @@ void FAnimNode_PS_AI_Agent_Posture::Initialize_AnyThread(const FAnimationInitial
}
}
void FAnimNode_PS_AI_Agent_Posture::CacheBones_AnyThread(const FAnimationCacheBonesContext& Context)
void FAnimNode_PS_AI_ConvAgent_Posture::CacheBones_AnyThread(const FAnimationCacheBonesContext& Context)
{
BasePose.CacheBones(Context);
@ -187,7 +187,7 @@ void FAnimNode_PS_AI_Agent_Posture::CacheBones_AnyThread(const FAnimationCacheBo
: FQuat::Identity;
ChainRefPoseRotations.Add(RefRot);
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Log,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Log,
TEXT(" Chain bone [%d] '%s' → index %d (weight=%.2f)"),
i, *ChainBoneNames[i].ToString(), CompactIdx.GetInt(),
ChainBoneWeights[i]);
@ -196,7 +196,7 @@ void FAnimNode_PS_AI_Agent_Posture::CacheBones_AnyThread(const FAnimationCacheBo
{
ChainBoneIndices.Add(FCompactPoseBoneIndex(INDEX_NONE));
ChainRefPoseRotations.Add(FQuat::Identity);
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT(" Chain bone [%d] '%s' NOT FOUND in skeleton!"),
i, *ChainBoneNames[i].ToString());
}
@ -217,20 +217,20 @@ void FAnimNode_PS_AI_Agent_Posture::CacheBones_AnyThread(const FAnimationCacheBo
? RefPose[MeshIndex].GetRotation()
: FQuat::Identity;
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Log,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Log,
TEXT("Head bone '%s' resolved to index %d."),
*HeadBoneName.ToString(), HeadBoneIndex.GetInt());
}
else
{
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT("Head bone '%s' NOT FOUND in skeleton. Available bones:"),
*HeadBoneName.ToString());
const int32 NumBones = FMath::Min(RefSkeleton.GetNum(), 10);
for (int32 i = 0; i < NumBones; ++i)
{
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT(" [%d] %s"), i, *RefSkeleton.GetBoneName(i).ToString());
}
}
@ -258,13 +258,13 @@ void FAnimNode_PS_AI_Agent_Posture::CacheBones_AnyThread(const FAnimationCacheBo
? RefPose[MeshIdx].GetRotation()
: FQuat::Identity;
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Log,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Log,
TEXT("Eye bone '%s' resolved to index %d."),
*BoneName.ToString(), OutIndex.GetInt());
}
else
{
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Log,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Log,
TEXT("Eye bone '%s' not found in skeleton (OK for Body AnimBP)."),
*BoneName.ToString());
}
@ -311,7 +311,7 @@ void FAnimNode_PS_AI_Agent_Posture::CacheBones_AnyThread(const FAnimationCacheBo
RefAccumAboveChain = RefAccumAboveChain * RefRot;
}
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Log,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Log,
TEXT("Body drift: %d ancestor bones above chain. RefAccum=(%s)"),
AncestorBoneIndices.Num(),
*RefAccumAboveChain.Rotator().ToCompactString());
@ -320,7 +320,7 @@ void FAnimNode_PS_AI_Agent_Posture::CacheBones_AnyThread(const FAnimationCacheBo
}
}
void FAnimNode_PS_AI_Agent_Posture::Update_AnyThread(const FAnimationUpdateContext& Context)
void FAnimNode_PS_AI_ConvAgent_Posture::Update_AnyThread(const FAnimationUpdateContext& Context)
{
BasePose.Update(Context);
@ -383,7 +383,7 @@ static FQuat ComputeCompensatedBoneRot(
return (CompensatedContrib * RefPoseRot).GetNormalized();
}
void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
void FAnimNode_PS_AI_ConvAgent_Posture::Evaluate_AnyThread(FPoseContext& Output)
{
// Evaluate the upstream pose (pass-through)
BasePose.Evaluate(Output);
@ -398,7 +398,7 @@ void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
const bool bHasEyeBones = (LeftEyeBoneIndex.GetInt() != INDEX_NONE);
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT("[%s] Posture Evaluate: HeadComp=%.2f EyeComp=%.2f DriftComp=%.2f Valid=%s HeadRot=(%s) Eyes=%d Chain=%d Ancestors=%d"),
NodeRole,
CachedHeadCompensation,
@ -420,7 +420,7 @@ void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
const FQuat Delta = AnimRot * RefRot.Inverse();
const FRotator DeltaRot = Delta.Rotator();
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT(" Chain[0] '%s' AnimDelta from RefPose: Y=%.2f P=%.2f R=%.2f (this gets removed at Comp=1)"),
*ChainBoneNames[0].ToString(),
DeltaRot.Yaw, DeltaRot.Pitch, DeltaRot.Roll);
@ -454,7 +454,7 @@ void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
}
if (++EyeDiagLogCounter % 300 == 1)
{
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT("[EYE DIAG MODE 1] Forcing CTRL_expressions_eyeLookUpL=1.0 | Left eye should look UP if Control Rig reads CTRL curves"));
}
@ -473,7 +473,7 @@ void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
}
if (++EyeDiagLogCounter % 300 == 1)
{
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT("[EYE DIAG MODE 2] Forcing ARKit eyeLookUpLeft=1.0 | Left eye should look UP if mh_arkit_mapping_pose drives eyes"));
}
@ -494,7 +494,7 @@ void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
Output.Curve.Set(ZeroCTRL, 0.0f);
if (++EyeDiagLogCounter % 300 == 1)
{
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT("[EYE DIAG MODE 3] Forcing FACIAL_L_Eye bone -25° pitch | Left eye should look UP if bone rotation drives eyes"));
}
#endif
@ -576,7 +576,7 @@ void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
#if !UE_BUILD_SHIPPING
if (EvalDebugFrameCounter % 300 == 1 && CachedEyeCurves.Num() > 0)
{
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT(" Eyes: Comp=%.2f → %s (anim weight=%.0f%%, posture weight=%.0f%%)"),
Comp,
Comp > 0.001f ? TEXT("BLEND") : TEXT("PASSTHROUGH"),
@ -633,7 +633,7 @@ void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
if (++DiagLogCounter % 90 == 0)
{
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT("DIAG Phase %d: %s | Timer=%.1f"), Phase, PhaseName, DiagTimer);
}
}
@ -695,7 +695,7 @@ void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
if (EvalDebugFrameCounter % 300 == 1)
{
const FRotator CorrRot = FullCorrection.Rotator();
UE_LOG(LogPS_AI_Agent_PostureAnimNode, Warning,
UE_LOG(LogPS_AI_ConvAgent_PostureAnimNode, Warning,
TEXT(" DriftCorrection: Y=%.1f P=%.1f R=%.1f | Comp=%.2f"),
CorrRot.Yaw, CorrRot.Pitch, CorrRot.Roll,
CachedBodyDriftCompensation);
@ -784,11 +784,11 @@ void FAnimNode_PS_AI_Agent_Posture::Evaluate_AnyThread(FPoseContext& Output)
#endif
}
void FAnimNode_PS_AI_Agent_Posture::GatherDebugData(FNodeDebugData& DebugData)
void FAnimNode_PS_AI_ConvAgent_Posture::GatherDebugData(FNodeDebugData& DebugData)
{
const FRotator DebugRot = CachedHeadRotation.Rotator();
FString DebugLine = FString::Printf(
TEXT("PS AI Agent Posture (eyes: %d, head: Y=%.1f P=%.1f, chain: %d, headComp: %.1f, eyeComp: %.1f, driftComp: %.1f)"),
TEXT("PS AI ConvAgent Posture (eyes: %d, head: Y=%.1f P=%.1f, chain: %d, headComp: %.1f, eyeComp: %.1f, driftComp: %.1f)"),
CachedEyeCurves.Num(),
DebugRot.Yaw, DebugRot.Pitch,
ChainBoneIndices.Num(),

8
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent.cpp
View File

@ -11,15 +11,15 @@ IMPLEMENT_MODULE(FPS_AI_ConvAgentModule, PS_AI_ConvAgent)
void FPS_AI_ConvAgentModule::StartupModule()
{
Settings = NewObject<UPS_AI_Agent_Settings_ElevenLabs>(GetTransientPackage(), "PS_AI_Agent_Settings_ElevenLabs", RF_Standalone);
Settings = NewObject<UPS_AI_ConvAgent_Settings_ElevenLabs>(GetTransientPackage(), "PS_AI_ConvAgent_Settings_ElevenLabs", RF_Standalone);
Settings->AddToRoot();
if (ISettingsModule* SettingsModule = FModuleManager::GetModulePtr<ISettingsModule>("Settings"))
{
SettingsModule->RegisterSettings(
"Project", "Plugins", "PS_AI_ConvAgent_ElevenLabs",
LOCTEXT("SettingsName", "PS AI Agent - ElevenLabs"),
LOCTEXT("SettingsDescription", "Configure the PS AI Agent - ElevenLabs plugin"),
LOCTEXT("SettingsName", "PS AI ConvAgent - ElevenLabs"),
LOCTEXT("SettingsDescription", "Configure the PS AI ConvAgent - ElevenLabs plugin"),
Settings);
}
}
@ -41,7 +41,7 @@ void FPS_AI_ConvAgentModule::ShutdownModule()
}
}
UPS_AI_Agent_Settings_ElevenLabs* FPS_AI_ConvAgentModule::GetSettings() const
UPS_AI_ConvAgent_Settings_ElevenLabs* FPS_AI_ConvAgentModule::GetSettings() const
{
check(Settings);
return Settings;

206
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_Agent_Conv_ElevenLabsComponent.cpp → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_ElevenLabsComponent.cpp
View File

@ -1,19 +1,19 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_Agent_Conv_ElevenLabsComponent.h"
#include "PS_AI_Agent_MicrophoneCaptureComponent.h"
#include "PS_AI_ConvAgent_ElevenLabsComponent.h"
#include "PS_AI_ConvAgent_MicrophoneCaptureComponent.h"
#include "PS_AI_ConvAgent.h"
#include "Components/AudioComponent.h"
#include "Sound/SoundWaveProcedural.h"
#include "GameFramework/Actor.h"
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_Agent_Conv_ElevenLabs, Log, All);
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_ConvAgent_ElevenLabs, Log, All);
// ─────────────────────────────────────────────────────────────────────────────
// Constructor
// ─────────────────────────────────────────────────────────────────────────────
UPS_AI_Agent_Conv_ElevenLabsComponent::UPS_AI_Agent_Conv_ElevenLabsComponent()
UPS_AI_ConvAgent_ElevenLabsComponent::UPS_AI_ConvAgent_ElevenLabsComponent()
{
PrimaryComponentTick.bCanEverTick = true;
// Tick is used only to detect silence (agent stopped speaking).
@ -24,19 +24,19 @@ UPS_AI_Agent_Conv_ElevenLabsComponent::UPS_AI_Agent_Conv_ElevenLabsComponent()
// ─────────────────────────────────────────────────────────────────────────────
// Lifecycle
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_Conv_ElevenLabsComponent::BeginPlay()
void UPS_AI_ConvAgent_ElevenLabsComponent::BeginPlay()
{
Super::BeginPlay();
InitAudioPlayback();
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::EndPlay(const EEndPlayReason::Type EndPlayReason)
void UPS_AI_ConvAgent_ElevenLabsComponent::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
EndConversation();
Super::EndPlay(EndPlayReason);
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::TickComponent(float DeltaTime, ELevelTick TickType,
void UPS_AI_ConvAgent_ElevenLabsComponent::TickComponent(float DeltaTime, ELevelTick TickType,
FActorComponentTickFunction* ThisTickFunction)
{
Super::TickComponent(DeltaTime, TickType, ThisTickFunction);
@ -50,7 +50,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::TickComponent(float DeltaTime, ELeve
{
bWaitingForAgentResponse = false;
const double T = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Warning,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
TEXT("[T+%.2fs] [Turn %d] Response timeout — server did not start generating after %.1fs. Firing OnAgentResponseTimeout."),
T, LastClosedTurnIndex, WaitTime);
OnAgentResponseTimeout.Broadcast();
@ -69,7 +69,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::TickComponent(float DeltaTime, ELeve
bAgentGenerating = false;
GeneratingTickCount = 0;
const double T = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Warning,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
TEXT("[T+%.2fs] [Turn %d] Generating timeout (10s) — server generated but no audio arrived. Clearing bAgentGenerating."),
T, LastClosedTurnIndex);
}
@ -89,7 +89,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::TickComponent(float DeltaTime, ELeve
{
bPreBuffering = false;
const double Tpb = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Pre-buffer timeout (%dms). Starting playback."),
Tpb, LastClosedTurnIndex, AudioPreBufferMs);
if (AudioPlaybackComponent && !AudioPlaybackComponent->IsPlaying())
@ -146,7 +146,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::TickComponent(float DeltaTime, ELeve
if (bHardTimeoutFired)
{
const double Tht = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Warning,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
TEXT("[T+%.2fs] [Turn %d] Agent silence hard-timeout (10s) without agent_response — declaring agent stopped."),
Tht, LastClosedTurnIndex);
}
@ -157,31 +157,31 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::TickComponent(float DeltaTime, ELeve
// ─────────────────────────────────────────────────────────────────────────────
// Control
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_Conv_ElevenLabsComponent::StartConversation()
void UPS_AI_ConvAgent_ElevenLabsComponent::StartConversation()
{
if (!WebSocketProxy)
{
WebSocketProxy = NewObject<UPS_AI_Agent_WebSocket_ElevenLabsProxy>(this);
WebSocketProxy = NewObject<UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy>(this);
WebSocketProxy->OnConnected.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleConnected);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleConnected);
WebSocketProxy->OnDisconnected.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleDisconnected);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleDisconnected);
WebSocketProxy->OnError.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleError);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleError);
WebSocketProxy->OnAudioReceived.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAudioReceived);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleAudioReceived);
WebSocketProxy->OnTranscript.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleTranscript);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleTranscript);
WebSocketProxy->OnAgentResponse.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAgentResponse);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleAgentResponse);
WebSocketProxy->OnInterrupted.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleInterrupted);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleInterrupted);
WebSocketProxy->OnAgentResponseStarted.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAgentResponseStarted);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleAgentResponseStarted);
WebSocketProxy->OnAgentResponsePart.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAgentResponsePart);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleAgentResponsePart);
WebSocketProxy->OnClientToolCall.AddDynamic(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::HandleClientToolCall);
&UPS_AI_ConvAgent_ElevenLabsComponent::HandleClientToolCall);
}
// Pass configuration to the proxy before connecting.
@ -191,7 +191,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StartConversation()
WebSocketProxy->Connect(AgentID);
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::EndConversation()
void UPS_AI_ConvAgent_ElevenLabsComponent::EndConversation()
{
StopListening();
// ISSUE-4: StopListening() may set bWaitingForAgentResponse=true (normal turn end path).
@ -207,11 +207,11 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::EndConversation()
}
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::StartListening()
void UPS_AI_ConvAgent_ElevenLabsComponent::StartListening()
{
if (!IsConnected())
{
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Warning, TEXT("StartListening: not connected."));
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning, TEXT("StartListening: not connected."));
return;
}
@ -230,14 +230,14 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StartListening()
{
if (bAgentSpeaking && bAllowInterruption)
{
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log, TEXT("StartListening: interrupting agent (speaking) to allow user to speak."));
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log, TEXT("StartListening: interrupting agent (speaking) to allow user to speak."));
InterruptAgent();
// InterruptAgent → StopAgentAudio clears bAgentSpeaking / bAgentGenerating,
// so we fall through and open the microphone immediately.
}
else
{
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log, TEXT("StartListening ignored: agent is %s%s — will listen after agent finishes."),
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log, TEXT("StartListening ignored: agent is %s%s — will listen after agent finishes."),
bAgentGenerating ? TEXT("generating") : TEXT("speaking"),
(bAgentSpeaking && !bAllowInterruption) ? TEXT(" (interruption disabled)") : TEXT(""));
return;
@ -248,19 +248,19 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StartListening()
bIsListening = true;
TurnStartTime = FPlatformTime::Seconds();
if (TurnMode == EPS_AI_Agent_TurnMode_ElevenLabs::Client)
if (TurnMode == EPS_AI_ConvAgent_TurnMode_ElevenLabs::Client)
{
WebSocketProxy->SendUserTurnStart();
}
// Find the microphone component on our owner actor, or create one.
UPS_AI_Agent_MicrophoneCaptureComponent* Mic =
GetOwner()->FindComponentByClass<UPS_AI_Agent_MicrophoneCaptureComponent>();
UPS_AI_ConvAgent_MicrophoneCaptureComponent* Mic =
GetOwner()->FindComponentByClass<UPS_AI_ConvAgent_MicrophoneCaptureComponent>();
if (!Mic)
{
Mic = NewObject<UPS_AI_Agent_MicrophoneCaptureComponent>(GetOwner(),
TEXT("PS_AI_Agent_Microphone"));
Mic = NewObject<UPS_AI_ConvAgent_MicrophoneCaptureComponent>(GetOwner(),
TEXT("PS_AI_ConvAgent_Microphone"));
Mic->RegisterComponent();
}
@ -268,13 +268,13 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StartListening()
// up if StartListening is called more than once without a matching StopListening.
Mic->OnAudioCaptured.RemoveAll(this);
Mic->OnAudioCaptured.AddUObject(this,
&UPS_AI_Agent_Conv_ElevenLabsComponent::OnMicrophoneDataCaptured);
&UPS_AI_ConvAgent_ElevenLabsComponent::OnMicrophoneDataCaptured);
// Echo suppression: point the mic at our atomic bAgentSpeaking flag so it skips
// capture entirely (before resampling) while the agent is speaking.
// In Server VAD + interruption mode, disable echo suppression so the server
// receives the user's voice even during agent playback — the server's own VAD
// handles echo filtering and interruption detection.
if (TurnMode == EPS_AI_Agent_TurnMode_ElevenLabs::Server && bAllowInterruption)
if (TurnMode == EPS_AI_ConvAgent_TurnMode_ElevenLabs::Server && bAllowInterruption)
{
Mic->EchoSuppressFlag = nullptr;
}
@ -285,16 +285,16 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StartListening()
Mic->StartCapture();
const double T = TurnStartTime - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log, TEXT("[T+%.2fs] [Turn %d] Mic opened — user speaking."), T, TurnIndex);
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log, TEXT("[T+%.2fs] [Turn %d] Mic opened — user speaking."), T, TurnIndex);
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::StopListening()
void UPS_AI_ConvAgent_ElevenLabsComponent::StopListening()
{
if (!bIsListening) return;
bIsListening = false;
if (UPS_AI_Agent_MicrophoneCaptureComponent* Mic =
GetOwner() ? GetOwner()->FindComponentByClass<UPS_AI_Agent_MicrophoneCaptureComponent>() : nullptr)
if (UPS_AI_ConvAgent_MicrophoneCaptureComponent* Mic =
GetOwner() ? GetOwner()->FindComponentByClass<UPS_AI_ConvAgent_MicrophoneCaptureComponent>() : nullptr)
{
Mic->StopCapture();
Mic->OnAudioCaptured.RemoveAll(this);
@ -316,7 +316,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StopListening()
{
if (MicAccumulationBuffer.Num() > 0)
{
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("StopListening: discarding %d bytes of accumulated mic audio (collision — server is mid-generation)."),
MicAccumulationBuffer.Num());
}
@ -328,7 +328,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StopListening()
MicAccumulationBuffer.Reset();
}
if (WebSocketProxy && TurnMode == EPS_AI_Agent_TurnMode_ElevenLabs::Client)
if (WebSocketProxy && TurnMode == EPS_AI_ConvAgent_TurnMode_ElevenLabs::Client)
{
WebSocketProxy->SendUserTurnEnd();
}
@ -349,7 +349,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StopListening()
const double TurnDuration = TurnStartTime > 0.0 ? TurnEndTime - TurnStartTime : 0.0;
if (bWaitingForAgentResponse)
{
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Mic closed — user spoke %.2fs. Waiting for server response (timeout %.0fs)..."),
T, TurnIndex, TurnDuration, ResponseTimeoutSeconds);
}
@ -357,23 +357,23 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StopListening()
{
// Collision avoidance: StopListening was called from HandleAgentResponseStarted
// while server was already generating — no need to wait or time out.
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Mic closed (collision avoidance) — user spoke %.2fs. Server is already generating Turn %d response."),
T, TurnIndex, TurnDuration, LastClosedTurnIndex);
}
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::SendTextMessage(const FString& Text)
void UPS_AI_ConvAgent_ElevenLabsComponent::SendTextMessage(const FString& Text)
{
if (!IsConnected())
{
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Warning, TEXT("SendTextMessage: not connected. Call StartConversation() first."));
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning, TEXT("SendTextMessage: not connected. Call StartConversation() first."));
return;
}
WebSocketProxy->SendTextMessage(Text);
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::InterruptAgent()
void UPS_AI_ConvAgent_ElevenLabsComponent::InterruptAgent()
{
bWaitingForAgentResponse = false; // Interrupting — no response expected from previous turn.
if (WebSocketProxy) WebSocketProxy->SendInterrupt();
@ -383,26 +383,26 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::InterruptAgent()
// ─────────────────────────────────────────────────────────────────────────────
// State queries
// ─────────────────────────────────────────────────────────────────────────────
bool UPS_AI_Agent_Conv_ElevenLabsComponent::IsConnected() const
bool UPS_AI_ConvAgent_ElevenLabsComponent::IsConnected() const
{
return WebSocketProxy && WebSocketProxy->IsConnected();
}
const FPS_AI_Agent_ConversationInfo_ElevenLabs& UPS_AI_Agent_Conv_ElevenLabsComponent::GetConversationInfo() const
const FPS_AI_ConvAgent_ConversationInfo_ElevenLabs& UPS_AI_ConvAgent_ElevenLabsComponent::GetConversationInfo() const
{
static FPS_AI_Agent_ConversationInfo_ElevenLabs Empty;
static FPS_AI_ConvAgent_ConversationInfo_ElevenLabs Empty;
return WebSocketProxy ? WebSocketProxy->GetConversationInfo() : Empty;
}
// ─────────────────────────────────────────────────────────────────────────────
// WebSocket event handlers
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleConnected(const FPS_AI_Agent_ConversationInfo_ElevenLabs& Info)
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleConnected(const FPS_AI_ConvAgent_ConversationInfo_ElevenLabs& Info)
{
SessionStartTime = FPlatformTime::Seconds();
TurnIndex = 0;
LastClosedTurnIndex = 0;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log, TEXT("[T+0.00s] Agent connected. ConversationID=%s"), *Info.ConversationID);
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log, TEXT("[T+0.00s] Agent connected. ConversationID=%s"), *Info.ConversationID);
OnAgentConnected.Broadcast(Info);
// In Client turn mode (push-to-talk), the user controls listening manually via
@ -410,15 +410,15 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleConnected(const FPS_AI_Agent_C
// permanently and interfere with push-to-talk — the T-release StopListening()
// would close the mic that auto-start opened, leaving the user unable to speak.
// Only auto-start in Server VAD mode where the mic stays open the whole session.
if (bAutoStartListening && TurnMode == EPS_AI_Agent_TurnMode_ElevenLabs::Server)
if (bAutoStartListening && TurnMode == EPS_AI_ConvAgent_TurnMode_ElevenLabs::Server)
{
StartListening();
}
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleDisconnected(int32 StatusCode, const FString& Reason)
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleDisconnected(int32 StatusCode, const FString& Reason)
{
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log, TEXT("Agent disconnected. Code=%d Reason=%s"), StatusCode, *Reason);
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log, TEXT("Agent disconnected. Code=%d Reason=%s"), StatusCode, *Reason);
// ISSUE-13: stop audio playback and clear the queue if the WebSocket drops while the
// agent is speaking. Without this the audio component kept playing buffered PCM after
@ -432,8 +432,8 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleDisconnected(int32 StatusCode,
GeneratingTickCount = 0;
TurnIndex = 0;
LastClosedTurnIndex = 0;
CurrentEmotion = EPS_AI_Agent_Emotion::Neutral;
CurrentEmotionIntensity = EPS_AI_Agent_EmotionIntensity::Medium;
CurrentEmotion = EPS_AI_ConvAgent_Emotion::Neutral;
CurrentEmotionIntensity = EPS_AI_ConvAgent_EmotionIntensity::Medium;
{
FScopeLock Lock(&MicSendLock);
MicAccumulationBuffer.Reset();
@ -441,13 +441,13 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleDisconnected(int32 StatusCode,
OnAgentDisconnected.Broadcast(StatusCode, Reason);
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleError(const FString& ErrorMessage)
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleError(const FString& ErrorMessage)
{
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Error, TEXT("Agent error: %s"), *ErrorMessage);
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Error, TEXT("Agent error: %s"), *ErrorMessage);
OnAgentError.Broadcast(ErrorMessage);
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAudioReceived(const TArray<uint8>& PCMData)
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleAudioReceived(const TArray<uint8>& PCMData)
{
const double T = FPlatformTime::Seconds() - SessionStartTime;
const int32 NumSamples = PCMData.Num() / sizeof(int16);
@ -457,7 +457,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAudioReceived(const TArray<uin
FScopeLock Lock(&AudioQueueLock);
QueueBefore = AudioQueue.Num() / sizeof(int16);
}
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Audio chunk received: %d samples (%.0fms) | AudioQueue before: %d samples (%.0fms)"),
T, LastClosedTurnIndex, NumSamples, DurationMs,
QueueBefore, (static_cast<float>(QueueBefore) / 16000.0f) * 1000.0f);
@ -467,7 +467,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAudioReceived(const TArray<uin
OnAgentAudioData.Broadcast(PCMData);
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleTranscript(const FPS_AI_Agent_TranscriptSegment_ElevenLabs& Segment)
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleTranscript(const FPS_AI_ConvAgent_TranscriptSegment_ElevenLabs& Segment)
{
if (bEnableUserTranscript)
{
@ -475,7 +475,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleTranscript(const FPS_AI_Agent_
}
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAgentResponse(const FString& ResponseText)
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleAgentResponse(const FString& ResponseText)
{
// The server sends agent_response when the full text response is complete.
// This is our reliable signal that no more TTS audio chunks will follow.
@ -488,14 +488,14 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAgentResponse(const FString& R
}
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleInterrupted()
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleInterrupted()
{
bWaitingForAgentResponse = false; // Interrupted — no response expected from previous turn.
StopAgentAudio();
OnAgentInterrupted.Broadcast();
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAgentResponseStarted()
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleAgentResponseStarted()
{
// The server has started generating a response (first agent_chat_response_part).
// Set bAgentGenerating BEFORE StopListening so that any StartListening call
@ -512,29 +512,29 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAgentResponseStarted()
// In Server VAD + interruption mode, keep the mic open so the server can
// detect if the user speaks over the agent and send an interruption event.
// The server handles echo filtering and VAD — we just keep streaming audio.
if (TurnMode == EPS_AI_Agent_TurnMode_ElevenLabs::Server && bAllowInterruption)
if (TurnMode == EPS_AI_ConvAgent_TurnMode_ElevenLabs::Server && bAllowInterruption)
{
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Agent generating — mic stays open (Server VAD + interruption). (%.2fs after turn end)"),
T, LastClosedTurnIndex, LatencyFromTurnEnd);
}
else
{
// Collision: server generating while mic was open — stop mic without flushing.
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Collision — mic was open, stopping. (%.2fs after turn end)"),
T, LastClosedTurnIndex, LatencyFromTurnEnd);
StopListening();
}
}
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Agent generating. (%.2fs after turn end)"),
T, LastClosedTurnIndex, LatencyFromTurnEnd);
OnAgentStartedGenerating.Broadcast();
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAgentResponsePart(const FString& PartialText)
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleAgentResponsePart(const FString& PartialText)
{
if (bEnableAgentPartialResponse)
{
@ -542,55 +542,55 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleAgentResponsePart(const FStrin
}
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleClientToolCall(const FPS_AI_Agent_ClientToolCall_ElevenLabs& ToolCall)
void UPS_AI_ConvAgent_ElevenLabsComponent::HandleClientToolCall(const FPS_AI_ConvAgent_ClientToolCall_ElevenLabs& ToolCall)
{
// Built-in handler for the "set_emotion" tool: parse emotion + intensity, auto-respond, broadcast.
if (ToolCall.ToolName == TEXT("set_emotion"))
{
// Parse emotion
EPS_AI_Agent_Emotion NewEmotion = EPS_AI_Agent_Emotion::Neutral;
EPS_AI_ConvAgent_Emotion NewEmotion = EPS_AI_ConvAgent_Emotion::Neutral;
const FString* EmotionStr = ToolCall.Parameters.Find(TEXT("emotion"));
if (EmotionStr)
{
const FString Lower = EmotionStr->ToLower();
if (Lower == TEXT("joy") || Lower == TEXT("happy") || Lower == TEXT("happiness"))
NewEmotion = EPS_AI_Agent_Emotion::Joy;
NewEmotion = EPS_AI_ConvAgent_Emotion::Joy;
else if (Lower == TEXT("sadness") || Lower == TEXT("sad"))
NewEmotion = EPS_AI_Agent_Emotion::Sadness;
NewEmotion = EPS_AI_ConvAgent_Emotion::Sadness;
else if (Lower == TEXT("anger") || Lower == TEXT("angry"))
NewEmotion = EPS_AI_Agent_Emotion::Anger;
NewEmotion = EPS_AI_ConvAgent_Emotion::Anger;
else if (Lower == TEXT("surprise") || Lower == TEXT("surprised"))
NewEmotion = EPS_AI_Agent_Emotion::Surprise;
NewEmotion = EPS_AI_ConvAgent_Emotion::Surprise;
else if (Lower == TEXT("fear") || Lower == TEXT("afraid") || Lower == TEXT("scared"))
NewEmotion = EPS_AI_Agent_Emotion::Fear;
NewEmotion = EPS_AI_ConvAgent_Emotion::Fear;
else if (Lower == TEXT("disgust") || Lower == TEXT("disgusted"))
NewEmotion = EPS_AI_Agent_Emotion::Disgust;
NewEmotion = EPS_AI_ConvAgent_Emotion::Disgust;
else if (Lower == TEXT("neutral"))
NewEmotion = EPS_AI_Agent_Emotion::Neutral;
NewEmotion = EPS_AI_ConvAgent_Emotion::Neutral;
else
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Warning, TEXT("Unknown emotion '%s', defaulting to Neutral."), **EmotionStr);
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning, TEXT("Unknown emotion '%s', defaulting to Neutral."), **EmotionStr);
}
// Parse intensity (default: medium)
EPS_AI_Agent_EmotionIntensity NewIntensity = EPS_AI_Agent_EmotionIntensity::Medium;
EPS_AI_ConvAgent_EmotionIntensity NewIntensity = EPS_AI_ConvAgent_EmotionIntensity::Medium;
const FString* IntensityStr = ToolCall.Parameters.Find(TEXT("intensity"));
if (IntensityStr)
{
const FString Lower = IntensityStr->ToLower();
if (Lower == TEXT("low") || Lower == TEXT("subtle") || Lower == TEXT("light"))
NewIntensity = EPS_AI_Agent_EmotionIntensity::Low;
NewIntensity = EPS_AI_ConvAgent_EmotionIntensity::Low;
else if (Lower == TEXT("medium") || Lower == TEXT("moderate") || Lower == TEXT("normal"))
NewIntensity = EPS_AI_Agent_EmotionIntensity::Medium;
NewIntensity = EPS_AI_ConvAgent_EmotionIntensity::Medium;
else if (Lower == TEXT("high") || Lower == TEXT("strong") || Lower == TEXT("extreme") || Lower == TEXT("intense"))
NewIntensity = EPS_AI_Agent_EmotionIntensity::High;
NewIntensity = EPS_AI_ConvAgent_EmotionIntensity::High;
else
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Warning, TEXT("Unknown intensity '%s', defaulting to Medium."), **IntensityStr);
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning, TEXT("Unknown intensity '%s', defaulting to Medium."), **IntensityStr);
}
CurrentEmotion = NewEmotion;
CurrentEmotionIntensity = NewIntensity;
const double T = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log, TEXT("[T+%.2fs] Agent emotion changed to: %s (%s)"),
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log, TEXT("[T+%.2fs] Agent emotion changed to: %s (%s)"),
T, *UEnum::GetValueAsString(NewEmotion), *UEnum::GetValueAsString(NewIntensity));
OnAgentEmotionChanged.Broadcast(NewEmotion, NewIntensity);
@ -616,21 +616,21 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::HandleClientToolCall(const FPS_AI_Ag
// ─────────────────────────────────────────────────────────────────────────────
// Audio playback
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_Conv_ElevenLabsComponent::InitAudioPlayback()
void UPS_AI_ConvAgent_ElevenLabsComponent::InitAudioPlayback()
{
AActor* Owner = GetOwner();
if (!Owner) return;
// USoundWaveProcedural lets us push raw PCM data at runtime.
ProceduralSoundWave = NewObject<USoundWaveProcedural>(this);
ProceduralSoundWave->SetSampleRate(PS_AI_Agent_Audio_ElevenLabs::SampleRate);
ProceduralSoundWave->NumChannels = PS_AI_Agent_Audio_ElevenLabs::Channels;
ProceduralSoundWave->SetSampleRate(PS_AI_ConvAgent_Audio_ElevenLabs::SampleRate);
ProceduralSoundWave->NumChannels = PS_AI_ConvAgent_Audio_ElevenLabs::Channels;
ProceduralSoundWave->Duration = INDEFINITELY_LOOPING_DURATION;
ProceduralSoundWave->SoundGroup = SOUNDGROUP_Voice;
ProceduralSoundWave->bLooping = false;
// Create the audio component attached to the owner.
AudioPlaybackComponent = NewObject<UAudioComponent>(Owner, TEXT("PS_AI_Agent_Audio_ElevenLabsPlayback"));
AudioPlaybackComponent = NewObject<UAudioComponent>(Owner, TEXT("PS_AI_ConvAgent_Audio_ElevenLabsPlayback"));
AudioPlaybackComponent->RegisterComponent();
AudioPlaybackComponent->bAutoActivate = false;
AudioPlaybackComponent->SetSound(ProceduralSoundWave);
@ -638,10 +638,10 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::InitAudioPlayback()
// When the procedural sound wave needs more audio data, pull from our queue.
ProceduralSoundWave->OnSoundWaveProceduralUnderflow =
FOnSoundWaveProceduralUnderflow::CreateUObject(
this, &UPS_AI_Agent_Conv_ElevenLabsComponent::OnProceduralUnderflow);
this, &UPS_AI_ConvAgent_ElevenLabsComponent::OnProceduralUnderflow);
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::OnProceduralUnderflow(
void UPS_AI_ConvAgent_ElevenLabsComponent::OnProceduralUnderflow(
USoundWaveProcedural* InProceduralWave, const int32 SamplesRequired)
{
FScopeLock Lock(&AudioQueueLock);
@ -672,7 +672,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::OnProceduralUnderflow(
{
bQueueWasDry = false;
const double T = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] AudioQueue recovered — feeding real data again (%d bytes remaining)."),
T, LastClosedTurnIndex, AudioQueue.Num());
}
@ -684,7 +684,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::OnProceduralUnderflow(
{
bQueueWasDry = true;
const double T = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Warning,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
TEXT("[T+%.2fs] [Turn %d] AudioQueue DRY — waiting for next TTS chunk (requested %d samples)."),
T, LastClosedTurnIndex, SamplesRequired);
}
@ -702,7 +702,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::OnProceduralUnderflow(
}
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::EnqueueAgentAudio(const TArray<uint8>& PCMData)
void UPS_AI_ConvAgent_ElevenLabsComponent::EnqueueAgentAudio(const TArray<uint8>& PCMData)
{
{
FScopeLock Lock(&AudioQueueLock);
@ -721,7 +721,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::EnqueueAgentAudio(const TArray<uint8
const double T = AgentSpeakStart - SessionStartTime;
const double LatencyFromTurnEnd = TurnEndTime > 0.0 ? AgentSpeakStart - TurnEndTime : 0.0;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Agent speaking — first audio chunk. (%.2fs after turn end)"),
T, LastClosedTurnIndex, LatencyFromTurnEnd);
@ -735,7 +735,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::EnqueueAgentAudio(const TArray<uint8
bPreBuffering = true;
PreBufferStartTime = FPlatformTime::Seconds();
const double Tpb2 = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Pre-buffering %dms before starting playback."),
Tpb2, LastClosedTurnIndex, AudioPreBufferMs);
}
@ -752,7 +752,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::EnqueueAgentAudio(const TArray<uint8
const double NowPb = FPlatformTime::Seconds();
const double BufferedMs = (NowPb - PreBufferStartTime) * 1000.0;
const double Tpb3 = NowPb - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Pre-buffer: second chunk arrived (%.0fms buffered). Starting playback."),
Tpb3, LastClosedTurnIndex, BufferedMs);
if (AudioPlaybackComponent && !AudioPlaybackComponent->IsPlaying())
@ -768,7 +768,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::EnqueueAgentAudio(const TArray<uint8
if (AudioPlaybackComponent && !AudioPlaybackComponent->IsPlaying())
{
const double Tbr = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Warning,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
TEXT("[T+%.2fs] [Turn %d] Audio component stopped during speech (buffer underrun). Restarting playback."),
Tbr, LastClosedTurnIndex);
AudioPlaybackComponent->Play();
@ -778,7 +778,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::EnqueueAgentAudio(const TArray<uint8
}
}
void UPS_AI_Agent_Conv_ElevenLabsComponent::StopAgentAudio()
void UPS_AI_ConvAgent_ElevenLabsComponent::StopAgentAudio()
{
// Flush the ProceduralSoundWave's internal buffer BEFORE stopping.
// QueueAudio() pushes data into the wave's internal ring buffer during
@ -824,7 +824,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StopAgentAudio()
const double T = Now - SessionStartTime;
const double AgentSpokeDuration = AgentSpeakStart > 0.0 ? Now - AgentSpeakStart : 0.0;
const double TotalTurnDuration = TurnEndTime > 0.0 ? Now - TurnEndTime : 0.0;
UE_LOG(LogPS_AI_Agent_Conv_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
TEXT("[T+%.2fs] [Turn %d] Agent stopped speaking (spoke %.2fs, full turn round-trip %.2fs)."),
T, LastClosedTurnIndex, AgentSpokeDuration, TotalTurnDuration);
@ -835,7 +835,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::StopAgentAudio()
// ─────────────────────────────────────────────────────────────────────────────
// Microphone → WebSocket
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_Conv_ElevenLabsComponent::OnMicrophoneDataCaptured(const TArray<float>& FloatPCM)
void UPS_AI_ConvAgent_ElevenLabsComponent::OnMicrophoneDataCaptured(const TArray<float>& FloatPCM)
{
if (!IsConnected() || !bIsListening) return;
@ -844,7 +844,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::OnMicrophoneDataCaptured(const TArra
// which would confuse the server's VAD and STT.
// In Server VAD + interruption mode, keep sending audio so the server can
// detect the user speaking over the agent and trigger an interruption.
if (bAgentSpeaking && !(TurnMode == EPS_AI_Agent_TurnMode_ElevenLabs::Server && bAllowInterruption))
if (bAgentSpeaking && !(TurnMode == EPS_AI_ConvAgent_TurnMode_ElevenLabs::Server && bAllowInterruption))
{
return;
}
@ -867,7 +867,7 @@ void UPS_AI_Agent_Conv_ElevenLabsComponent::OnMicrophoneDataCaptured(const TArra
}
}
TArray<uint8> UPS_AI_Agent_Conv_ElevenLabsComponent::FloatPCMToInt16Bytes(const TArray<float>& FloatPCM)
TArray<uint8> UPS_AI_ConvAgent_ElevenLabsComponent::FloatPCMToInt16Bytes(const TArray<float>& FloatPCM)
{
TArray<uint8> Out;
Out.Reserve(FloatPCM.Num() * 2);

62
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_Agent_FacialExpressionComponent.cpp → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_FacialExpressionComponent.cpp
View File

@ -1,18 +1,18 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_Agent_FacialExpressionComponent.h"
#include "PS_AI_Agent_Conv_ElevenLabsComponent.h"
#include "PS_AI_Agent_EmotionPoseMap.h"
#include "PS_AI_ConvAgent_FacialExpressionComponent.h"
#include "PS_AI_ConvAgent_ElevenLabsComponent.h"
#include "PS_AI_ConvAgent_EmotionPoseMap.h"
#include "Animation/AnimSequence.h"
#include "Animation/AnimData/IAnimationDataModel.h"
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_Agent_FacialExpr, Log, All);
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_ConvAgent_FacialExpr, Log, All);
// ─────────────────────────────────────────────────────────────────────────────
// Construction
// ─────────────────────────────────────────────────────────────────────────────
UPS_AI_Agent_FacialExpressionComponent::UPS_AI_Agent_FacialExpressionComponent()
UPS_AI_ConvAgent_FacialExpressionComponent::UPS_AI_ConvAgent_FacialExpressionComponent()
{
PrimaryComponentTick.bCanEverTick = true;
PrimaryComponentTick.TickGroup = TG_PrePhysics;
@ -22,32 +22,32 @@ UPS_AI_Agent_FacialExpressionComponent::UPS_AI_Agent_FacialExpressionComponent()
// BeginPlay / EndPlay
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_FacialExpressionComponent::BeginPlay()
void UPS_AI_ConvAgent_FacialExpressionComponent::BeginPlay()
{
Super::BeginPlay();
AActor* Owner = GetOwner();
if (!Owner)
{
UE_LOG(LogPS_AI_Agent_FacialExpr, Warning, TEXT("No owner actor — facial expressions disabled."));
UE_LOG(LogPS_AI_ConvAgent_FacialExpr, Warning, TEXT("No owner actor — facial expressions disabled."));
return;
}
// Find and bind to agent component
auto* Agent = Owner->FindComponentByClass<UPS_AI_Agent_Conv_ElevenLabsComponent>();
auto* Agent = Owner->FindComponentByClass<UPS_AI_ConvAgent_ElevenLabsComponent>();
if (Agent)
{
AgentComponent = Agent;
Agent->OnAgentEmotionChanged.AddDynamic(
this, &UPS_AI_Agent_FacialExpressionComponent::OnEmotionChanged);
this, &UPS_AI_ConvAgent_FacialExpressionComponent::OnEmotionChanged);
UE_LOG(LogPS_AI_Agent_FacialExpr, Log,
UE_LOG(LogPS_AI_ConvAgent_FacialExpr, Log,
TEXT("Facial expression bound to agent component on %s."), *Owner->GetName());
}
else
{
UE_LOG(LogPS_AI_Agent_FacialExpr, Warning,
TEXT("No PS_AI_Agent_Conv_ElevenLabsComponent found on %s — "
UE_LOG(LogPS_AI_ConvAgent_FacialExpr, Warning,
TEXT("No PS_AI_ConvAgent_ElevenLabsComponent found on %s — "
"facial expression will not respond to emotion changes."),
*Owner->GetName());
}
@ -63,17 +63,17 @@ void UPS_AI_Agent_FacialExpressionComponent::BeginPlay()
{
ActivePlaybackTime = 0.0f;
CrossfadeAlpha = 1.0f; // No crossfade needed on startup
UE_LOG(LogPS_AI_Agent_FacialExpr, Log,
UE_LOG(LogPS_AI_ConvAgent_FacialExpr, Log,
TEXT("Auto-started default emotion anim: %s"), *ActiveAnim->GetName());
}
}
void UPS_AI_Agent_FacialExpressionComponent::EndPlay(const EEndPlayReason::Type EndPlayReason)
void UPS_AI_ConvAgent_FacialExpressionComponent::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
if (AgentComponent.IsValid())
{
AgentComponent->OnAgentEmotionChanged.RemoveDynamic(
this, &UPS_AI_Agent_FacialExpressionComponent::OnEmotionChanged);
this, &UPS_AI_ConvAgent_FacialExpressionComponent::OnEmotionChanged);
}
Super::EndPlay(EndPlayReason);
@ -83,11 +83,11 @@ void UPS_AI_Agent_FacialExpressionComponent::EndPlay(const EEndPlayReason::Type
// Validation
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_FacialExpressionComponent::ValidateEmotionPoses()
void UPS_AI_ConvAgent_FacialExpressionComponent::ValidateEmotionPoses()
{
if (!EmotionPoseMap || EmotionPoseMap->EmotionPoses.Num() == 0)
{
UE_LOG(LogPS_AI_Agent_FacialExpr, Log,
UE_LOG(LogPS_AI_ConvAgent_FacialExpr, Log,
TEXT("No emotion poses assigned in EmotionPoseMap — facial expressions disabled."));
return;
}
@ -95,13 +95,13 @@ void UPS_AI_Agent_FacialExpressionComponent::ValidateEmotionPoses()
int32 AnimCount = 0;
for (const auto& EmotionPair : EmotionPoseMap->EmotionPoses)
{
const FPS_AI_Agent_EmotionPoseSet& PoseSet = EmotionPair.Value;
const FPS_AI_ConvAgent_EmotionPoseSet& PoseSet = EmotionPair.Value;
if (PoseSet.Normal) ++AnimCount;
if (PoseSet.Medium) ++AnimCount;
if (PoseSet.Extreme) ++AnimCount;
}
UE_LOG(LogPS_AI_Agent_FacialExpr, Log,
UE_LOG(LogPS_AI_ConvAgent_FacialExpr, Log,
TEXT("=== Emotion poses: %d emotions, %d anim slots available ==="),
EmotionPoseMap->EmotionPoses.Num(), AnimCount);
}
@ -110,21 +110,21 @@ void UPS_AI_Agent_FacialExpressionComponent::ValidateEmotionPoses()
// Find AnimSequence for emotion + intensity (with fallback)
// ─────────────────────────────────────────────────────────────────────────────
UAnimSequence* UPS_AI_Agent_FacialExpressionComponent::FindAnimForEmotion(
EPS_AI_Agent_Emotion Emotion, EPS_AI_Agent_EmotionIntensity Intensity) const
UAnimSequence* UPS_AI_ConvAgent_FacialExpressionComponent::FindAnimForEmotion(
EPS_AI_ConvAgent_Emotion Emotion, EPS_AI_ConvAgent_EmotionIntensity Intensity) const
{
if (!EmotionPoseMap) return nullptr;
const FPS_AI_Agent_EmotionPoseSet* PoseSet = EmotionPoseMap->EmotionPoses.Find(Emotion);
const FPS_AI_ConvAgent_EmotionPoseSet* PoseSet = EmotionPoseMap->EmotionPoses.Find(Emotion);
if (!PoseSet) return nullptr;
// Direct match
UAnimSequence* Anim = nullptr;
switch (Intensity)
{
case EPS_AI_Agent_EmotionIntensity::Low: Anim = PoseSet->Normal; break;
case EPS_AI_Agent_EmotionIntensity::Medium: Anim = PoseSet->Medium; break;
case EPS_AI_Agent_EmotionIntensity::High: Anim = PoseSet->Extreme; break;
case EPS_AI_ConvAgent_EmotionIntensity::Low: Anim = PoseSet->Normal; break;
case EPS_AI_ConvAgent_EmotionIntensity::Medium: Anim = PoseSet->Medium; break;
case EPS_AI_ConvAgent_EmotionIntensity::High: Anim = PoseSet->Extreme; break;
}
if (Anim) return Anim;
@ -141,7 +141,7 @@ UAnimSequence* UPS_AI_Agent_FacialExpressionComponent::FindAnimForEmotion(
// Evaluate all FloatCurves from an AnimSequence at a given time
// ─────────────────────────────────────────────────────────────────────────────
TMap<FName, float> UPS_AI_Agent_FacialExpressionComponent::EvaluateAnimCurves(
TMap<FName, float> UPS_AI_ConvAgent_FacialExpressionComponent::EvaluateAnimCurves(
UAnimSequence* AnimSeq, float Time) const
{
TMap<FName, float> CurveValues;
@ -167,8 +167,8 @@ TMap<FName, float> UPS_AI_Agent_FacialExpressionComponent::EvaluateAnimCurves(
// Emotion change handler
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_FacialExpressionComponent::OnEmotionChanged(
EPS_AI_Agent_Emotion Emotion, EPS_AI_Agent_EmotionIntensity Intensity)
void UPS_AI_ConvAgent_FacialExpressionComponent::OnEmotionChanged(
EPS_AI_ConvAgent_Emotion Emotion, EPS_AI_ConvAgent_EmotionIntensity Intensity)
{
if (Emotion == ActiveEmotion && Intensity == ActiveEmotionIntensity)
return; // No change
@ -190,7 +190,7 @@ void UPS_AI_Agent_FacialExpressionComponent::OnEmotionChanged(
// Begin crossfade
CrossfadeAlpha = 0.0f;
UE_LOG(LogPS_AI_Agent_FacialExpr, Log,
UE_LOG(LogPS_AI_ConvAgent_FacialExpr, Log,
TEXT("Emotion changed: %s (%s) — anim: %s, crossfading over %.1fs..."),
*UEnum::GetValueAsString(Emotion), *UEnum::GetValueAsString(Intensity),
NewAnim ? *NewAnim->GetName() : TEXT("(none)"), EmotionBlendDuration);
@ -200,7 +200,7 @@ void UPS_AI_Agent_FacialExpressionComponent::OnEmotionChanged(
// Tick — play emotion animation and crossfade
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_FacialExpressionComponent::TickComponent(
void UPS_AI_ConvAgent_FacialExpressionComponent::TickComponent(
float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction)
{
Super::TickComponent(DeltaTime, TickType, ThisTickFunction);
@ -286,7 +286,7 @@ void UPS_AI_Agent_FacialExpressionComponent::TickComponent(
// Mouth curve classification
// ─────────────────────────────────────────────────────────────────────────────
bool UPS_AI_Agent_FacialExpressionComponent::IsMouthCurve(const FName& CurveName)
bool UPS_AI_ConvAgent_FacialExpressionComponent::IsMouthCurve(const FName& CurveName)
{
const FString Name = CurveName.ToString().ToLower();
return Name.Contains(TEXT("jaw"))

192
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_Agent_LipSyncComponent.cpp → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_LipSyncComponent.cpp
View File

@ -1,9 +1,9 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_Agent_LipSyncComponent.h"
#include "PS_AI_Agent_FacialExpressionComponent.h"
#include "PS_AI_Agent_LipSyncPoseMap.h"
#include "PS_AI_Agent_Conv_ElevenLabsComponent.h"
#include "PS_AI_ConvAgent_LipSyncComponent.h"
#include "PS_AI_ConvAgent_FacialExpressionComponent.h"
#include "PS_AI_ConvAgent_LipSyncPoseMap.h"
#include "PS_AI_ConvAgent_ElevenLabsComponent.h"
#include "Components/SkeletalMeshComponent.h"
#include "Engine/SkeletalMesh.h"
#include "Animation/MorphTarget.h"
@ -11,13 +11,13 @@
#include "Animation/AnimData/IAnimationDataModel.h"
#include "GameFramework/Actor.h"
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_Agent_LipSync, Log, All);
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_ConvAgent_LipSync, Log, All);
// ─────────────────────────────────────────────────────────────────────────────
// Static data
// ─────────────────────────────────────────────────────────────────────────────
const TArray<FName> UPS_AI_Agent_LipSyncComponent::VisemeNames = {
const TArray<FName> UPS_AI_ConvAgent_LipSyncComponent::VisemeNames = {
FName("sil"), FName("PP"), FName("FF"), FName("TH"), FName("DD"),
FName("kk"), FName("CH"), FName("SS"), FName("nn"), FName("RR"),
FName("aa"), FName("E"), FName("ih"), FName("oh"), FName("ou")
@ -43,7 +43,7 @@ static const TSet<FName> ExpressionCurveNames = {
// OVR Viseme → ARKit blendshape mapping.
// Each viseme activates a combination of ARKit morph targets with specific weights.
// These values are tuned for MetaHuman faces and can be adjusted per project.
TMap<FName, TMap<FName, float>> UPS_AI_Agent_LipSyncComponent::CreateVisemeToBlendshapeMap()
TMap<FName, TMap<FName, float>> UPS_AI_ConvAgent_LipSyncComponent::CreateVisemeToBlendshapeMap()
{
TMap<FName, TMap<FName, float>> Map;
@ -189,14 +189,14 @@ TMap<FName, TMap<FName, float>> UPS_AI_Agent_LipSyncComponent::CreateVisemeToBle
return Map;
}
const TMap<FName, TMap<FName, float>> UPS_AI_Agent_LipSyncComponent::VisemeToBlendshapeMap =
UPS_AI_Agent_LipSyncComponent::CreateVisemeToBlendshapeMap();
const TMap<FName, TMap<FName, float>> UPS_AI_ConvAgent_LipSyncComponent::VisemeToBlendshapeMap =
UPS_AI_ConvAgent_LipSyncComponent::CreateVisemeToBlendshapeMap();
// ─────────────────────────────────────────────────────────────────────────────
// Constructor / Destructor
// ─────────────────────────────────────────────────────────────────────────────
UPS_AI_Agent_LipSyncComponent::UPS_AI_Agent_LipSyncComponent()
UPS_AI_ConvAgent_LipSyncComponent::UPS_AI_ConvAgent_LipSyncComponent()
{
PrimaryComponentTick.bCanEverTick = true;
PrimaryComponentTick.TickInterval = 1.0f / 60.0f; // 60 fps for smooth animation
@ -211,13 +211,13 @@ UPS_AI_Agent_LipSyncComponent::UPS_AI_Agent_LipSyncComponent()
SmoothedVisemes.FindOrAdd(FName("sil")) = 1.0f;
}
UPS_AI_Agent_LipSyncComponent::~UPS_AI_Agent_LipSyncComponent() = default;
UPS_AI_ConvAgent_LipSyncComponent::~UPS_AI_ConvAgent_LipSyncComponent() = default;
// ─────────────────────────────────────────────────────────────────────────────
// Lifecycle
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::BeginPlay()
void UPS_AI_ConvAgent_LipSyncComponent::BeginPlay()
{
Super::BeginPlay();
@ -226,54 +226,54 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
Settings.FFTSize = Audio::FSpectrumAnalyzerSettings::EFFTSize::Medium_512;
Settings.WindowType = Audio::EWindowType::Hann;
SpectrumAnalyzer = MakeUnique<Audio::FSpectrumAnalyzer>(
Settings, static_cast<float>(PS_AI_Agent_Audio_ElevenLabs::SampleRate));
Settings, static_cast<float>(PS_AI_ConvAgent_Audio_ElevenLabs::SampleRate));
// Auto-discover the agent component on the same actor
AActor* Owner = GetOwner();
if (!Owner) return;
UPS_AI_Agent_Conv_ElevenLabsComponent* Agent =
Owner->FindComponentByClass<UPS_AI_Agent_Conv_ElevenLabsComponent>();
UPS_AI_ConvAgent_ElevenLabsComponent* Agent =
Owner->FindComponentByClass<UPS_AI_ConvAgent_ElevenLabsComponent>();
if (Agent)
{
AgentComponent = Agent;
AudioDataHandle = Agent->OnAgentAudioData.AddUObject(
this, &UPS_AI_Agent_LipSyncComponent::OnAudioChunkReceived);
this, &UPS_AI_ConvAgent_LipSyncComponent::OnAudioChunkReceived);
// Bind to text response delegates for text-driven lip sync.
// Partial text (streaming) provides text BEFORE audio arrives.
// Full text provides the complete sentence (arrives just after audio).
Agent->OnAgentPartialResponse.AddDynamic(
this, &UPS_AI_Agent_LipSyncComponent::OnPartialTextReceived);
this, &UPS_AI_ConvAgent_LipSyncComponent::OnPartialTextReceived);
Agent->OnAgentTextResponse.AddDynamic(
this, &UPS_AI_Agent_LipSyncComponent::OnTextResponseReceived);
this, &UPS_AI_ConvAgent_LipSyncComponent::OnTextResponseReceived);
// Bind to interruption/stop events so lip sync resets immediately
// when the agent is cut off or finishes speaking.
Agent->OnAgentInterrupted.AddDynamic(
this, &UPS_AI_Agent_LipSyncComponent::OnAgentInterrupted);
this, &UPS_AI_ConvAgent_LipSyncComponent::OnAgentInterrupted);
Agent->OnAgentStoppedSpeaking.AddDynamic(
this, &UPS_AI_Agent_LipSyncComponent::OnAgentStopped);
this, &UPS_AI_ConvAgent_LipSyncComponent::OnAgentStopped);
// Enable partial response streaming if not already enabled
Agent->bEnableAgentPartialResponse = true;
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Lip sync bound to agent component on %s (audio + text + interruption)."), *Owner->GetName());
}
else
{
UE_LOG(LogPS_AI_Agent_LipSync, Warning,
TEXT("No PS_AI_Agent_Conv_ElevenLabsComponent found on %s. Lip sync will not work."),
UE_LOG(LogPS_AI_ConvAgent_LipSync, Warning,
TEXT("No PS_AI_ConvAgent_ElevenLabsComponent found on %s. Lip sync will not work."),
*Owner->GetName());
}
// Cache the facial expression component for emotion-aware blending.
CachedFacialExprComp = Owner->FindComponentByClass<UPS_AI_Agent_FacialExpressionComponent>();
CachedFacialExprComp = Owner->FindComponentByClass<UPS_AI_ConvAgent_FacialExpressionComponent>();
if (CachedFacialExprComp.IsValid())
{
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Facial expression component found — emotion blending enabled (blend=%.2f)."),
EmotionExpressionBlend);
}
@ -286,7 +286,7 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
Owner->GetComponents<USkeletalMeshComponent>(SkeletalMeshes);
// Log all found skeletal mesh components for debugging
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT("Found %d SkeletalMeshComponent(s) on %s:"),
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT("Found %d SkeletalMeshComponent(s) on %s:"),
SkeletalMeshes.Num(), *Owner->GetName());
for (const USkeletalMeshComponent* Mesh : SkeletalMeshes)
{
@ -297,7 +297,7 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
{
MorphCount = Mesh->GetSkeletalMeshAsset()->GetMorphTargets().Num();
}
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT(" - '%s' (%d morph targets)"),
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT(" - '%s' (%d morph targets)"),
*Mesh->GetName(), MorphCount);
}
}
@ -308,7 +308,7 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
if (Mesh && Mesh->GetFName().ToString().Contains(TEXT("Face")))
{
TargetMesh = Mesh;
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT("Auto-detected face mesh by name: %s"), *Mesh->GetName());
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT("Auto-detected face mesh by name: %s"), *Mesh->GetName());
break;
}
}
@ -322,7 +322,7 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
&& Mesh->GetSkeletalMeshAsset()->GetMorphTargets().Num() > 0)
{
TargetMesh = Mesh;
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT("Auto-detected face mesh by morph targets: %s (%d morphs)"),
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT("Auto-detected face mesh by morph targets: %s (%d morphs)"),
*Mesh->GetName(), Mesh->GetSkeletalMeshAsset()->GetMorphTargets().Num());
break;
}
@ -337,7 +337,7 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
if (Mesh)
{
TargetMesh = Mesh;
UE_LOG(LogPS_AI_Agent_LipSync, Warning,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Warning,
TEXT("No mesh with morph targets found. Using fallback: %s (lip sync may not work)"),
*Mesh->GetName());
break;
@ -347,7 +347,7 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
if (!TargetMesh)
{
UE_LOG(LogPS_AI_Agent_LipSync, Warning,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Warning,
TEXT("No SkeletalMeshComponent found on %s. Set TargetMesh manually or use GetCurrentBlendshapes() in Blueprint."),
*Owner->GetName());
}
@ -357,7 +357,7 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
if (TargetMesh && TargetMesh->GetSkeletalMeshAsset())
{
const TArray<UMorphTarget*>& MorphTargets = TargetMesh->GetSkeletalMeshAsset()->GetMorphTargets();
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT("TargetMesh '%s' has %d morph targets."),
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT("TargetMesh '%s' has %d morph targets."),
*TargetMesh->GetName(), MorphTargets.Num());
// Log first 20 morph target names to verify ARKit naming
@ -374,7 +374,7 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
}
if (Count > 0)
{
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT("Morph target sample: %s"), *Names);
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT("Morph target sample: %s"), *Names);
}
// Verify our blendshape names exist as morph targets on this mesh
@ -390,7 +390,7 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
break;
}
}
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT(" Morph target '%s': %s"),
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT(" Morph target '%s': %s"),
*TestName.ToString(), bFound ? TEXT("FOUND") : TEXT("NOT FOUND"));
}
}
@ -403,12 +403,12 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
if (MorphCount == 0)
{
bUseCurveMode = true;
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("No morph targets found — switching to MetaHuman curve mode (CTRL_expressions_*)."));
}
else
{
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Found %d morph targets — using standard SetMorphTarget mode."), MorphCount);
}
}
@ -422,14 +422,14 @@ void UPS_AI_Agent_LipSyncComponent::BeginPlay()
// Phoneme pose curve extraction
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::ExtractPoseCurves(const FName& VisemeName, UAnimSequence* AnimSeq)
void UPS_AI_ConvAgent_LipSyncComponent::ExtractPoseCurves(const FName& VisemeName, UAnimSequence* AnimSeq)
{
if (!AnimSeq) return;
const IAnimationDataModel* DataModel = AnimSeq->GetDataModel();
if (!DataModel)
{
UE_LOG(LogPS_AI_Agent_LipSync, Warning,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Warning,
TEXT("Pose '%s' (%s): No DataModel available — skipping."),
*VisemeName.ToString(), *AnimSeq->GetName());
return;
@ -452,7 +452,7 @@ void UPS_AI_Agent_LipSyncComponent::ExtractPoseCurves(const FName& VisemeName, U
if (!bPosesUseCTRLNaming && PoseExtractedCurveMap.Num() == 0 && CurveValues.Num() == 1)
{
bPosesUseCTRLNaming = CurveName.ToString().StartsWith(TEXT("CTRL_"));
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Pose curve naming detected: %s (from curve '%s')"),
bPosesUseCTRLNaming ? TEXT("CTRL_expressions_*") : TEXT("ARKit / other"),
*CurveName.ToString());
@ -462,7 +462,7 @@ void UPS_AI_Agent_LipSyncComponent::ExtractPoseCurves(const FName& VisemeName, U
if (CurveValues.Num() > 0)
{
PoseExtractedCurveMap.Add(VisemeName, MoveTemp(CurveValues));
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Pose '%s' (%s): Extracted %d non-zero curves."),
*VisemeName.ToString(), *AnimSeq->GetName(),
PoseExtractedCurveMap[VisemeName].Num());
@ -471,25 +471,25 @@ void UPS_AI_Agent_LipSyncComponent::ExtractPoseCurves(const FName& VisemeName, U
{
// Still add an empty map so we know this viseme was assigned (silence pose)
PoseExtractedCurveMap.Add(VisemeName, TMap<FName, float>());
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Pose '%s' (%s): All curves are zero — neutral/silence pose."),
*VisemeName.ToString(), *AnimSeq->GetName());
}
}
void UPS_AI_Agent_LipSyncComponent::InitializePoseMappings()
void UPS_AI_ConvAgent_LipSyncComponent::InitializePoseMappings()
{
PoseExtractedCurveMap.Reset();
bPosesUseCTRLNaming = false;
if (!PoseMap)
{
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("No PoseMap assigned — using hardcoded ARKit mapping."));
return;
}
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Loading pose map: %s"), *PoseMap->GetName());
// Map each PoseMap property to its corresponding OVR viseme name
@ -524,18 +524,18 @@ void UPS_AI_Agent_LipSyncComponent::InitializePoseMappings()
if (AssignedCount > 0)
{
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("=== Pose-based lip sync: %d/%d poses assigned, %d curve mappings extracted ==="),
AssignedCount, Mappings.Num(), PoseExtractedCurveMap.Num());
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Curve naming: %s"),
bPosesUseCTRLNaming ? TEXT("CTRL_expressions_* (MetaHuman native)") : TEXT("ARKit / standard"));
if (bPosesUseCTRLNaming)
{
UE_LOG(LogPS_AI_Agent_LipSync, Warning,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Warning,
TEXT("IMPORTANT: Poses use CTRL_expressions_* curves. "
"Move the PS AI Agent Lip Sync AnimNode AFTER mh_arkit_mapping_pose "
"Move the PS AI ConvAgent Lip Sync AnimNode AFTER mh_arkit_mapping_pose "
"in the Face AnimBP for correct results."));
}
@ -553,7 +553,7 @@ void UPS_AI_Agent_LipSyncComponent::InitializePoseMappings()
*Pair.Key.ToString(), Pair.Value);
if (++Count >= 8) { CurveList += TEXT(" ..."); break; }
}
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT(" Sample '%s': [%s]"),
*PoseEntry.Key.ToString(), *CurveList);
break; // Only log one sample
@ -562,13 +562,13 @@ void UPS_AI_Agent_LipSyncComponent::InitializePoseMappings()
}
else
{
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("No phoneme pose AnimSequences assigned — using hardcoded ARKit mapping."));
}
}
void UPS_AI_Agent_LipSyncComponent::EndPlay(const EEndPlayReason::Type EndPlayReason)
void UPS_AI_ConvAgent_LipSyncComponent::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
// Unbind from agent component
if (AgentComponent.IsValid())
@ -579,13 +579,13 @@ void UPS_AI_Agent_LipSyncComponent::EndPlay(const EEndPlayReason::Type EndPlayRe
AudioDataHandle.Reset();
}
AgentComponent->OnAgentPartialResponse.RemoveDynamic(
this, &UPS_AI_Agent_LipSyncComponent::OnPartialTextReceived);
this, &UPS_AI_ConvAgent_LipSyncComponent::OnPartialTextReceived);
AgentComponent->OnAgentTextResponse.RemoveDynamic(
this, &UPS_AI_Agent_LipSyncComponent::OnTextResponseReceived);
this, &UPS_AI_ConvAgent_LipSyncComponent::OnTextResponseReceived);
AgentComponent->OnAgentInterrupted.RemoveDynamic(
this, &UPS_AI_Agent_LipSyncComponent::OnAgentInterrupted);
this, &UPS_AI_ConvAgent_LipSyncComponent::OnAgentInterrupted);
AgentComponent->OnAgentStoppedSpeaking.RemoveDynamic(
this, &UPS_AI_Agent_LipSyncComponent::OnAgentStopped);
this, &UPS_AI_ConvAgent_LipSyncComponent::OnAgentStopped);
}
AgentComponent.Reset();
SpectrumAnalyzer.Reset();
@ -597,7 +597,7 @@ void UPS_AI_Agent_LipSyncComponent::EndPlay(const EEndPlayReason::Type EndPlayRe
// Tick — smooth visemes and apply morph targets
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::TickComponent(float DeltaTime, ELevelTick TickType,
void UPS_AI_ConvAgent_LipSyncComponent::TickComponent(float DeltaTime, ELevelTick TickType,
FActorComponentTickFunction* ThisTickFunction)
{
Super::TickComponent(DeltaTime, TickType, ThisTickFunction);
@ -626,7 +626,7 @@ void UPS_AI_Agent_LipSyncComponent::TickComponent(float DeltaTime, ELevelTick Ti
// Timeout — start playback with spectral shapes as fallback
bWaitingForText = false;
PlaybackTimer = 0.0f;
UE_LOG(LogPS_AI_Agent_LipSync, Warning,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Warning,
TEXT("Text wait timeout (%.0fms). Starting lip sync with spectral shapes (Queue=%d)."),
WaitElapsed * 1000.0, VisemeQueue.Num());
}
@ -913,18 +913,18 @@ void UPS_AI_Agent_LipSyncComponent::TickComponent(float DeltaTime, ELevelTick Ti
// Interruption / stop handlers
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::OnAgentInterrupted()
void UPS_AI_ConvAgent_LipSyncComponent::OnAgentInterrupted()
{
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT("Agent interrupted — resetting lip sync to neutral."));
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT("Agent interrupted — resetting lip sync to neutral."));
ResetToNeutral();
}
void UPS_AI_Agent_LipSyncComponent::OnAgentStopped()
void UPS_AI_ConvAgent_LipSyncComponent::OnAgentStopped()
{
// Don't clear text state here — it's already handled by TickComponent's
// "queue runs dry" logic which checks bFullTextReceived.
// Just clear the queues so the mouth returns to neutral immediately.
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT("Agent stopped speaking — clearing lip sync queues."));
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT("Agent stopped speaking — clearing lip sync queues."));
VisemeQueue.Reset();
AmplitudeQueue.Reset();
PlaybackTimer = 0.0f;
@ -938,7 +938,7 @@ void UPS_AI_Agent_LipSyncComponent::OnAgentStopped()
TotalActiveFramesSeen = 0;
}
void UPS_AI_Agent_LipSyncComponent::ResetToNeutral()
void UPS_AI_ConvAgent_LipSyncComponent::ResetToNeutral()
{
// Clear all queued viseme and amplitude data
VisemeQueue.Reset();
@ -978,7 +978,7 @@ void UPS_AI_Agent_LipSyncComponent::ResetToNeutral()
// Audio analysis
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::OnAudioChunkReceived(const TArray<uint8>& PCMData)
void UPS_AI_ConvAgent_LipSyncComponent::OnAudioChunkReceived(const TArray<uint8>& PCMData)
{
if (!SpectrumAnalyzer) return;
@ -989,7 +989,7 @@ void UPS_AI_Agent_LipSyncComponent::OnAudioChunkReceived(const TArray<uint8>& PC
static bool bFirstChunkLogged = false;
if (!bFirstChunkLogged)
{
UE_LOG(LogPS_AI_Agent_LipSync, Verbose, TEXT("First audio chunk received: %d bytes (%d samples)"), PCMData.Num(), NumSamples);
UE_LOG(LogPS_AI_ConvAgent_LipSync, Verbose, TEXT("First audio chunk received: %d bytes (%d samples)"), PCMData.Num(), NumSamples);
bFirstChunkLogged = true;
}
@ -1275,7 +1275,7 @@ void UPS_AI_Agent_LipSyncComponent::OnAudioChunkReceived(const TArray<uint8>& PC
if (PseudoCount > 0)
{
UE_LOG(LogPS_AI_Agent_LipSync, Verbose,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Verbose,
TEXT("Pseudo-speech: %d active frames (%d syllables)"),
PseudoCount, (PseudoCount + SyllableFrames - 1) / SyllableFrames);
}
@ -1309,7 +1309,7 @@ void UPS_AI_Agent_LipSyncComponent::OnAudioChunkReceived(const TArray<uint8>& PC
if (FixedCount > 0)
{
UE_LOG(LogPS_AI_Agent_LipSync, Verbose,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Verbose,
TEXT("Late start fix: overrode %d leading silent frames with min amplitude %.2f"),
FixedCount, MinStartAmplitude);
}
@ -1325,7 +1325,7 @@ void UPS_AI_Agent_LipSyncComponent::OnAudioChunkReceived(const TArray<uint8>& PC
else
ApplyTextVisemesToQueue();
PlaybackTimer = 0.0f;
UE_LOG(LogPS_AI_Agent_LipSync, Verbose,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Verbose,
TEXT("Text already available (%d visemes). Starting lip sync immediately."),
TextVisemeSequence.Num());
}
@ -1335,7 +1335,7 @@ void UPS_AI_Agent_LipSyncComponent::OnAudioChunkReceived(const TArray<uint8>& PC
bWaitingForText = true;
WaitingForTextStartTime = FPlatformTime::Seconds();
PlaybackTimer = 0.0f;
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Waiting for text before starting lip sync (%d frames queued)."),
WindowsQueued);
}
@ -1349,7 +1349,7 @@ void UPS_AI_Agent_LipSyncComponent::OnAudioChunkReceived(const TArray<uint8>& PC
ApplyTextVisemesToQueue();
}
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Audio chunk: %d samples → %d windows | Amp=[%.2f-%.2f] | Queue=%d (%.1fs) | TextVisemes=%d"),
NumSamples, WindowsQueued,
MinAmp, MaxAmp, VisemeQueue.Num(),
@ -1360,7 +1360,7 @@ void UPS_AI_Agent_LipSyncComponent::OnAudioChunkReceived(const TArray<uint8>& PC
// Text-driven lip sync
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::OnPartialTextReceived(const FString& PartialText)
void UPS_AI_ConvAgent_LipSyncComponent::OnPartialTextReceived(const FString& PartialText)
{
// If the previous utterance's full text was already received,
// this partial text belongs to a NEW utterance — start fresh.
@ -1378,7 +1378,7 @@ void UPS_AI_Agent_LipSyncComponent::OnPartialTextReceived(const FString& Partial
// Convert accumulated text to viseme sequence progressively
ConvertTextToVisemes(AccumulatedText);
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Partial text: \"%s\" → %d visemes (accumulated: \"%s\")"),
*PartialText, TextVisemeSequence.Num(), *AccumulatedText);
@ -1397,20 +1397,20 @@ void UPS_AI_Agent_LipSyncComponent::OnPartialTextReceived(const FString& Partial
bWaitingForText = false;
PlaybackTimer = 0.0f; // Start consuming now
const double WaitElapsed = FPlatformTime::Seconds() - WaitingForTextStartTime;
UE_LOG(LogPS_AI_Agent_LipSync, Verbose,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Verbose,
TEXT("Text arrived after %.0fms wait. Starting lip sync playback."),
WaitElapsed * 1000.0);
}
}
void UPS_AI_Agent_LipSyncComponent::OnTextResponseReceived(const FString& ResponseText)
void UPS_AI_ConvAgent_LipSyncComponent::OnTextResponseReceived(const FString& ResponseText)
{
// Full text arrived — use it as the definitive source
bFullTextReceived = true;
AccumulatedText = ResponseText;
ConvertTextToVisemes(ResponseText);
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Full text: \"%s\" → %d visemes"), *ResponseText, TextVisemeSequence.Num());
// Apply to any remaining queued frames (or extend timeline in pose mode)
@ -1428,7 +1428,7 @@ void UPS_AI_Agent_LipSyncComponent::OnTextResponseReceived(const FString& Respon
bWaitingForText = false;
PlaybackTimer = 0.0f;
const double WaitElapsed = FPlatformTime::Seconds() - WaitingForTextStartTime;
UE_LOG(LogPS_AI_Agent_LipSync, Verbose,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Verbose,
TEXT("Full text arrived after %.0fms wait. Starting lip sync playback."),
WaitElapsed * 1000.0);
}
@ -1443,7 +1443,7 @@ void UPS_AI_Agent_LipSyncComponent::OnTextResponseReceived(const FString& Respon
VisSeq += V.ToString();
if (++Count >= 50) { VisSeq += TEXT(" ..."); break; }
}
UE_LOG(LogPS_AI_Agent_LipSync, Log, TEXT("Viseme sequence (%d): [%s]"),
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log, TEXT("Viseme sequence (%d): [%s]"),
TextVisemeSequence.Num(), *VisSeq);
}
@ -1453,7 +1453,7 @@ void UPS_AI_Agent_LipSyncComponent::OnTextResponseReceived(const FString& Respon
// persists and corrupts the next utterance's partial text accumulation.
}
void UPS_AI_Agent_LipSyncComponent::ConvertTextToVisemes(const FString& Text)
void UPS_AI_ConvAgent_LipSyncComponent::ConvertTextToVisemes(const FString& Text)
{
TextVisemeSequence.Reset();
@ -1835,7 +1835,7 @@ void UPS_AI_Agent_LipSyncComponent::ConvertTextToVisemes(const FString& Text)
}
}
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Syllable filter: %d raw → %d visible (removed %d invisible consonants/sil)"),
RawCount, Filtered.Num(), RawCount - Filtered.Num());
@ -1866,7 +1866,7 @@ static float GetVisemeDurationWeight(const FName& Viseme)
return 1.0f;
}
void UPS_AI_Agent_LipSyncComponent::ApplyTextVisemesToQueue()
void UPS_AI_ConvAgent_LipSyncComponent::ApplyTextVisemesToQueue()
{
if (TextVisemeSequence.Num() == 0 || VisemeQueue.Num() == 0) return;
@ -1980,7 +1980,7 @@ void UPS_AI_Agent_LipSyncComponent::ApplyTextVisemesToQueue()
const float FinalRatio = static_cast<float>(ActiveFrames) / static_cast<float>(SeqNum);
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Applied %d syllable-visemes to %d active frames (of %d total) — %.1f frames/viseme (%.0fms/viseme)"),
SeqNum, ActiveFrames, VisemeQueue.Num(),
FinalRatio, FinalRatio * 32.0f);
@ -1990,7 +1990,7 @@ void UPS_AI_Agent_LipSyncComponent::ApplyTextVisemesToQueue()
// Decoupled viseme timeline (pose mode)
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::BuildVisemeTimeline()
void UPS_AI_ConvAgent_LipSyncComponent::BuildVisemeTimeline()
{
if (TextVisemeSequence.Num() == 0) return;
@ -2063,14 +2063,14 @@ void UPS_AI_Agent_LipSyncComponent::BuildVisemeTimeline()
bVisemeTimelineActive = true;
bTextVisemesApplied = true;
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("Built viseme timeline: %d entries (from %d, max %d), audio=%.0fms, scale=%.2f → %.0fms/viseme avg"),
FinalSequence.Num(), TextVisemeSequence.Num(), MaxVisemes,
AudioDurationSec * 1000.0f, Scale,
(FinalSequence.Num() > 0) ? (CursorSec * 1000.0f / FinalSequence.Num()) : 0.0f);
}
void UPS_AI_Agent_LipSyncComponent::AnalyzeSpectrum()
void UPS_AI_ConvAgent_LipSyncComponent::AnalyzeSpectrum()
{
if (!SpectrumAnalyzer) return;
@ -2086,14 +2086,14 @@ void UPS_AI_Agent_LipSyncComponent::AnalyzeSpectrum()
const float TotalEnergy = VoiceEnergy + F1Energy + F2Energy + F3Energy + SibilantEnergy;
UE_LOG(LogPS_AI_Agent_LipSync, Verbose,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Verbose,
TEXT("Spectrum: Total=%.4f F1=%.4f F2=%.4f F3=%.4f Sibilant=%.4f"),
TotalEnergy, F1Energy, F2Energy, F3Energy, SibilantEnergy);
EstimateVisemes(TotalEnergy, F1Energy, F2Energy, F3Energy, SibilantEnergy);
}
float UPS_AI_Agent_LipSyncComponent::GetBandEnergy(float LowFreq, float HighFreq, int32 NumSamples) const
float UPS_AI_ConvAgent_LipSyncComponent::GetBandEnergy(float LowFreq, float HighFreq, int32 NumSamples) const
{
if (!SpectrumAnalyzer || NumSamples <= 0) return 0.0f;
@ -2113,7 +2113,7 @@ float UPS_AI_Agent_LipSyncComponent::GetBandEnergy(float LowFreq, float HighFreq
// Viseme estimation from spectral analysis
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::EstimateVisemes(float TotalEnergy,
void UPS_AI_ConvAgent_LipSyncComponent::EstimateVisemes(float TotalEnergy,
float F1Energy, float F2Energy, float F3Energy, float SibilantEnergy)
{
// Reset all visemes to zero
@ -2226,7 +2226,7 @@ void UPS_AI_Agent_LipSyncComponent::EstimateVisemes(float TotalEnergy,
// Viseme → ARKit blendshape mapping
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::MapVisemesToBlendshapes()
void UPS_AI_ConvAgent_LipSyncComponent::MapVisemesToBlendshapes()
{
CurrentBlendshapes.Reset();
@ -2261,7 +2261,7 @@ void UPS_AI_Agent_LipSyncComponent::MapVisemesToBlendshapes()
if (!WarnedVisemes.Contains(VisemeName))
{
WarnedVisemes.Add(VisemeName);
UE_LOG(LogPS_AI_Agent_LipSync, Warning,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Warning,
TEXT("No pose assigned for viseme '%s' — falling back to hardcoded ARKit mapping. "
"Mixing pose and hardcoded visemes may produce inconsistent results."),
*VisemeName.ToString());
@ -2325,7 +2325,7 @@ void UPS_AI_Agent_LipSyncComponent::MapVisemesToBlendshapes()
// Morph target application
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_LipSyncComponent::ApplyMorphTargets()
void UPS_AI_ConvAgent_LipSyncComponent::ApplyMorphTargets()
{
if (!TargetMesh) return;
@ -2350,14 +2350,14 @@ void UPS_AI_Agent_LipSyncComponent::ApplyMorphTargets()
}
if (DebugStr.Len() > 0)
{
UE_LOG(LogPS_AI_Agent_LipSync, Verbose, TEXT("%s: %s"),
UE_LOG(LogPS_AI_ConvAgent_LipSync, Verbose, TEXT("%s: %s"),
bUseCurveMode ? TEXT("Curves") : TEXT("Blendshapes"), *DebugStr);
}
}
if (bUseCurveMode)
{
// MetaHuman mode: curves are injected by the PS AI Agent Lip Sync AnimNode
// MetaHuman mode: curves are injected by the PS AI ConvAgent Lip Sync AnimNode
// placed in the Face AnimBP (AnimGraph evaluation context).
// OverrideCurveValue() does NOT work because the AnimGraph resets curves
// before the Control Rig reads them. The custom AnimNode injects curves
@ -2365,9 +2365,9 @@ void UPS_AI_Agent_LipSyncComponent::ApplyMorphTargets()
static bool bLoggedOnce = false;
if (!bLoggedOnce)
{
UE_LOG(LogPS_AI_Agent_LipSync, Log,
UE_LOG(LogPS_AI_ConvAgent_LipSync, Log,
TEXT("MetaHuman curve mode: lip sync curves will be injected by the "
"PS AI Agent Lip Sync AnimNode in the Face AnimBP. "
"PS AI ConvAgent Lip Sync AnimNode in the Face AnimBP. "
"Ensure the node is placed before the Control Rig in the AnimGraph."));
bLoggedOnce = true;
}
@ -2387,7 +2387,7 @@ void UPS_AI_Agent_LipSyncComponent::ApplyMorphTargets()
// ARKit → MetaHuman curve name conversion
// ─────────────────────────────────────────────────────────────────────────────
FName UPS_AI_Agent_LipSyncComponent::ARKitToMetaHumanCurveName(const FName& ARKitName) const
FName UPS_AI_ConvAgent_LipSyncComponent::ARKitToMetaHumanCurveName(const FName& ARKitName) const
{
// Check cache first to avoid per-frame string allocations
if (const FName* Cached = CurveNameCache.Find(ARKitName))
@ -2411,10 +2411,10 @@ FName UPS_AI_Agent_LipSyncComponent::ARKitToMetaHumanCurveName(const FName& ARKi
FName Result = FName(*(TEXT("CTRL_expressions_") + Name));
// Cache for next frame
const_cast<UPS_AI_Agent_LipSyncComponent*>(this)->CurveNameCache.Add(ARKitName, Result);
const_cast<UPS_AI_ConvAgent_LipSyncComponent*>(this)->CurveNameCache.Add(ARKitName, Result);
// Log once per curve name for debugging
UE_LOG(LogPS_AI_Agent_LipSync, Verbose, TEXT("Curve mapping: %s → %s"), *ARKitName.ToString(), *Result.ToString());
UE_LOG(LogPS_AI_ConvAgent_LipSync, Verbose, TEXT("Curve mapping: %s → %s"), *ARKitName.ToString(), *Result.ToString());
return Result;
}

32
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_Agent_MicrophoneCaptureComponent.cpp → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_MicrophoneCaptureComponent.cpp
View File

@ -1,17 +1,17 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_Agent_MicrophoneCaptureComponent.h"
#include "PS_AI_Agent_Definitions.h"
#include "PS_AI_ConvAgent_MicrophoneCaptureComponent.h"
#include "PS_AI_ConvAgent_Definitions.h"
#include "AudioCaptureCore.h"
#include "Async/Async.h"
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_Agent_Mic, Log, All);
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_ConvAgent_Mic, Log, All);
// ─────────────────────────────────────────────────────────────────────────────
// Constructor
// ─────────────────────────────────────────────────────────────────────────────
UPS_AI_Agent_MicrophoneCaptureComponent::UPS_AI_Agent_MicrophoneCaptureComponent()
UPS_AI_ConvAgent_MicrophoneCaptureComponent::UPS_AI_ConvAgent_MicrophoneCaptureComponent()
{
PrimaryComponentTick.bCanEverTick = false;
}
@ -19,7 +19,7 @@ UPS_AI_Agent_MicrophoneCaptureComponent::UPS_AI_Agent_MicrophoneCaptureComponent
// ─────────────────────────────────────────────────────────────────────────────
// Lifecycle
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_MicrophoneCaptureComponent::EndPlay(const EEndPlayReason::Type EndPlayReason)
void UPS_AI_ConvAgent_MicrophoneCaptureComponent::EndPlay(const EEndPlayReason::Type EndPlayReason)
{
StopCapture();
Super::EndPlay(EndPlayReason);
@ -28,11 +28,11 @@ void UPS_AI_Agent_MicrophoneCaptureComponent::EndPlay(const EEndPlayReason::Type
// ─────────────────────────────────────────────────────────────────────────────
// Capture control
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_MicrophoneCaptureComponent::StartCapture()
void UPS_AI_ConvAgent_MicrophoneCaptureComponent::StartCapture()
{
if (bCapturing)
{
UE_LOG(LogPS_AI_Agent_Mic, Warning, TEXT("StartCapture called while already capturing."));
UE_LOG(LogPS_AI_ConvAgent_Mic, Warning, TEXT("StartCapture called while already capturing."));
return;
}
@ -47,7 +47,7 @@ void UPS_AI_Agent_MicrophoneCaptureComponent::StartCapture()
if (!AudioCapture.OpenAudioCaptureStream(DeviceParams, MoveTemp(CaptureCallback), 1024))
{
UE_LOG(LogPS_AI_Agent_Mic, Error, TEXT("Failed to open default audio capture stream."));
UE_LOG(LogPS_AI_ConvAgent_Mic, Error, TEXT("Failed to open default audio capture stream."));
return;
}
@ -57,36 +57,36 @@ void UPS_AI_Agent_MicrophoneCaptureComponent::StartCapture()
{
DeviceSampleRate = DeviceInfo.PreferredSampleRate;
DeviceChannels = DeviceInfo.InputChannels;
UE_LOG(LogPS_AI_Agent_Mic, Log, TEXT("Capture device: %s | Rate=%d | Channels=%d"),
UE_LOG(LogPS_AI_ConvAgent_Mic, Log, TEXT("Capture device: %s | Rate=%d | Channels=%d"),
*DeviceInfo.DeviceName, DeviceSampleRate, DeviceChannels);
}
AudioCapture.StartStream();
bCapturing = true;
UE_LOG(LogPS_AI_Agent_Mic, Log, TEXT("Audio capture started."));
UE_LOG(LogPS_AI_ConvAgent_Mic, Log, TEXT("Audio capture started."));
}
void UPS_AI_Agent_MicrophoneCaptureComponent::StopCapture()
void UPS_AI_ConvAgent_MicrophoneCaptureComponent::StopCapture()
{
if (!bCapturing) return;
AudioCapture.StopStream();
AudioCapture.CloseStream();
bCapturing = false;
UE_LOG(LogPS_AI_Agent_Mic, Log, TEXT("Audio capture stopped."));
UE_LOG(LogPS_AI_ConvAgent_Mic, Log, TEXT("Audio capture stopped."));
}
// ─────────────────────────────────────────────────────────────────────────────
// Audio callback (background thread)
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_MicrophoneCaptureComponent::OnAudioGenerate(
void UPS_AI_ConvAgent_MicrophoneCaptureComponent::OnAudioGenerate(
const void* InAudio, int32 NumFrames,
int32 InNumChannels, int32 InSampleRate,
double StreamTime, bool bOverflow)
{
if (bOverflow)
{
UE_LOG(LogPS_AI_Agent_Mic, Verbose, TEXT("Audio capture buffer overflow."));
UE_LOG(LogPS_AI_ConvAgent_Mic, Verbose, TEXT("Audio capture buffer overflow."));
}
// Echo suppression: skip resampling + broadcasting entirely when agent is speaking.
@ -129,11 +129,11 @@ void UPS_AI_Agent_MicrophoneCaptureComponent::OnAudioGenerate(
// ─────────────────────────────────────────────────────────────────────────────
// Resampling
// ─────────────────────────────────────────────────────────────────────────────
TArray<float> UPS_AI_Agent_MicrophoneCaptureComponent::ResampleTo16000(
TArray<float> UPS_AI_ConvAgent_MicrophoneCaptureComponent::ResampleTo16000(
const float* InAudio, int32 NumFrames,
int32 InChannels, int32 InSampleRate)
{
const int32 TargetRate = PS_AI_Agent_Audio_ElevenLabs::SampleRate; // 16000
const int32 TargetRate = PS_AI_ConvAgent_Audio_ElevenLabs::SampleRate; // 16000
// --- Step 1: Downmix to mono ---
// NOTE: NumFrames is the number of audio frames (not total samples).

22
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_Agent_PostureComponent.cpp → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_PostureComponent.cpp
View File

@ -1,12 +1,12 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_Agent_PostureComponent.h"
#include "PS_AI_ConvAgent_PostureComponent.h"
#include "Components/SkeletalMeshComponent.h"
#include "GameFramework/Actor.h"
#include "Math/UnrealMathUtility.h"
#include "DrawDebugHelpers.h"
DEFINE_LOG_CATEGORY(LogPS_AI_Agent_Posture);
DEFINE_LOG_CATEGORY(LogPS_AI_ConvAgent_Posture);
// ── ARKit eye curve names ────────────────────────────────────────────────────
static const FName EyeLookUpLeft(TEXT("eyeLookUpLeft"));
@ -30,7 +30,7 @@ static constexpr float ARKitEyeRangeVertical = 35.0f;
// Construction
// ─────────────────────────────────────────────────────────────────────────────
UPS_AI_Agent_PostureComponent::UPS_AI_Agent_PostureComponent()
UPS_AI_ConvAgent_PostureComponent::UPS_AI_ConvAgent_PostureComponent()
{
PrimaryComponentTick.bCanEverTick = true;
PrimaryComponentTick.TickGroup = TG_PrePhysics;
@ -46,14 +46,14 @@ UPS_AI_Agent_PostureComponent::UPS_AI_Agent_PostureComponent()
// BeginPlay
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_PostureComponent::BeginPlay()
void UPS_AI_ConvAgent_PostureComponent::BeginPlay()
{
Super::BeginPlay();
AActor* Owner = GetOwner();
if (!Owner)
{
UE_LOG(LogPS_AI_Agent_Posture, Warning, TEXT("No owner actor — posture disabled."));
UE_LOG(LogPS_AI_ConvAgent_Posture, Warning, TEXT("No owner actor — posture disabled."));
return;
}
@ -82,11 +82,11 @@ void UPS_AI_Agent_PostureComponent::BeginPlay()
}
if (!CachedMesh.IsValid())
{
UE_LOG(LogPS_AI_Agent_Posture, Warning,
UE_LOG(LogPS_AI_ConvAgent_Posture, Warning,
TEXT("No SkeletalMeshComponent found on %s — head bone lookup will be unavailable."),
*Owner->GetName());
}
UE_LOG(LogPS_AI_Agent_Posture, Log,
UE_LOG(LogPS_AI_ConvAgent_Posture, Log,
TEXT("Mesh cache: Body=%s Face=%s"),
CachedMesh.IsValid() ? *CachedMesh->GetName() : TEXT("NONE"),
CachedFaceMesh.IsValid() ? *CachedFaceMesh->GetName() : TEXT("NONE"));
@ -96,7 +96,7 @@ void UPS_AI_Agent_PostureComponent::BeginPlay()
OriginalActorYaw = Owner->GetActorRotation().Yaw + MeshForwardYawOffset;
TargetBodyWorldYaw = Owner->GetActorRotation().Yaw;
UE_LOG(LogPS_AI_Agent_Posture, Log,
UE_LOG(LogPS_AI_ConvAgent_Posture, Log,
TEXT("Posture initialized on %s. MeshOffset=%.0f OriginalYaw=%.0f MaxEye=%.0f/%.0f MaxHead=%.0f/%.0f"),
*Owner->GetName(), MeshForwardYawOffset, OriginalActorYaw,
MaxEyeHorizontal, MaxEyeVertical, MaxHeadYaw, MaxHeadPitch);
@ -106,7 +106,7 @@ void UPS_AI_Agent_PostureComponent::BeginPlay()
// Map eye angles to 8 ARKit eye curves
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_PostureComponent::UpdateEyeCurves(float EyeYaw, float EyePitch)
void UPS_AI_ConvAgent_PostureComponent::UpdateEyeCurves(float EyeYaw, float EyePitch)
{
CurrentEyeCurves.Reset();
@ -176,7 +176,7 @@ void UPS_AI_Agent_PostureComponent::UpdateEyeCurves(float EyeYaw, float EyePitch
// point shifts so small movements don't re-trigger higher layers.
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_PostureComponent::TickComponent(
void UPS_AI_ConvAgent_PostureComponent::TickComponent(
float DeltaTime, ELevelTick TickType, FActorComponentTickFunction* ThisTickFunction)
{
Super::TickComponent(DeltaTime, TickType, ThisTickFunction);
@ -446,7 +446,7 @@ void UPS_AI_Agent_PostureComponent::TickComponent(
const float TgtYaw = FVector(Dir.X, Dir.Y, 0.0f).Rotation().Yaw;
const float Delta = FMath::FindDeltaAngleDegrees(FacingYaw, TgtYaw);
UE_LOG(LogPS_AI_Agent_Posture, Log,
UE_LOG(LogPS_AI_ConvAgent_Posture, Log,
TEXT("Posture [%s -> %s]: Delta=%.1f | Head=%.1f/%.1f | Eyes=%.1f/%.1f | EyeGap=%.1f"),
*Owner->GetName(), *TargetActor->GetName(),
Delta,

196
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_Agent_WebSocket_ElevenLabsProxy.cpp → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_WebSocket_ElevenLabsProxy.cpp
View File

@ -1,6 +1,6 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_Agent_WebSocket_ElevenLabsProxy.h"
#include "PS_AI_ConvAgent_WebSocket_ElevenLabsProxy.h"
#include "PS_AI_ConvAgent.h"
#include "WebSocketsModule.h"
@ -10,7 +10,7 @@
#include "JsonUtilities.h"
#include "Misc/Base64.h"
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_Agent_WS_ElevenLabs, Log, All);
DEFINE_LOG_CATEGORY_STATIC(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, All);
// ─────────────────────────────────────────────────────────────────────────────
// Helpers
@ -24,18 +24,18 @@ static void EL_LOG(bool bVerbose, const TCHAR* Format, ...)
TCHAR Buffer[2048];
FCString::GetVarArgs(Buffer, UE_ARRAY_COUNT(Buffer), Format, Args);
va_end(Args);
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT("%s"), Buffer);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT("%s"), Buffer);
}
// ─────────────────────────────────────────────────────────────────────────────
// Connect / Disconnect
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::Connect(const FString& AgentIDOverride, const FString& APIKeyOverride)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::Connect(const FString& AgentIDOverride, const FString& APIKeyOverride)
{
if (ConnectionState == EPS_AI_Agent_ConnectionState_ElevenLabs::Connected ||
ConnectionState == EPS_AI_Agent_ConnectionState_ElevenLabs::Connecting)
if (ConnectionState == EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Connected ||
ConnectionState == EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Connecting)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("Connect called but already connecting/connected. Ignoring."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("Connect called but already connecting/connected. Ignoring."));
return;
}
@ -48,19 +48,19 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::Connect(const FString& AgentIDOverr
if (URL.IsEmpty())
{
const FString Msg = TEXT("Cannot connect: no Agent ID configured. Set it in Project Settings or pass it to Connect().");
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Error, TEXT("%s"), *Msg);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Error, TEXT("%s"), *Msg);
OnError.Broadcast(Msg);
ConnectionState = EPS_AI_Agent_ConnectionState_ElevenLabs::Error;
ConnectionState = EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Error;
return;
}
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("Connecting to ElevenLabs: %s"), *URL);
ConnectionState = EPS_AI_Agent_ConnectionState_ElevenLabs::Connecting;
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("Connecting to ElevenLabs: %s"), *URL);
ConnectionState = EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Connecting;
// Headers: the ElevenLabs Conversational AI WS endpoint accepts the
// xi-api-key header on the initial HTTP upgrade request.
TMap<FString, FString> UpgradeHeaders;
const UPS_AI_Agent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
const UPS_AI_ConvAgent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
const FString ResolvedKey = APIKeyOverride.IsEmpty() ? Settings->API_Key : APIKeyOverride;
if (!ResolvedKey.IsEmpty())
{
@ -69,36 +69,36 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::Connect(const FString& AgentIDOverr
WebSocket = FWebSocketsModule::Get().CreateWebSocket(URL, TEXT(""), UpgradeHeaders);
WebSocket->OnConnected().AddUObject(this, &UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsConnected);
WebSocket->OnConnectionError().AddUObject(this, &UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsConnectionError);
WebSocket->OnClosed().AddUObject(this, &UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsClosed);
WebSocket->OnConnected().AddUObject(this, &UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::OnWsConnected);
WebSocket->OnConnectionError().AddUObject(this, &UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::OnWsConnectionError);
WebSocket->OnClosed().AddUObject(this, &UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::OnWsClosed);
// NOTE: We bind ONLY OnRawMessage (binary frames), NOT OnMessage (text frames).
// UE's WebSocket implementation fires BOTH callbacks for the same frame when using
// the libwebsockets backend — binding both causes every audio packet to be decoded
// and played twice. OnRawMessage handles all frame types: raw binary audio AND
// text-framed JSON (detected by peeking first byte for '{').
WebSocket->OnRawMessage().AddUObject(this, &UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsBinaryMessage);
WebSocket->OnRawMessage().AddUObject(this, &UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::OnWsBinaryMessage);
WebSocket->Connect();
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::Disconnect()
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::Disconnect()
{
if (WebSocket.IsValid() && WebSocket->IsConnected())
{
WebSocket->Close(1000, TEXT("Client disconnected"));
}
ConnectionState = EPS_AI_Agent_ConnectionState_ElevenLabs::Disconnected;
ConnectionState = EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Disconnected;
}
// ─────────────────────────────────────────────────────────────────────────────
// Audio & turn control
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendAudioChunk(const TArray<uint8>& PCMData)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::SendAudioChunk(const TArray<uint8>& PCMData)
{
if (!IsConnected())
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("SendAudioChunk: not connected (state=%d). Audio dropped."),
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("SendAudioChunk: not connected (state=%d). Audio dropped."),
(int32)ConnectionState);
return;
}
@ -118,7 +118,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendAudioChunk(const TArray<uint8>&
const FString AudioJson = FString::Printf(TEXT("{\"user_audio_chunk\":\"%s\"}"), *Base64Audio);
// Per-chunk log at Verbose only — Log level is too spammy (10+ lines per second).
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT("SendAudioChunk: %d bytes"), PCMData.Num());
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT("SendAudioChunk: %d bytes"), PCMData.Num());
{
FScopeLock Lock(&WebSocketSendLock);
@ -129,7 +129,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendAudioChunk(const TArray<uint8>&
}
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendUserTurnStart()
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::SendUserTurnStart()
{
// No-op: the PS_AI_Agent|ElevenLabs API does not require a "start speaking" signal.
// The server's VAD detects speech from the audio chunks we send.
@ -145,10 +145,10 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendUserTurnStart()
bAgentResponseStartedFired = false;
const double T = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("[T+%.2fs] User turn started — mic open, audio chunks will follow."), T);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("[T+%.2fs] User turn started — mic open, audio chunks will follow."), T);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendUserTurnEnd()
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::SendUserTurnEnd()
{
// No explicit "end turn" message exists in the PS_AI_Agent|ElevenLabs API.
// The server detects end-of-speech via VAD when we stop sending audio chunks.
@ -162,42 +162,42 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendUserTurnEnd()
// The flag is only reset in SendUserTurnStart() at the beginning of a new user turn.
const double T = UserTurnEndTime - SessionStartTime;
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("[T+%.2fs] User turn ended — server VAD silence detection started (turn_timeout=1s)."), T);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("[T+%.2fs] User turn ended — server VAD silence detection started (turn_timeout=1s)."), T);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendTextMessage(const FString& Text)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::SendTextMessage(const FString& Text)
{
if (!IsConnected())
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("SendTextMessage: not connected."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("SendTextMessage: not connected."));
return;
}
if (Text.IsEmpty()) return;
// API: { "type": "user_message", "text": "Hello agent" }
TSharedPtr<FJsonObject> Msg = MakeShareable(new FJsonObject());
Msg->SetStringField(TEXT("type"), PS_AI_Agent_MessageType_ElevenLabs::UserMessage);
Msg->SetStringField(TEXT("type"), PS_AI_ConvAgent_MessageType_ElevenLabs::UserMessage);
Msg->SetStringField(TEXT("text"), Text);
SendJsonMessage(Msg);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendInterrupt()
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::SendInterrupt()
{
if (!IsConnected()) return;
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("Sending interrupt."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("Sending interrupt."));
TSharedPtr<FJsonObject> Msg = MakeShareable(new FJsonObject());
Msg->SetStringField(TEXT("type"), PS_AI_Agent_MessageType_ElevenLabs::Interrupt);
Msg->SetStringField(TEXT("type"), PS_AI_ConvAgent_MessageType_ElevenLabs::Interrupt);
SendJsonMessage(Msg);
}
// ─────────────────────────────────────────────────────────────────────────────
// WebSocket callbacks
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsConnected()
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::OnWsConnected()
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("WebSocket connected. Sending conversation_initiation_client_data..."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("WebSocket connected. Sending conversation_initiation_client_data..."));
// State stays Connecting until we receive conversation_initiation_metadata from the server.
// ElevenLabs requires this message immediately after the WebSocket handshake to
@ -231,7 +231,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsConnected()
// In Server VAD mode, the config override is empty (matches C++ sample exactly).
TSharedPtr<FJsonObject> ConversationConfigOverride = MakeShareable(new FJsonObject());
if (TurnMode == EPS_AI_Agent_TurnMode_ElevenLabs::Client)
if (TurnMode == EPS_AI_ConvAgent_TurnMode_ElevenLabs::Client)
{
// turn_timeout: how long the server waits after VAD detects silence before
// processing the user's turn. Default is ~3s. In push-to-talk mode this
@ -267,7 +267,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsConnected()
// agent's configured custom_llm_extra_body with nothing. Omit entirely.
TSharedPtr<FJsonObject> InitMsg = MakeShareable(new FJsonObject());
InitMsg->SetStringField(TEXT("type"), PS_AI_Agent_MessageType_ElevenLabs::ConversationClientData);
InitMsg->SetStringField(TEXT("type"), PS_AI_ConvAgent_MessageType_ElevenLabs::ConversationClientData);
InitMsg->SetObjectField(TEXT("conversation_config_override"), ConversationConfigOverride);
// NOTE: We bypass SendJsonMessage() here intentionally.
@ -277,38 +277,38 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsConnected()
FString InitJson;
TSharedRef<TJsonWriter<>> InitWriter = TJsonWriterFactory<>::Create(&InitJson);
FJsonSerializer::Serialize(InitMsg.ToSharedRef(), InitWriter);
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("Sending initiation: %s"), *InitJson);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("Sending initiation: %s"), *InitJson);
WebSocket->Send(InitJson);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsConnectionError(const FString& Error)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::OnWsConnectionError(const FString& Error)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Error, TEXT("WebSocket connection error: %s"), *Error);
ConnectionState = EPS_AI_Agent_ConnectionState_ElevenLabs::Error;
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Error, TEXT("WebSocket connection error: %s"), *Error);
ConnectionState = EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Error;
OnError.Broadcast(Error);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsClosed(int32 StatusCode, const FString& Reason, bool bWasClean)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::OnWsClosed(int32 StatusCode, const FString& Reason, bool bWasClean)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("WebSocket closed. Code=%d Reason=%s Clean=%d"), StatusCode, *Reason, bWasClean);
ConnectionState = EPS_AI_Agent_ConnectionState_ElevenLabs::Disconnected;
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("WebSocket closed. Code=%d Reason=%s Clean=%d"), StatusCode, *Reason, bWasClean);
ConnectionState = EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Disconnected;
WebSocket.Reset();
OnDisconnected.Broadcast(StatusCode, Reason);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsMessage(const FString& Message)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::OnWsMessage(const FString& Message)
{
const UPS_AI_Agent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
const UPS_AI_ConvAgent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
if (Settings->bVerboseLogging)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT(">> %s"), *Message);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT(">> %s"), *Message);
}
TSharedPtr<FJsonObject> Root;
TSharedRef<TJsonReader<>> Reader = TJsonReaderFactory<>::Create(Message);
if (!FJsonSerializer::Deserialize(Reader, Root) || !Root.IsValid())
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("Failed to parse WebSocket message as JSON (first 80 chars): %.80s"), *Message);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("Failed to parse WebSocket message as JSON (first 80 chars): %.80s"), *Message);
return;
}
@ -318,26 +318,26 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsMessage(const FString& Message)
{
// Fallback: some messages use the top-level key as the type
// e.g. { "user_audio_chunk": "..." } from ourselves (shouldn't arrive)
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT("Message has no 'type' field, ignoring."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT("Message has no 'type' field, ignoring."));
return;
}
// Suppress ping from the visible log — they arrive every ~2s and flood the output.
// Handle ping early before the generic type log.
if (MsgType == PS_AI_Agent_MessageType_ElevenLabs::PingEvent)
if (MsgType == PS_AI_ConvAgent_MessageType_ElevenLabs::PingEvent)
{
HandlePing(Root);
return;
}
// Log every non-ping message type received from the server.
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("Received message type: %s"), *MsgType);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("Received message type: %s"), *MsgType);
if (MsgType == PS_AI_Agent_MessageType_ElevenLabs::ConversationInitiation)
if (MsgType == PS_AI_ConvAgent_MessageType_ElevenLabs::ConversationInitiation)
{
HandleConversationInitiation(Root);
}
else if (MsgType == PS_AI_Agent_MessageType_ElevenLabs::AudioResponse)
else if (MsgType == PS_AI_ConvAgent_MessageType_ElevenLabs::AudioResponse)
{
// Log time-to-first-audio: latency between end of user turn and first agent audio.
if (bWaitingForResponse && !bFirstAudioResponseLogged)
@ -346,14 +346,14 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsMessage(const FString& Message)
const double T = Now - SessionStartTime;
const double LatencyFromTurnEnd = (Now - UserTurnEndTime) * 1000.0;
const double LatencyFromLastChunk = (Now - LastAudioChunkSentTime) * 1000.0;
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning,
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning,
TEXT("[T+%.2fs] [LATENCY] First audio: %.0f ms after turn end (%.0f ms after last chunk)"),
T, LatencyFromTurnEnd, LatencyFromLastChunk);
bFirstAudioResponseLogged = true;
}
HandleAudioResponse(Root);
}
else if (MsgType == PS_AI_Agent_MessageType_ElevenLabs::UserTranscript)
else if (MsgType == PS_AI_ConvAgent_MessageType_ElevenLabs::UserTranscript)
{
// Log transcription latency.
if (bWaitingForResponse)
@ -361,14 +361,14 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsMessage(const FString& Message)
const double Now = FPlatformTime::Seconds();
const double T = Now - SessionStartTime;
const double LatencyFromTurnEnd = (Now - UserTurnEndTime) * 1000.0;
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning,
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning,
TEXT("[T+%.2fs] [LATENCY] User transcript: %.0f ms after turn end"),
T, LatencyFromTurnEnd);
bWaitingForResponse = false;
}
HandleTranscript(Root);
}
else if (MsgType == PS_AI_Agent_MessageType_ElevenLabs::AgentResponse)
else if (MsgType == PS_AI_ConvAgent_MessageType_ElevenLabs::AgentResponse)
{
// Log agent text response latency.
if (UserTurnEndTime > 0.0)
@ -376,36 +376,36 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsMessage(const FString& Message)
const double Now = FPlatformTime::Seconds();
const double T = Now - SessionStartTime;
const double LatencyFromTurnEnd = (Now - UserTurnEndTime) * 1000.0;
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning,
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning,
TEXT("[T+%.2fs] [LATENCY] Agent text response: %.0f ms after turn end"),
T, LatencyFromTurnEnd);
}
HandleAgentResponse(Root);
}
else if (MsgType == PS_AI_Agent_MessageType_ElevenLabs::AgentChatResponsePart)
else if (MsgType == PS_AI_ConvAgent_MessageType_ElevenLabs::AgentChatResponsePart)
{
HandleAgentChatResponsePart(Root);
}
else if (MsgType == PS_AI_Agent_MessageType_ElevenLabs::AgentResponseCorrection)
else if (MsgType == PS_AI_ConvAgent_MessageType_ElevenLabs::AgentResponseCorrection)
{
// Silently ignore — corrected text after interruption.
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT("agent_response_correction received (ignored)."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT("agent_response_correction received (ignored)."));
}
else if (MsgType == PS_AI_Agent_MessageType_ElevenLabs::ClientToolCall)
else if (MsgType == PS_AI_ConvAgent_MessageType_ElevenLabs::ClientToolCall)
{
HandleClientToolCall(Root);
}
else if (MsgType == PS_AI_Agent_MessageType_ElevenLabs::InterruptionEvent)
else if (MsgType == PS_AI_ConvAgent_MessageType_ElevenLabs::InterruptionEvent)
{
HandleInterruption(Root);
}
else
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT("Unhandled message type: %s"), *MsgType);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT("Unhandled message type: %s"), *MsgType);
}
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsBinaryMessage(const void* Data, SIZE_T Size, SIZE_T BytesRemaining)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::OnWsBinaryMessage(const void* Data, SIZE_T Size, SIZE_T BytesRemaining)
{
// Accumulate fragments until BytesRemaining == 0.
const uint8* Bytes = static_cast<const uint8*>(Data);
@ -430,10 +430,10 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsBinaryMessage(const void* Data,
reinterpret_cast<const char*>(BinaryFrameBuffer.GetData())));
BinaryFrameBuffer.Reset();
const UPS_AI_Agent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
const UPS_AI_ConvAgent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
if (Settings->bVerboseLogging)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT("Binary JSON frame (%d bytes): %.120s"), TotalSize, *JsonString);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT("Binary JSON frame (%d bytes): %.120s"), TotalSize, *JsonString);
}
OnWsMessage(JsonString);
@ -442,7 +442,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsBinaryMessage(const void* Data,
{
// Raw binary audio frame — PCM bytes sent directly without Base64/JSON wrapper.
// Log first few bytes as hex to help diagnose the format.
const UPS_AI_Agent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
const UPS_AI_ConvAgent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
if (Settings->bVerboseLogging)
{
FString HexPreview;
@ -451,7 +451,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsBinaryMessage(const void* Data,
{
HexPreview += FString::Printf(TEXT("%02X "), BinaryFrameBuffer[i]);
}
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT("Binary audio frame: %d bytes | first bytes: %s"), TotalSize, *HexPreview);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT("Binary audio frame: %d bytes | first bytes: %s"), TotalSize, *HexPreview);
}
// Broadcast raw PCM bytes directly to the audio queue.
@ -464,7 +464,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::OnWsBinaryMessage(const void* Data,
// ─────────────────────────────────────────────────────────────────────────────
// Message handlers
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleConversationInitiation(const TSharedPtr<FJsonObject>& Root)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::HandleConversationInitiation(const TSharedPtr<FJsonObject>& Root)
{
// Expected structure:
// { "type": "conversation_initiation_metadata",
@ -480,12 +480,12 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleConversationInitiation(const
}
SessionStartTime = FPlatformTime::Seconds();
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("[T+0.00s] Conversation initiated. ID=%s"), *ConversationInfo.ConversationID);
ConnectionState = EPS_AI_Agent_ConnectionState_ElevenLabs::Connected;
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("[T+0.00s] Conversation initiated. ID=%s"), *ConversationInfo.ConversationID);
ConnectionState = EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Connected;
OnConnected.Broadcast(ConversationInfo);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleAudioResponse(const TSharedPtr<FJsonObject>& Root)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::HandleAudioResponse(const TSharedPtr<FJsonObject>& Root)
{
// Expected structure:
// { "type": "audio",
@ -494,7 +494,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleAudioResponse(const TSharedPt
const TSharedPtr<FJsonObject>* AudioEvent = nullptr;
if (!Root->TryGetObjectField(TEXT("audio_event"), AudioEvent) || !AudioEvent)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("audio message missing 'audio_event' field."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("audio message missing 'audio_event' field."));
return;
}
@ -505,28 +505,28 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleAudioResponse(const TSharedPt
(*AudioEvent)->TryGetNumberField(TEXT("event_id"), EventId);
if (EventId > 0 && EventId <= LastInterruptEventId)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT("Discarding audio event_id=%d (interrupted at %d)."), EventId, LastInterruptEventId);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT("Discarding audio event_id=%d (interrupted at %d)."), EventId, LastInterruptEventId);
return;
}
FString Base64Audio;
if (!(*AudioEvent)->TryGetStringField(TEXT("audio_base_64"), Base64Audio))
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("audio_event missing 'audio_base_64' field."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("audio_event missing 'audio_base_64' field."));
return;
}
TArray<uint8> PCMData;
if (!FBase64::Decode(Base64Audio, PCMData))
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("Failed to Base64-decode audio data."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("Failed to Base64-decode audio data."));
return;
}
OnAudioReceived.Broadcast(PCMData);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleTranscript(const TSharedPtr<FJsonObject>& Root)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::HandleTranscript(const TSharedPtr<FJsonObject>& Root)
{
// API structure:
// { "type": "user_transcript",
@ -536,11 +536,11 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleTranscript(const TSharedPtr<F
const TSharedPtr<FJsonObject>* TranscriptEvent = nullptr;
if (!Root->TryGetObjectField(TEXT("user_transcription_event"), TranscriptEvent) || !TranscriptEvent)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("user_transcript message missing 'user_transcription_event' field."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("user_transcript message missing 'user_transcription_event' field."));
return;
}
FPS_AI_Agent_TranscriptSegment_ElevenLabs Segment;
FPS_AI_ConvAgent_TranscriptSegment_ElevenLabs Segment;
Segment.Speaker = TEXT("user");
(*TranscriptEvent)->TryGetStringField(TEXT("user_transcript"), Segment.Text);
// user_transcript messages are always final (interim results are not sent for user speech)
@ -549,7 +549,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleTranscript(const TSharedPtr<F
OnTranscript.Broadcast(Segment);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleAgentResponse(const TSharedPtr<FJsonObject>& Root)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::HandleAgentResponse(const TSharedPtr<FJsonObject>& Root)
{
// ISSUE-22: reset bAgentResponseStartedFired so OnAgentResponseStarted fires again on
// the next turn. In Server VAD mode SendUserTurnStart() is never called — it is the only
@ -574,7 +574,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleAgentResponse(const TSharedPt
OnAgentResponse.Broadcast(ResponseText);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleAgentChatResponsePart(const TSharedPtr<FJsonObject>& Root)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::HandleAgentChatResponsePart(const TSharedPtr<FJsonObject>& Root)
{
// agent_chat_response_part = the server is actively generating a response (LLM token stream).
// Fire OnAgentResponseStarted once per turn so the component can auto-stop the microphone
@ -591,7 +591,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleAgentChatResponsePart(const T
// of the previous response).
if (LastInterruptEventId > 0)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log,
TEXT("New generation started — resetting LastInterruptEventId (was %d)."),
LastInterruptEventId);
LastInterruptEventId = 0;
@ -600,7 +600,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleAgentChatResponsePart(const T
const double Now = FPlatformTime::Seconds();
const double T = Now - SessionStartTime;
const double LatencyFromTurnEnd = UserTurnEndTime > 0.0 ? (Now - UserTurnEndTime) * 1000.0 : 0.0;
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log,
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log,
TEXT("[T+%.2fs] Agent started generating (%.0f ms after turn end — includes VAD silence timeout + LLM start)."),
T, LatencyFromTurnEnd);
OnAgentResponseStarted.Broadcast();
@ -643,7 +643,7 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleAgentChatResponsePart(const T
}
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleInterruption(const TSharedPtr<FJsonObject>& Root)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::HandleInterruption(const TSharedPtr<FJsonObject>& Root)
{
// Extract the interrupt event_id so we can filter stale audio frames.
// { "type": "interruption", "interruption_event": { "event_id": 42 } }
@ -658,11 +658,11 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleInterruption(const TSharedPtr
}
}
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("Agent interrupted (server ack, LastInterruptEventId=%d)."), LastInterruptEventId);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("Agent interrupted (server ack, LastInterruptEventId=%d)."), LastInterruptEventId);
OnInterrupted.Broadcast();
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleClientToolCall(const TSharedPtr<FJsonObject>& Root)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::HandleClientToolCall(const TSharedPtr<FJsonObject>& Root)
{
// Incoming: { "type": "client_tool_call", "client_tool_call": {
// "tool_name": "set_emotion", "tool_call_id": "abc123",
@ -670,11 +670,11 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleClientToolCall(const TSharedP
const TSharedPtr<FJsonObject>* ToolCallObj = nullptr;
if (!Root->TryGetObjectField(TEXT("client_tool_call"), ToolCallObj) || !ToolCallObj)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("client_tool_call: missing client_tool_call object."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("client_tool_call: missing client_tool_call object."));
return;
}
FPS_AI_Agent_ClientToolCall_ElevenLabs ToolCall;
FPS_AI_ConvAgent_ClientToolCall_ElevenLabs ToolCall;
(*ToolCallObj)->TryGetStringField(TEXT("tool_name"), ToolCall.ToolName);
(*ToolCallObj)->TryGetStringField(TEXT("tool_call_id"), ToolCall.ToolCallId);
@ -698,29 +698,29 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandleClientToolCall(const TSharedP
}
const double T = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("[T+%.2fs] Client tool call: %s (id=%s, %d params)"),
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("[T+%.2fs] Client tool call: %s (id=%s, %d params)"),
T, *ToolCall.ToolName, *ToolCall.ToolCallId, ToolCall.Parameters.Num());
OnClientToolCall.Broadcast(ToolCall);
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendClientToolResult(const FString& ToolCallId, const FString& Result, bool bIsError)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::SendClientToolResult(const FString& ToolCallId, const FString& Result, bool bIsError)
{
// Outgoing: { "type": "client_tool_result", "tool_call_id": "abc123",
// "result": "emotion set to surprise", "is_error": false }
TSharedPtr<FJsonObject> Msg = MakeShareable(new FJsonObject());
Msg->SetStringField(TEXT("type"), PS_AI_Agent_MessageType_ElevenLabs::ClientToolResult);
Msg->SetStringField(TEXT("type"), PS_AI_ConvAgent_MessageType_ElevenLabs::ClientToolResult);
Msg->SetStringField(TEXT("tool_call_id"), ToolCallId);
Msg->SetStringField(TEXT("result"), Result);
Msg->SetBoolField(TEXT("is_error"), bIsError);
SendJsonMessage(Msg);
const double T = FPlatformTime::Seconds() - SessionStartTime;
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Log, TEXT("[T+%.2fs] Sent client_tool_result for %s: %s (error=%s)"),
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Log, TEXT("[T+%.2fs] Sent client_tool_result for %s: %s (error=%s)"),
T, *ToolCallId, *Result, bIsError ? TEXT("true") : TEXT("false"));
}
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandlePing(const TSharedPtr<FJsonObject>& Root)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::HandlePing(const TSharedPtr<FJsonObject>& Root)
{
// Reply with a pong to keep the connection alive.
// Incoming: { "type": "ping", "ping_event": { "event_id": 1, "ping_ms": 150 } }
@ -741,11 +741,11 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::HandlePing(const TSharedPtr<FJsonOb
// ─────────────────────────────────────────────────────────────────────────────
// Helpers
// ─────────────────────────────────────────────────────────────────────────────
void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendJsonMessage(const TSharedPtr<FJsonObject>& JsonObj)
void UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::SendJsonMessage(const TSharedPtr<FJsonObject>& JsonObj)
{
if (!WebSocket.IsValid() || !WebSocket->IsConnected())
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Warning, TEXT("SendJsonMessage: WebSocket not connected."));
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Warning, TEXT("SendJsonMessage: WebSocket not connected."));
return;
}
@ -753,10 +753,10 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendJsonMessage(const TSharedPtr<FJ
TSharedRef<TJsonWriter<>> Writer = TJsonWriterFactory<>::Create(&Out);
FJsonSerializer::Serialize(JsonObj.ToSharedRef(), Writer);
const UPS_AI_Agent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
const UPS_AI_ConvAgent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
if (Settings->bVerboseLogging)
{
UE_LOG(LogPS_AI_Agent_WS_ElevenLabs, Verbose, TEXT("<< %s"), *Out);
UE_LOG(LogPS_AI_ConvAgent_WS_ElevenLabs, Verbose, TEXT("<< %s"), *Out);
}
{
@ -765,9 +765,9 @@ void UPS_AI_Agent_WebSocket_ElevenLabsProxy::SendJsonMessage(const TSharedPtr<FJ
}
}
FString UPS_AI_Agent_WebSocket_ElevenLabsProxy::BuildWebSocketURL(const FString& AgentIDOverride, const FString& APIKeyOverride) const
FString UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy::BuildWebSocketURL(const FString& AgentIDOverride, const FString& APIKeyOverride) const
{
const UPS_AI_Agent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
const UPS_AI_ConvAgent_Settings_ElevenLabs* Settings = FPS_AI_ConvAgentModule::Get().GetSettings();
// Custom URL override takes full precedence
if (!Settings->CustomWebSocketURL.IsEmpty())

14
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/AnimNode_PS_AI_Agent_FacialExpression.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/AnimNode_PS_AI_ConvAgent_FacialExpression.h
View File

@ -4,28 +4,28 @@
#include "CoreMinimal.h"
#include "Animation/AnimNodeBase.h"
#include "AnimNode_PS_AI_Agent_FacialExpression.generated.h"
#include "AnimNode_PS_AI_ConvAgent_FacialExpression.generated.h"
class UPS_AI_Agent_FacialExpressionComponent;
class UPS_AI_ConvAgent_FacialExpressionComponent;
/**
* Animation node that injects facial expression curves into the AnimGraph.
*
* Place this node in the MetaHuman Face AnimBP BEFORE the PS AI Agent Lip Sync node.
* Place this node in the MetaHuman Face AnimBP BEFORE the PS AI ConvAgent Lip Sync node.
* It reads emotion curves (eyes, eyebrows, cheeks, mouth mood) from the
* PS_AI_Agent_FacialExpressionComponent on the same Actor and outputs them as
* PS_AI_ConvAgent_FacialExpressionComponent on the same Actor and outputs them as
* CTRL_expressions_* animation curves.
*
* The Lip Sync node placed AFTER this one will override mouth-area curves
* during speech, while non-mouth emotion curves (eyes, brows) pass through.
*
* Graph layout:
* [Live Link Pose] → [PS AI Agent Facial Expression] → [PS AI Agent Lip Sync] → [mh_arkit_mapping_pose] → ...
* [Live Link Pose] → [PS AI ConvAgent Facial Expression] → [PS AI ConvAgent Lip Sync] → [mh_arkit_mapping_pose] → ...
*
* The node auto-discovers the FacialExpressionComponent — no manual wiring needed.
*/
USTRUCT(BlueprintInternalUseOnly)
struct PS_AI_CONVAGENT_API FAnimNode_PS_AI_Agent_FacialExpression : public FAnimNode_Base
struct PS_AI_CONVAGENT_API FAnimNode_PS_AI_ConvAgent_FacialExpression : public FAnimNode_Base
{
GENERATED_USTRUCT_BODY()
@ -43,7 +43,7 @@ struct PS_AI_CONVAGENT_API FAnimNode_PS_AI_Agent_FacialExpression : public FAnim
private:
/** Cached reference to the facial expression component on the owning actor. */
TWeakObjectPtr<UPS_AI_Agent_FacialExpressionComponent> FacialExpressionComponent;
TWeakObjectPtr<UPS_AI_ConvAgent_FacialExpressionComponent> FacialExpressionComponent;
/** Emotion expression curves to inject (CTRL_expressions_* format).
* Copied from the component during Update (game thread safe). */

12
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/AnimNode_PS_AI_Agent_LipSync.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/AnimNode_PS_AI_ConvAgent_LipSync.h
View File

@ -4,26 +4,26 @@
#include "CoreMinimal.h"
#include "Animation/AnimNodeBase.h"
#include "AnimNode_PS_AI_Agent_LipSync.generated.h"
#include "AnimNode_PS_AI_ConvAgent_LipSync.generated.h"
class UPS_AI_Agent_LipSyncComponent;
class UPS_AI_ConvAgent_LipSyncComponent;
/**
* Animation node that injects lip sync curves into the AnimGraph.
*
* Place this node in the MetaHuman Face AnimBP BEFORE the mh_arkit_mapping_pose
* node. It reads ARKit blendshape weights from the PS_AI_Agent_LipSyncComponent
* node. It reads ARKit blendshape weights from the PS_AI_ConvAgent_LipSyncComponent
* on the same Actor and outputs them as animation curves (jawOpen, mouthFunnel,
* etc.). The downstream mh_arkit_mapping_pose node then converts these ARKit
* names to CTRL_expressions_* curves that drive the MetaHuman facial bones.
*
* Graph layout:
* [Live Link Pose] → [PS AI Agent Lip Sync] → [mh_arkit_mapping_pose] → ...
* [Live Link Pose] → [PS AI ConvAgent Lip Sync] → [mh_arkit_mapping_pose] → ...
*
* The node auto-discovers the LipSyncComponent — no manual wiring needed.
*/
USTRUCT(BlueprintInternalUseOnly)
struct PS_AI_CONVAGENT_API FAnimNode_PS_AI_Agent_LipSync : public FAnimNode_Base
struct PS_AI_CONVAGENT_API FAnimNode_PS_AI_ConvAgent_LipSync : public FAnimNode_Base
{
GENERATED_USTRUCT_BODY()
@ -41,7 +41,7 @@ struct PS_AI_CONVAGENT_API FAnimNode_PS_AI_Agent_LipSync : public FAnimNode_Base
private:
/** Cached reference to the lip sync component on the owning actor. */
TWeakObjectPtr<UPS_AI_Agent_LipSyncComponent> LipSyncComponent;
TWeakObjectPtr<UPS_AI_ConvAgent_LipSyncComponent> LipSyncComponent;
/** ARKit blendshape curves to inject (jawOpen, mouthFunnel, etc.).
* Copied from the component during Update (game thread safe). */

10
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/AnimNode_PS_AI_Agent_Posture.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/AnimNode_PS_AI_ConvAgent_Posture.h
View File

@ -5,9 +5,9 @@
#include "CoreMinimal.h"
#include "Animation/AnimNodeBase.h"
#include "BoneContainer.h"
#include "AnimNode_PS_AI_Agent_Posture.generated.h"
#include "AnimNode_PS_AI_ConvAgent_Posture.generated.h"
class UPS_AI_Agent_PostureComponent;
class UPS_AI_ConvAgent_PostureComponent;
/**
* Animation node that injects posture data into the AnimGraph.
@ -25,10 +25,10 @@ class UPS_AI_Agent_PostureComponent;
* → Injects ARKit eye curves before mh_arkit_mapping_pose.
* → Graph: [Source] → [Facial Expression] → [Posture] → [Lip Sync] → [mh_arkit] → ...
*
* The node auto-discovers the PS_AI_Agent_PostureComponent — no manual wiring needed.
* The node auto-discovers the PS_AI_ConvAgent_PostureComponent — no manual wiring needed.
*/
USTRUCT(BlueprintInternalUseOnly)
struct PS_AI_CONVAGENT_API FAnimNode_PS_AI_Agent_Posture : public FAnimNode_Base
struct PS_AI_CONVAGENT_API FAnimNode_PS_AI_ConvAgent_Posture : public FAnimNode_Base
{
GENERATED_USTRUCT_BODY()
@ -56,7 +56,7 @@ struct PS_AI_CONVAGENT_API FAnimNode_PS_AI_Agent_Posture : public FAnimNode_Base
private:
/** Cached reference to the posture component on the owning actor. */
TWeakObjectPtr<UPS_AI_Agent_PostureComponent> PostureComponent;
TWeakObjectPtr<UPS_AI_ConvAgent_PostureComponent> PostureComponent;
/** Eye gaze curves to inject (8 ARKit eye look curves).
* Copied from the component during Update (game thread safe). */

10
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent.h
View File

@ -7,15 +7,15 @@
#include "PS_AI_ConvAgent.generated.h"
// ─────────────────────────────────────────────────────────────────────────────
// Settings object – exposed in Project Settings → Plugins → PS AI Agent - ElevenLabs
// Settings object – exposed in Project Settings → Plugins → PS AI ConvAgent - ElevenLabs
// ─────────────────────────────────────────────────────────────────────────────
UCLASS(config = Engine, defaultconfig)
class PS_AI_CONVAGENT_API UPS_AI_Agent_Settings_ElevenLabs : public UObject
class PS_AI_CONVAGENT_API UPS_AI_ConvAgent_Settings_ElevenLabs : public UObject
{
GENERATED_BODY()
public:
UPS_AI_Agent_Settings_ElevenLabs(const FObjectInitializer& ObjectInitializer)
UPS_AI_ConvAgent_Settings_ElevenLabs(const FObjectInitializer& ObjectInitializer)
: Super(ObjectInitializer)
{
API_Key = TEXT("");
@ -92,8 +92,8 @@ public:
}
/** Access the settings object at runtime */
UPS_AI_Agent_Settings_ElevenLabs* GetSettings() const;
UPS_AI_ConvAgent_Settings_ElevenLabs* GetSettings() const;
private:
UPS_AI_Agent_Settings_ElevenLabs* Settings = nullptr;
UPS_AI_ConvAgent_Settings_ElevenLabs* Settings = nullptr;
};

36
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_Agent_Definitions.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_Definitions.h
View File

@ -3,13 +3,13 @@
#pragma once
#include "CoreMinimal.h"
#include "PS_AI_Agent_Definitions.generated.h"
#include "PS_AI_ConvAgent_Definitions.generated.h"
// ─────────────────────────────────────────────────────────────────────────────
// Connection state
// ─────────────────────────────────────────────────────────────────────────────
UENUM(BlueprintType)
enum class EPS_AI_Agent_ConnectionState_ElevenLabs : uint8
enum class EPS_AI_ConvAgent_ConnectionState_ElevenLabs : uint8
{
Disconnected UMETA(DisplayName = "Disconnected"),
Connecting UMETA(DisplayName = "Connecting"),
@ -21,7 +21,7 @@ enum class EPS_AI_Agent_ConnectionState_ElevenLabs : uint8
// Agent turn mode
// ─────────────────────────────────────────────────────────────────────────────
UENUM(BlueprintType)
enum class EPS_AI_Agent_TurnMode_ElevenLabs : uint8
enum class EPS_AI_ConvAgent_TurnMode_ElevenLabs : uint8
{
/** ElevenLabs server decides when the user has finished speaking (default). */
Server UMETA(DisplayName = "Server VAD"),
@ -32,7 +32,7 @@ enum class EPS_AI_Agent_TurnMode_ElevenLabs : uint8
// ─────────────────────────────────────────────────────────────────────────────
// WebSocket message type helpers (internal, not exposed to Blueprint)
// ─────────────────────────────────────────────────────────────────────────────
namespace PS_AI_Agent_MessageType_ElevenLabs
namespace PS_AI_ConvAgent_MessageType_ElevenLabs
{
// Client → Server
static const FString AudioChunk = TEXT("user_audio_chunk");
@ -62,7 +62,7 @@ namespace PS_AI_Agent_MessageType_ElevenLabs
// Audio format exchanged with ElevenLabs
// PCM 16-bit signed, 16000 Hz, mono, little-endian.
// ─────────────────────────────────────────────────────────────────────────────
namespace PS_AI_Agent_Audio_ElevenLabs
namespace PS_AI_ConvAgent_Audio_ElevenLabs
{
static constexpr int32 SampleRate = 16000;
static constexpr int32 Channels = 1;
@ -75,16 +75,16 @@ namespace PS_AI_Agent_Audio_ElevenLabs
// Conversation metadata received on successful connection
// ─────────────────────────────────────────────────────────────────────────────
USTRUCT(BlueprintType)
struct PS_AI_CONVAGENT_API FPS_AI_Agent_ConversationInfo_ElevenLabs
struct PS_AI_CONVAGENT_API FPS_AI_ConvAgent_ConversationInfo_ElevenLabs
{
GENERATED_BODY()
/** Unique ID of this conversation session assigned by ElevenLabs. */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
FString ConversationID;
/** Agent ID that is responding. */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
FString AgentID;
};
@ -92,20 +92,20 @@ struct PS_AI_CONVAGENT_API FPS_AI_Agent_ConversationInfo_ElevenLabs
// Transcript segment
// ─────────────────────────────────────────────────────────────────────────────
USTRUCT(BlueprintType)
struct PS_AI_CONVAGENT_API FPS_AI_Agent_TranscriptSegment_ElevenLabs
struct PS_AI_CONVAGENT_API FPS_AI_ConvAgent_TranscriptSegment_ElevenLabs
{
GENERATED_BODY()
/** Transcribed text. */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
FString Text;
/** "user" or "agent". */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
FString Speaker;
/** Whether this is a final transcript or a tentative (in-progress) one. */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
bool bIsFinal = false;
};
@ -113,7 +113,7 @@ struct PS_AI_CONVAGENT_API FPS_AI_Agent_TranscriptSegment_ElevenLabs
// Agent emotion (driven by client tool "set_emotion" from the LLM)
// ─────────────────────────────────────────────────────────────────────────────
UENUM(BlueprintType)
enum class EPS_AI_Agent_Emotion : uint8
enum class EPS_AI_ConvAgent_Emotion : uint8
{
Neutral UMETA(DisplayName = "Neutral"),
Joy UMETA(DisplayName = "Joy"),
@ -128,7 +128,7 @@ enum class EPS_AI_Agent_Emotion : uint8
// Emotion intensity (maps to Normal/Medium/Extreme pose variants)
// ─────────────────────────────────────────────────────────────────────────────
UENUM(BlueprintType)
enum class EPS_AI_Agent_EmotionIntensity : uint8
enum class EPS_AI_ConvAgent_EmotionIntensity : uint8
{
Low UMETA(DisplayName = "Low (Normal)"),
Medium UMETA(DisplayName = "Medium"),
@ -139,19 +139,19 @@ enum class EPS_AI_Agent_EmotionIntensity : uint8
// Client tool call received from ElevenLabs server
// ─────────────────────────────────────────────────────────────────────────────
USTRUCT(BlueprintType)
struct PS_AI_CONVAGENT_API FPS_AI_Agent_ClientToolCall_ElevenLabs
struct PS_AI_CONVAGENT_API FPS_AI_ConvAgent_ClientToolCall_ElevenLabs
{
GENERATED_BODY()
/** Name of the tool the agent wants to invoke (e.g. "set_emotion"). */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
FString ToolName;
/** Unique ID for this tool invocation — must be echoed back in client_tool_result. */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
FString ToolCallId;
/** Raw JSON parameters as key-value string pairs. */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
TMap<FString, FString> Parameters;
};

126
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_Agent_Conv_ElevenLabsComponent.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_ElevenLabsComponent.h
View File

@ -4,20 +4,20 @@
#include "CoreMinimal.h"
#include "Components/ActorComponent.h"
#include "PS_AI_Agent_Definitions.h"
#include "PS_AI_Agent_WebSocket_ElevenLabsProxy.h"
#include "PS_AI_ConvAgent_Definitions.h"
#include "PS_AI_ConvAgent_WebSocket_ElevenLabsProxy.h"
#include "Sound/SoundWaveProcedural.h"
#include <atomic>
#include "PS_AI_Agent_Conv_ElevenLabsComponent.generated.h"
#include "PS_AI_ConvAgent_ElevenLabsComponent.generated.h"
class UAudioComponent;
class UPS_AI_Agent_MicrophoneCaptureComponent;
class UPS_AI_ConvAgent_MicrophoneCaptureComponent;
// ─────────────────────────────────────────────────────────────────────────────
// Delegates exposed to Blueprint
// ─────────────────────────────────────────────────────────────────────────────
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnAgentConnected,
const FPS_AI_Agent_ConversationInfo_ElevenLabs&, ConversationInfo);
const FPS_AI_ConvAgent_ConversationInfo_ElevenLabs&, ConversationInfo);
DECLARE_DYNAMIC_MULTICAST_DELEGATE_TwoParams(FOnAgentDisconnected,
int32, StatusCode, const FString&, Reason);
@ -26,7 +26,7 @@ DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnAgentError,
const FString&, ErrorMessage);
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnAgentTranscript,
const FPS_AI_Agent_TranscriptSegment_ElevenLabs&, Segment);
const FPS_AI_ConvAgent_TranscriptSegment_ElevenLabs&, Segment);
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnAgentTextResponse,
const FString&, ResponseText);
@ -68,8 +68,8 @@ DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnAgentResponseTimeout);
* The emotion changes BEFORE the corresponding audio arrives, giving time to blend.
*/
DECLARE_DYNAMIC_MULTICAST_DELEGATE_TwoParams(FOnAgentEmotionChanged,
EPS_AI_Agent_Emotion, Emotion,
EPS_AI_Agent_EmotionIntensity, Intensity);
EPS_AI_ConvAgent_Emotion, Emotion,
EPS_AI_ConvAgent_EmotionIntensity, Intensity);
/**
* Fired for any client tool call that is NOT automatically handled (i.e. not "set_emotion").
@ -77,14 +77,14 @@ DECLARE_DYNAMIC_MULTICAST_DELEGATE_TwoParams(FOnAgentEmotionChanged,
* You MUST call SendClientToolResult on the WebSocketProxy to acknowledge the call.
*/
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnAgentClientToolCall,
const FPS_AI_Agent_ClientToolCall_ElevenLabs&, ToolCall);
const FPS_AI_ConvAgent_ClientToolCall_ElevenLabs&, ToolCall);
// Non-dynamic delegate for raw agent audio (high-frequency, C++ consumers only).
// Delivers PCM chunks as int16, 16kHz mono, little-endian.
DECLARE_MULTICAST_DELEGATE_OneParam(FOnAgentAudioData, const TArray<uint8>& /*PCMData*/);
// ─────────────────────────────────────────────────────────────────────────────
// UPS_AI_Agent_Conv_ElevenLabsComponent
// UPS_AI_ConvAgent_ElevenLabsComponent
//
// Attach this to any Actor (e.g. a character NPC) to give it a voice powered by
// the ElevenLabs Conversational AI API.
@ -96,60 +96,60 @@ DECLARE_MULTICAST_DELEGATE_OneParam(FOnAgentAudioData, const TArray<uint8>& /*PC
// 4. React to events (OnAgentTranscript, OnAgentTextResponse, etc.) in Blueprint.
// 5. Call EndConversation() when done.
// ─────────────────────────────────────────────────────────────────────────────
UCLASS(ClassGroup = "PS AI Agent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI Agent - ElevenLabs Conv")
class PS_AI_CONVAGENT_API UPS_AI_Agent_Conv_ElevenLabsComponent : public UActorComponent
UCLASS(ClassGroup = "PS AI ConvAgent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI ConvAgent - ElevenLabs Conv")
class PS_AI_CONVAGENT_API UPS_AI_ConvAgent_ElevenLabsComponent : public UActorComponent
{
GENERATED_BODY()
public:
UPS_AI_Agent_Conv_ElevenLabsComponent();
UPS_AI_ConvAgent_ElevenLabsComponent();
// ── Configuration ─────────────────────────────────────────────────────────
/** ElevenLabs Agent ID used for this conversation. Leave empty to use the default from Project Settings > PS AI Agent - ElevenLabs. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs",
/** ElevenLabs Agent ID used for this conversation. Leave empty to use the default from Project Settings > PS AI ConvAgent - ElevenLabs. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs",
meta = (ToolTip = "ElevenLabs Agent ID. Leave empty to use the project default from Project Settings."))
FString AgentID;
/** How turn-taking is managed between the user and the agent.\n- Server VAD (recommended): ElevenLabs automatically detects when the user stops speaking.\n- Client Controlled: You manually call StartListening/StopListening (push-to-talk with a key). */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs",
meta = (ToolTip = "Turn-taking mode.\n- Server VAD: ElevenLabs detects end-of-speech automatically (hands-free).\n- Client Controlled: You call StartListening/StopListening manually (push-to-talk)."))
EPS_AI_Agent_TurnMode_ElevenLabs TurnMode = EPS_AI_Agent_TurnMode_ElevenLabs::Server;
EPS_AI_ConvAgent_TurnMode_ElevenLabs TurnMode = EPS_AI_ConvAgent_TurnMode_ElevenLabs::Server;
/** Automatically open the microphone as soon as the WebSocket connection is established. Only applies in Server VAD mode. In Client (push-to-talk) mode, you must call StartListening manually. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs",
meta = (ToolTip = "Auto-open the microphone when the conversation starts.\nOnly applies in Server VAD mode. In push-to-talk mode, call StartListening() manually."))
bool bAutoStartListening = true;
/** Let the LLM start generating a response during silence, before the VAD is fully confident the user has finished speaking. Saves 200-500ms of latency but may be unstable in long multi-turn sessions. Disabled by default. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs|Latency",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs|Latency",
meta = (ToolTip = "Speculative turn: the LLM begins generating during silence before full turn-end confidence.\nReduces latency by 200-500ms. May be unstable in long sessions — test before enabling in production."))
bool bSpeculativeTurn = false;
/** How many milliseconds of microphone audio to accumulate before sending a chunk to ElevenLabs. Lower values reduce latency but may degrade voice detection accuracy. Higher values are more reliable but add delay. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs|Latency",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs|Latency",
meta = (ClampMin = "20", ClampMax = "500",
ToolTip = "Mic audio chunk duration sent to ElevenLabs.\n- 50-80ms: lower latency, less reliable voice detection.\n- 100ms (default): good balance.\n- 150-250ms: more reliable, higher latency."))
int32 MicChunkDurationMs = 100;
/** Allow the user to interrupt the agent while it is speaking.\n- In Server VAD mode: the microphone stays open during agent speech and the server detects interruptions automatically.\n- In Client (push-to-talk) mode: pressing the talk key while the agent speaks sends an interrupt signal.\n- When disabled: the user must wait for the agent to finish speaking before talking. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs",
meta = (ToolTip = "Allow the user to interrupt the agent while it speaks.\n- Server VAD: mic stays open, server detects user voice automatically.\n- Push-to-talk: pressing the talk key interrupts the agent.\n- Disabled: user must wait for the agent to finish."))
bool bAllowInterruption = true;
/** Enable the OnAgentTranscript event, which provides real-time speech-to-text of what the user is saying. Disable if you don't need to display user speech to reduce processing overhead. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fire OnAgentTranscript with real-time speech-to-text of user speech.\nDisable if you don't need to display what the user said."))
bool bEnableUserTranscript = true;
/** Enable the OnAgentTextResponse event, which provides the agent's complete text response once fully generated. Disable if you only need the audio output. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fire OnAgentTextResponse with the agent's complete text once fully generated.\nDisable if you only need the audio output."))
bool bEnableAgentTextResponse = true;
/** Enable the OnAgentPartialResponse event, which streams the agent's text word-by-word as the LLM generates it. Use this for real-time subtitles that appear while the agent speaks, rather than waiting for the full response. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fire OnAgentPartialResponse with streaming text fragments as the LLM generates them.\nIdeal for real-time subtitles. Each event gives one text chunk, not the accumulated text."))
bool bEnableAgentPartialResponse = false;
@ -157,13 +157,13 @@ public:
* Delays playback start so early TTS chunks can accumulate, preventing
* mid-sentence pauses when the second chunk hasn't arrived yet.
* Set to 0 for immediate playback. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs|Latency",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs|Latency",
meta = (ClampMin = "0", ClampMax = "4000",
ToolTip = "Pre-buffer delay in ms before starting audio playback.\nHigher values reduce mid-sentence pauses but add initial latency.\n0 = immediate playback."))
int32 AudioPreBufferMs = 2000;
/** Safety timeout: if the server does not start generating a response within this many seconds after the user stops speaking, fire OnAgentResponseTimeout. Set to 0 to disable. A normal response starts within 0.1-0.8s. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI Agent|ElevenLabs",
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS AI ConvAgent|ElevenLabs",
meta = (ClampMin = "0.0",
ToolTip = "Seconds to wait for a server response after the user stops speaking.\nFires OnAgentResponseTimeout if exceeded. Normal latency is 0.1-0.8s.\nSet to 0 to disable. Default: 10s."))
float ResponseTimeoutSeconds = 10.0f;
@ -171,81 +171,81 @@ public:
// ── Events ────────────────────────────────────────────────────────────────
/** Fired when the WebSocket connection is established and the conversation session is ready. Provides the ConversationID and AgentID. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires when the connection to ElevenLabs is established and the conversation is ready to begin."))
FOnAgentConnected OnAgentConnected;
/** Fired when the WebSocket connection is closed (gracefully or due to an error). Provides the status code and reason. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires when the connection to ElevenLabs is closed. Check StatusCode and Reason for details."))
FOnAgentDisconnected OnAgentDisconnected;
/** Fired on any connection or protocol error. The error message describes what went wrong. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires on connection or protocol errors. The ErrorMessage describes the issue."))
FOnAgentError OnAgentError;
/** Fired with real-time speech-to-text of the user's voice. Includes both tentative (in-progress) and final transcripts. Requires bEnableUserTranscript to be true. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Real-time speech-to-text of the user's voice.\nIncludes tentative and final transcripts. Enable with bEnableUserTranscript."))
FOnAgentTranscript OnAgentTranscript;
/** Fired once when the agent's complete text response is available. This is the full text that corresponds to the audio the agent speaks. Requires bEnableAgentTextResponse to be true. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "The agent's complete text response (matches the spoken audio).\nFires once when the full text is ready. Enable with bEnableAgentTextResponse."))
FOnAgentTextResponse OnAgentTextResponse;
/** Fired repeatedly as the LLM generates text, providing one word/fragment at a time. Use for real-time subtitles. Each call gives a new fragment, NOT the accumulated text. Requires bEnableAgentPartialResponse to be true. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Streaming text fragments as the LLM generates them (word by word).\nIdeal for real-time subtitles. Enable with bEnableAgentPartialResponse."))
FOnAgentPartialResponse OnAgentPartialResponse;
/** Fired when the agent begins playing audio (first audio chunk received). Use this to trigger speech animations or UI indicators. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires when the agent starts speaking (first audio chunk). Use for lip-sync or UI feedback."))
FOnAgentStartedSpeaking OnAgentStartedSpeaking;
/** Fired when the agent finishes playing all audio. Use this to re-open the microphone (in Server VAD mode without interruption) or update UI. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires when the agent finishes speaking. Use to re-open the mic or update UI."))
FOnAgentStoppedSpeaking OnAgentStoppedSpeaking;
/** Fired when the agent's speech is interrupted (either by the user speaking over it, or by a manual InterruptAgent call). The audio playback is automatically stopped. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires when the agent is interrupted mid-speech. Audio is automatically stopped."))
FOnAgentInterrupted OnAgentInterrupted;
/** Fired when the server starts generating a response (before any audio arrives). Use this for "thinking..." UI feedback. In push-to-talk mode, the microphone is automatically closed when this fires. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires when the server starts generating (before audio arrives).\nUse for 'thinking...' UI. Mic is auto-closed in push-to-talk mode."))
FOnAgentStartedGenerating OnAgentStartedGenerating;
/** Fired if the server does not start generating a response within ResponseTimeoutSeconds after the user stops speaking. Use this to show a "try again" message or automatically re-open the microphone. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires if the server doesn't respond within ResponseTimeoutSeconds.\nUse to show 'try again' or re-open the mic automatically."))
FOnAgentResponseTimeout OnAgentResponseTimeout;
/** Fired when the agent changes emotion via the "set_emotion" client tool. The emotion is set BEFORE the corresponding audio arrives, giving you time to smoothly blend facial expressions. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires when the agent sets an emotion (joy, sadness, surprise, fear, anger, disgust).\nDriven by the 'set_emotion' client tool. Arrives before the audio."))
FOnAgentEmotionChanged OnAgentEmotionChanged;
/** Fired for client tool calls that are NOT automatically handled (i.e. not "set_emotion"). You must call GetWebSocketProxy()->SendClientToolResult() to respond. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events",
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events",
meta = (ToolTip = "Fires for custom client tool calls (not set_emotion).\nYou must respond via GetWebSocketProxy()->SendClientToolResult()."))
FOnAgentClientToolCall OnAgentClientToolCall;
/** The current emotion of the agent, as set by the "set_emotion" client tool. Defaults to Neutral. */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
EPS_AI_Agent_Emotion CurrentEmotion = EPS_AI_Agent_Emotion::Neutral;
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
EPS_AI_ConvAgent_Emotion CurrentEmotion = EPS_AI_ConvAgent_Emotion::Neutral;
/** The current emotion intensity. Defaults to Medium. */
UPROPERTY(BlueprintReadOnly, Category = "PS AI Agent|ElevenLabs")
EPS_AI_Agent_EmotionIntensity CurrentEmotionIntensity = EPS_AI_Agent_EmotionIntensity::Medium;
UPROPERTY(BlueprintReadOnly, Category = "PS AI ConvAgent|ElevenLabs")
EPS_AI_ConvAgent_EmotionIntensity CurrentEmotionIntensity = EPS_AI_ConvAgent_EmotionIntensity::Medium;
// ── Raw audio data (C++ only, used by LipSync component) ────────────────
/** Raw PCM audio from the agent (int16, 16kHz mono). Fires for each WebSocket audio chunk.
* Used internally by UPS_AI_Agent_LipSyncComponent for spectral analysis. */
* Used internally by UPS_AI_ConvAgent_LipSyncComponent for spectral analysis. */
FOnAgentAudioData OnAgentAudioData;
// ── Control ───────────────────────────────────────────────────────────────
@ -254,25 +254,25 @@ public:
* Open the WebSocket connection and start the conversation.
* If bAutoStartListening is true, microphone capture also starts once connected.
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void StartConversation();
/** Close the WebSocket and stop all audio. */
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void EndConversation();
/**
* Start capturing microphone audio and streaming it to ElevenLabs.
* In Client turn mode, also sends a UserTurnStart signal.
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void StartListening();
/**
* Stop capturing microphone audio.
* In Client turn mode, also sends a UserTurnEnd signal.
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void StopListening();
/**
@ -280,35 +280,35 @@ public:
* The agent will respond with audio and text just as if it heard you speak.
* Useful for testing in the Editor or for text-based interaction.
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void SendTextMessage(const FString& Text);
/** Interrupt the agent's current utterance. */
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void InterruptAgent();
// ── State queries ─────────────────────────────────────────────────────────
UFUNCTION(BlueprintPure, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintPure, Category = "PS AI ConvAgent|ElevenLabs")
bool IsConnected() const;
UFUNCTION(BlueprintPure, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintPure, Category = "PS AI ConvAgent|ElevenLabs")
bool IsListening() const { return bIsListening; }
UFUNCTION(BlueprintPure, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintPure, Category = "PS AI ConvAgent|ElevenLabs")
bool IsAgentSpeaking() const { return bAgentSpeaking; }
UFUNCTION(BlueprintPure, Category = "PS AI Agent|ElevenLabs")
const FPS_AI_Agent_ConversationInfo_ElevenLabs& GetConversationInfo() const;
UFUNCTION(BlueprintPure, Category = "PS AI ConvAgent|ElevenLabs")
const FPS_AI_ConvAgent_ConversationInfo_ElevenLabs& GetConversationInfo() const;
/** True while audio is being pre-buffered (playback hasn't started yet).
* Used by the LipSync component to pause viseme queue consumption. */
UFUNCTION(BlueprintPure, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintPure, Category = "PS AI ConvAgent|ElevenLabs")
bool IsPreBuffering() const { return bPreBuffering; }
/** Access the underlying WebSocket proxy (advanced use). */
UFUNCTION(BlueprintPure, Category = "PS AI Agent|ElevenLabs")
UPS_AI_Agent_WebSocket_ElevenLabsProxy* GetWebSocketProxy() const { return WebSocketProxy; }
UFUNCTION(BlueprintPure, Category = "PS AI ConvAgent|ElevenLabs")
UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy* GetWebSocketProxy() const { return WebSocketProxy; }
// ─────────────────────────────────────────────────────────────────────────
// UActorComponent overrides
@ -321,7 +321,7 @@ public:
private:
// ── Internal event handlers ───────────────────────────────────────────────
UFUNCTION()
void HandleConnected(const FPS_AI_Agent_ConversationInfo_ElevenLabs& Info);
void HandleConnected(const FPS_AI_ConvAgent_ConversationInfo_ElevenLabs& Info);
UFUNCTION()
void HandleDisconnected(int32 StatusCode, const FString& Reason);
@ -333,7 +333,7 @@ private:
void HandleAudioReceived(const TArray<uint8>& PCMData);
UFUNCTION()
void HandleTranscript(const FPS_AI_Agent_TranscriptSegment_ElevenLabs& Segment);
void HandleTranscript(const FPS_AI_ConvAgent_TranscriptSegment_ElevenLabs& Segment);
UFUNCTION()
void HandleAgentResponse(const FString& ResponseText);
@ -348,7 +348,7 @@ private:
void HandleAgentResponsePart(const FString& PartialText);
UFUNCTION()
void HandleClientToolCall(const FPS_AI_Agent_ClientToolCall_ElevenLabs& ToolCall);
void HandleClientToolCall(const FPS_AI_ConvAgent_ClientToolCall_ElevenLabs& ToolCall);
// ── Audio playback ────────────────────────────────────────────────────────
void InitAudioPlayback();
@ -364,7 +364,7 @@ private:
// ── Sub-objects ───────────────────────────────────────────────────────────
UPROPERTY()
UPS_AI_Agent_WebSocket_ElevenLabsProxy* WebSocketProxy = nullptr;
UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy* WebSocketProxy = nullptr;
UPROPERTY()
UAudioComponent* AudioPlaybackComponent = nullptr;

16
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_Agent_EmotionPoseMap.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_EmotionPoseMap.h
View File

@ -5,8 +5,8 @@
#include "CoreMinimal.h"
#include "Engine/DataAsset.h"
#include "Engine/AssetManager.h"
#include "PS_AI_Agent_Definitions.h"
#include "PS_AI_Agent_EmotionPoseMap.generated.h"
#include "PS_AI_ConvAgent_Definitions.h"
#include "PS_AI_ConvAgent_EmotionPoseMap.generated.h"
class UAnimSequence;
@ -14,7 +14,7 @@ class UAnimSequence;
// Emotion pose set: 3 intensity levels (Normal / Medium / Extreme)
// ─────────────────────────────────────────────────────────────────────────────
USTRUCT(BlueprintType)
struct PS_AI_CONVAGENT_API FPS_AI_Agent_EmotionPoseSet
struct PS_AI_CONVAGENT_API FPS_AI_ConvAgent_EmotionPoseSet
{
GENERATED_BODY()
@ -38,16 +38,16 @@ struct PS_AI_CONVAGENT_API FPS_AI_Agent_EmotionPoseSet
* Reusable data asset that maps emotions to facial expression AnimSequences.
*
* Create ONE instance of this asset in the Content Browser
* (right-click → Miscellaneous → Data Asset → PS_AI_Agent_EmotionPoseMap),
* (right-click → Miscellaneous → Data Asset → PS_AI_ConvAgent_EmotionPoseMap),
* assign your emotion AnimSequences, then reference this asset
* on the PS AI Agent Facial Expression component.
* on the PS AI ConvAgent Facial Expression component.
*
* The component plays the AnimSequence in real-time (looping) to drive
* emotion-based facial expressions (eyes, eyebrows, cheeks, mouth mood).
* Lip sync overrides the mouth-area curves on top.
*/
UCLASS(BlueprintType, Blueprintable, DisplayName = "PS AI Agent Emotion Pose Map")
class PS_AI_CONVAGENT_API UPS_AI_Agent_EmotionPoseMap : public UPrimaryDataAsset
UCLASS(BlueprintType, Blueprintable, DisplayName = "PS AI ConvAgent Emotion Pose Map")
class PS_AI_CONVAGENT_API UPS_AI_ConvAgent_EmotionPoseMap : public UPrimaryDataAsset
{
GENERATED_BODY()
@ -57,5 +57,5 @@ public:
* Neutral is recommended — it plays by default at startup (blinking, breathing). */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "Emotion Poses",
meta = (ToolTip = "Emotion → AnimSequence mapping with 3 intensity levels.\nThese drive the base facial expression (eyes, brows, cheeks).\nLip sync overrides the mouth area on top."))
TMap<EPS_AI_Agent_Emotion, FPS_AI_Agent_EmotionPoseSet> EmotionPoses;
TMap<EPS_AI_ConvAgent_Emotion, FPS_AI_ConvAgent_EmotionPoseSet> EmotionPoses;
};

44
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_Agent_FacialExpressionComponent.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_FacialExpressionComponent.h
View File

@ -4,18 +4,18 @@
#include "CoreMinimal.h"
#include "Components/ActorComponent.h"
#include "PS_AI_Agent_Definitions.h"
#include "PS_AI_Agent_FacialExpressionComponent.generated.h"
#include "PS_AI_ConvAgent_Definitions.h"
#include "PS_AI_ConvAgent_FacialExpressionComponent.generated.h"
class UPS_AI_Agent_Conv_ElevenLabsComponent;
class UPS_AI_Agent_EmotionPoseMap;
class UPS_AI_ConvAgent_ElevenLabsComponent;
class UPS_AI_ConvAgent_EmotionPoseMap;
class UAnimSequence;
// ─────────────────────────────────────────────────────────────────────────────
// UPS_AI_Agent_FacialExpressionComponent
// UPS_AI_ConvAgent_FacialExpressionComponent
//
// Drives emotion-based facial expressions on a MetaHuman (or any skeletal mesh)
// as a BASE layer. Lip sync (from PS_AI_Agent_LipSyncComponent) modulates on top,
// as a BASE layer. Lip sync (from PS_AI_ConvAgent_LipSyncComponent) modulates on top,
// overriding only mouth-area curves during speech.
//
// Emotion AnimSequences are played back in real-time (looping), not sampled
@ -24,28 +24,28 @@ class UAnimSequence;
//
// Workflow:
// 1. Assign a PoseMap data asset with Emotion Poses filled in.
// 2. Add the AnimNode "PS AI Agent Facial Expression" in the Face AnimBP
// BEFORE the "PS AI Agent Lip Sync" node.
// 2. Add the AnimNode "PS AI ConvAgent Facial Expression" in the Face AnimBP
// BEFORE the "PS AI ConvAgent Lip Sync" node.
// 3. The component listens to OnAgentEmotionChanged from the agent component.
// 4. Emotion animations crossfade smoothly (configurable duration).
// 5. The AnimNode reads GetCurrentEmotionCurves() and injects them into the pose.
// ─────────────────────────────────────────────────────────────────────────────
UCLASS(ClassGroup = "PS AI Agent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI Agent Facial Expression")
class PS_AI_CONVAGENT_API UPS_AI_Agent_FacialExpressionComponent : public UActorComponent
UCLASS(ClassGroup = "PS AI ConvAgent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI ConvAgent Facial Expression")
class PS_AI_CONVAGENT_API UPS_AI_ConvAgent_FacialExpressionComponent : public UActorComponent
{
GENERATED_BODY()
public:
UPS_AI_Agent_FacialExpressionComponent();
UPS_AI_ConvAgent_FacialExpressionComponent();
// ── Configuration ─────────────────────────────────────────────────────────
/** Emotion pose map asset containing emotion AnimSequences (Normal / Medium / Extreme per emotion).
* Create a dedicated PS_AI_Agent_EmotionPoseMap asset in the Content Browser. */
* Create a dedicated PS_AI_ConvAgent_EmotionPoseMap asset in the Content Browser. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS_AI_Agent|FacialExpression",
meta = (ToolTip = "Dedicated Emotion Pose Map asset.\nRight-click Content Browser → Miscellaneous → PS AI Agent Emotion Pose Map."))
TObjectPtr<UPS_AI_Agent_EmotionPoseMap> EmotionPoseMap;
meta = (ToolTip = "Dedicated Emotion Pose Map asset.\nRight-click Content Browser → Miscellaneous → PS AI ConvAgent Emotion Pose Map."))
TObjectPtr<UPS_AI_ConvAgent_EmotionPoseMap> EmotionPoseMap;
/** Emotion crossfade duration in seconds. */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS_AI_Agent|FacialExpression",
@ -61,11 +61,11 @@ public:
/** Get the active emotion. */
UFUNCTION(BlueprintPure, Category = "PS_AI_Agent|FacialExpression")
EPS_AI_Agent_Emotion GetActiveEmotion() const { return ActiveEmotion; }
EPS_AI_ConvAgent_Emotion GetActiveEmotion() const { return ActiveEmotion; }
/** Get the active emotion intensity. */
UFUNCTION(BlueprintPure, Category = "PS_AI_Agent|FacialExpression")
EPS_AI_Agent_EmotionIntensity GetActiveIntensity() const { return ActiveEmotionIntensity; }
EPS_AI_ConvAgent_EmotionIntensity GetActiveIntensity() const { return ActiveEmotionIntensity; }
/** Check if a curve name belongs to the mouth area (overridden by lip sync). */
UFUNCTION(BlueprintPure, Category = "PS_AI_Agent|FacialExpression")
@ -82,7 +82,7 @@ private:
/** Called when the agent changes emotion via client tool. */
UFUNCTION()
void OnEmotionChanged(EPS_AI_Agent_Emotion Emotion, EPS_AI_Agent_EmotionIntensity Intensity);
void OnEmotionChanged(EPS_AI_ConvAgent_Emotion Emotion, EPS_AI_ConvAgent_EmotionIntensity Intensity);
// ── Helpers ───────────────────────────────────────────────────────────────
@ -90,7 +90,7 @@ private:
void ValidateEmotionPoses();
/** Find the best AnimSequence for a given emotion + intensity (with fallback). */
UAnimSequence* FindAnimForEmotion(EPS_AI_Agent_Emotion Emotion, EPS_AI_Agent_EmotionIntensity Intensity) const;
UAnimSequence* FindAnimForEmotion(EPS_AI_ConvAgent_Emotion Emotion, EPS_AI_ConvAgent_EmotionIntensity Intensity) const;
/** Evaluate all FloatCurves from an AnimSequence at a given time. */
TMap<FName, float> EvaluateAnimCurves(UAnimSequence* AnimSeq, float Time) const;
@ -118,9 +118,9 @@ private:
TMap<FName, float> CurrentEmotionCurves;
/** Active emotion (for change detection). */
EPS_AI_Agent_Emotion ActiveEmotion = EPS_AI_Agent_Emotion::Neutral;
EPS_AI_Agent_EmotionIntensity ActiveEmotionIntensity = EPS_AI_Agent_EmotionIntensity::Medium;
EPS_AI_ConvAgent_Emotion ActiveEmotion = EPS_AI_ConvAgent_Emotion::Neutral;
EPS_AI_ConvAgent_EmotionIntensity ActiveEmotionIntensity = EPS_AI_ConvAgent_EmotionIntensity::Medium;
/** Cached reference to the agent component on the same Actor. */
TWeakObjectPtr<UPS_AI_Agent_Conv_ElevenLabsComponent> AgentComponent;
TWeakObjectPtr<UPS_AI_ConvAgent_ElevenLabsComponent> AgentComponent;
};

32
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_Agent_LipSyncComponent.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_LipSyncComponent.h
View File

@ -5,11 +5,11 @@
#include "CoreMinimal.h"
#include "Components/ActorComponent.h"
#include "DSP/SpectrumAnalyzer.h"
#include "PS_AI_Agent_LipSyncComponent.generated.h"
#include "PS_AI_ConvAgent_LipSyncComponent.generated.h"
class UPS_AI_Agent_Conv_ElevenLabsComponent;
class UPS_AI_Agent_FacialExpressionComponent;
class UPS_AI_Agent_LipSyncPoseMap;
class UPS_AI_ConvAgent_ElevenLabsComponent;
class UPS_AI_ConvAgent_FacialExpressionComponent;
class UPS_AI_ConvAgent_LipSyncPoseMap;
class USkeletalMeshComponent;
/** A single entry in the decoupled viseme timeline.
@ -22,7 +22,7 @@ struct FVisemeTimelineEntry
};
// Fired every tick when viseme/blendshape data has been updated.
DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnPS_AI_Agent_VisemesReady);
DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnPS_AI_ConvAgent_VisemesReady);
/**
* Real-time lip sync component for conversational AI agents.
@ -38,15 +38,15 @@ DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnPS_AI_Agent_VisemesReady);
* 3. Conversation starts → lip sync works automatically.
* 4. (Optional) Bind OnVisemesReady for custom Blueprint handling.
*/
UCLASS(ClassGroup = "PS AI Agent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI Agent Lip Sync")
class PS_AI_CONVAGENT_API UPS_AI_Agent_LipSyncComponent : public UActorComponent
UCLASS(ClassGroup = "PS AI ConvAgent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI ConvAgent Lip Sync")
class PS_AI_CONVAGENT_API UPS_AI_ConvAgent_LipSyncComponent : public UActorComponent
{
GENERATED_BODY()
public:
UPS_AI_Agent_LipSyncComponent();
~UPS_AI_Agent_LipSyncComponent();
UPS_AI_ConvAgent_LipSyncComponent();
~UPS_AI_ConvAgent_LipSyncComponent();
// ── Configuration ─────────────────────────────────────────────────────────
@ -78,7 +78,7 @@ public:
// ── Emotion Expression Blend ─────────────────────────────────────────────
/** How much facial emotion (from PS_AI_Agent_FacialExpressionComponent) bleeds through
/** How much facial emotion (from PS_AI_ConvAgent_FacialExpressionComponent) bleeds through
* during speech on expression curves (smile, frown, sneer, dimple, etc.).
* Shape curves (jaw, tongue, lip shape) are always 100% lip sync.
* 0.0 = lip sync completely overrides expression curves (old behavior).
@ -110,14 +110,14 @@ public:
// ── Phoneme Pose Map ─────────────────────────────────────────────────────
/** Optional pose map asset mapping OVR visemes to phoneme AnimSequences.
* Create once (right-click → Miscellaneous → Data Asset → PS_AI_Agent_LipSyncPoseMap),
* Create once (right-click → Miscellaneous → Data Asset → PS_AI_ConvAgent_LipSyncPoseMap),
* assign your MHF_* poses, then reference it on every MetaHuman.
* When assigned, curve data is extracted at BeginPlay and replaces the
* hardcoded ARKit blendshape mapping with artist-crafted poses.
* Leave empty to use the built-in ARKit mapping (backward compatible). */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS_AI_Agent|LipSync",
meta = (ToolTip = "Reusable pose map asset.\nAssign your MHF_* AnimSequences once, reuse on all MetaHumans.\nLeave empty for built-in ARKit mapping."))
TObjectPtr<UPS_AI_Agent_LipSyncPoseMap> PoseMap;
TObjectPtr<UPS_AI_ConvAgent_LipSyncPoseMap> PoseMap;
// ── Events ────────────────────────────────────────────────────────────────
@ -125,7 +125,7 @@ public:
* Use GetCurrentVisemes() or GetCurrentBlendshapes() to read values. */
UPROPERTY(BlueprintAssignable, Category = "PS_AI_Agent|LipSync",
meta = (ToolTip = "Fires each frame with updated viseme data.\nCall GetCurrentVisemes() or GetCurrentBlendshapes() to read values."))
FOnPS_AI_Agent_VisemesReady OnVisemesReady;
FOnPS_AI_ConvAgent_VisemesReady OnVisemesReady;
// ── Getters ───────────────────────────────────────────────────────────────
@ -285,11 +285,11 @@ private:
double WaitingForTextStartTime = 0.0;
// Cached reference to the agent component on the same Actor
TWeakObjectPtr<UPS_AI_Agent_Conv_ElevenLabsComponent> AgentComponent;
TWeakObjectPtr<UPS_AI_ConvAgent_ElevenLabsComponent> AgentComponent;
FDelegateHandle AudioDataHandle;
// Cached reference to the facial expression component for emotion blending
TWeakObjectPtr<UPS_AI_Agent_FacialExpressionComponent> CachedFacialExprComp;
TWeakObjectPtr<UPS_AI_ConvAgent_FacialExpressionComponent> CachedFacialExprComp;
// ── Pose-extracted curve data ────────────────────────────────────────────

10
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_Agent_LipSyncPoseMap.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_LipSyncPoseMap.h
View File

@ -5,7 +5,7 @@
#include "CoreMinimal.h"
#include "Engine/DataAsset.h"
#include "Engine/AssetManager.h"
#include "PS_AI_Agent_LipSyncPoseMap.generated.h"
#include "PS_AI_ConvAgent_LipSyncPoseMap.generated.h"
class UAnimSequence;
@ -13,16 +13,16 @@ class UAnimSequence;
* Reusable data asset that maps OVR visemes to phoneme pose AnimSequences.
*
* Create ONE instance of this asset in the Content Browser
* (right-click → Miscellaneous → Data Asset → PS_AI_Agent_LipSyncPoseMap),
* (right-click → Miscellaneous → Data Asset → PS_AI_ConvAgent_LipSyncPoseMap),
* assign your MHF_* AnimSequences once, then reference this single asset
* on every MetaHuman's PS AI Agent Lip Sync component.
* on every MetaHuman's PS AI ConvAgent Lip Sync component.
*
* The component extracts curve data from each pose at BeginPlay and uses it
* to drive lip sync — replacing the hardcoded ARKit blendshape mapping with
* artist-crafted poses that coordinate dozens of facial curves.
*/
UCLASS(BlueprintType, Blueprintable, DisplayName = "PS AI Agent Lip Sync Pose Map")
class PS_AI_CONVAGENT_API UPS_AI_Agent_LipSyncPoseMap : public UPrimaryDataAsset
UCLASS(BlueprintType, Blueprintable, DisplayName = "PS AI ConvAgent Lip Sync Pose Map")
class PS_AI_CONVAGENT_API UPS_AI_ConvAgent_LipSyncPoseMap : public UPrimaryDataAsset
{
GENERATED_BODY()

16
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_Agent_MicrophoneCaptureComponent.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_MicrophoneCaptureComponent.h
View File

@ -6,27 +6,27 @@
#include "Components/ActorComponent.h"
#include "AudioCapture.h"
#include <atomic>
#include "PS_AI_Agent_MicrophoneCaptureComponent.generated.h"
#include "PS_AI_ConvAgent_MicrophoneCaptureComponent.generated.h"
// Delivers captured float PCM samples (16000 Hz mono, resampled from device rate).
DECLARE_MULTICAST_DELEGATE_OneParam(FOnPS_AI_Agent_AudioCaptured, const TArray<float>& /*FloatPCM*/);
DECLARE_MULTICAST_DELEGATE_OneParam(FOnPS_AI_ConvAgent_AudioCaptured, const TArray<float>& /*FloatPCM*/);
/**
* Lightweight microphone capture component.
* Captures from the default audio input device, resamples to 16000 Hz mono,
* and delivers chunks via FOnPS_AI_Agent_AudioCaptured.
* and delivers chunks via FOnPS_AI_ConvAgent_AudioCaptured.
*
* Modelled after Convai's ConvaiAudioCaptureComponent but stripped to the
* minimal functionality needed for conversational AI agents.
*/
UCLASS(ClassGroup = "PS AI Agent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI Agent Microphone Capture")
class PS_AI_CONVAGENT_API UPS_AI_Agent_MicrophoneCaptureComponent : public UActorComponent
UCLASS(ClassGroup = "PS AI ConvAgent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI ConvAgent Microphone Capture")
class PS_AI_CONVAGENT_API UPS_AI_ConvAgent_MicrophoneCaptureComponent : public UActorComponent
{
GENERATED_BODY()
public:
UPS_AI_Agent_MicrophoneCaptureComponent();
UPS_AI_ConvAgent_MicrophoneCaptureComponent();
/** Multiplier applied to the microphone input volume before sending to the agent. Increase if the agent has trouble hearing you, decrease if your audio is clipping. Default: 1.0 (no change). */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS_AI_Agent|MicCapture",
@ -40,7 +40,7 @@ public:
* Audio is captured on a WASAPI background thread, resampled there (with
* echo suppression), then dispatched to the game thread for this broadcast.
*/
FOnPS_AI_Agent_AudioCaptured OnAudioCaptured;
FOnPS_AI_ConvAgent_AudioCaptured OnAudioCaptured;
/** Optional pointer to an atomic bool that suppresses capture when true.
* Set by the agent component for echo suppression (skip mic while agent speaks). */

24
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_Agent_PostureComponent.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_PostureComponent.h
View File

@ -5,18 +5,18 @@
#include "CoreMinimal.h"
#include "Components/ActorComponent.h"
#include "HAL/CriticalSection.h"
#include "PS_AI_Agent_PostureComponent.generated.h"
#include "PS_AI_ConvAgent_PostureComponent.generated.h"
class USkeletalMeshComponent;
DECLARE_LOG_CATEGORY_EXTERN(LogPS_AI_Agent_Posture, Log, All);
DECLARE_LOG_CATEGORY_EXTERN(LogPS_AI_ConvAgent_Posture, Log, All);
// ─────────────────────────────────────────────────────────────────────────────
// Neck bone chain entry for distributing head rotation across multiple bones
// ─────────────────────────────────────────────────────────────────────────────
USTRUCT(BlueprintType)
struct PS_AI_CONVAGENT_API FPS_AI_Agent_NeckBoneEntry
struct PS_AI_CONVAGENT_API FPS_AI_ConvAgent_NeckBoneEntry
{
GENERATED_BODY()
@ -31,7 +31,7 @@ struct PS_AI_CONVAGENT_API FPS_AI_Agent_NeckBoneEntry
};
// ─────────────────────────────────────────────────────────────────────────────
// UPS_AI_Agent_PostureComponent
// UPS_AI_ConvAgent_PostureComponent
//
// Chase-based multi-layer look-at system for MetaHuman characters.
// Smoothly orients the character's body, head, and eyes toward a TargetActor.
@ -47,23 +47,23 @@ struct PS_AI_CONVAGENT_API FPS_AI_Agent_NeckBoneEntry
//
// Workflow:
// 1. Add this component to the character Blueprint.
// 2. Add the AnimNode "PS AI Agent Posture" in the Body AnimBP
// 2. Add the AnimNode "PS AI ConvAgent Posture" in the Body AnimBP
// with bApplyHeadRotation = true, bApplyEyeCurves = false.
// 3. Add the AnimNode "PS AI Agent Posture" in the Face AnimBP
// 3. Add the AnimNode "PS AI ConvAgent Posture" in the Face AnimBP
// with bApplyHeadRotation = false, bApplyEyeCurves = true
// (between "Facial Expression" and "Lip Sync" nodes).
// 4. Set TargetActor to any actor (player pawn, a prop, etc.).
// 5. Set TargetOffset for actors without a skeleton (e.g. (0,0,160) for
// eye-level on a simple actor).
// ─────────────────────────────────────────────────────────────────────────────
UCLASS(ClassGroup = "PS AI Agent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI Agent Posture")
class PS_AI_CONVAGENT_API UPS_AI_Agent_PostureComponent : public UActorComponent
UCLASS(ClassGroup = "PS AI ConvAgent", meta = (BlueprintSpawnableComponent),
DisplayName = "PS AI ConvAgent Posture")
class PS_AI_CONVAGENT_API UPS_AI_ConvAgent_PostureComponent : public UActorComponent
{
GENERATED_BODY()
public:
UPS_AI_Agent_PostureComponent();
UPS_AI_ConvAgent_PostureComponent();
// ── Target ───────────────────────────────────────────────────────────────
@ -194,7 +194,7 @@ public:
* Weights should sum to ~1.0.
* If empty, falls back to single-bone behavior (HeadBoneName, weight 1.0). */
UPROPERTY(EditAnywhere, BlueprintReadWrite, Category = "PS_AI_Agent|Posture")
TArray<FPS_AI_Agent_NeckBoneEntry> NeckBoneChain;
TArray<FPS_AI_ConvAgent_NeckBoneEntry> NeckBoneChain;
// ── Getters (read by AnimNode) ───────────────────────────────────────────
@ -220,7 +220,7 @@ public:
FName GetHeadBoneName() const { return HeadBoneName; }
/** Get the neck bone chain (used by AnimNode to resolve bone indices). */
const TArray<FPS_AI_Agent_NeckBoneEntry>& GetNeckBoneChain() const { return NeckBoneChain; }
const TArray<FPS_AI_ConvAgent_NeckBoneEntry>& GetNeckBoneChain() const { return NeckBoneChain; }
/** Get head animation compensation factor (0 = additive, 1 = full override). */
float GetHeadAnimationCompensation() const { return HeadAnimationCompensation; }

110
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_Agent_WebSocket_ElevenLabsProxy.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_WebSocket_ElevenLabsProxy.h
View File

@ -4,63 +4,63 @@
#include "CoreMinimal.h"
#include "UObject/NoExportTypes.h"
#include "PS_AI_Agent_Definitions.h"
#include "PS_AI_ConvAgent_Definitions.h"
#include "IWebSocket.h"
#include "PS_AI_Agent_WebSocket_ElevenLabsProxy.generated.h"
#include "PS_AI_ConvAgent_WebSocket_ElevenLabsProxy.generated.h"
// ─────────────────────────────────────────────────────────────────────────────
// Delegates (all Blueprint-assignable)
// ─────────────────────────────────────────────────────────────────────────────
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_Agent_ElevenLabs_Connected,
const FPS_AI_Agent_ConversationInfo_ElevenLabs&, ConversationInfo);
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_ConvAgent_ElevenLabs_Connected,
const FPS_AI_ConvAgent_ConversationInfo_ElevenLabs&, ConversationInfo);
DECLARE_DYNAMIC_MULTICAST_DELEGATE_TwoParams(FOnPS_AI_Agent_ElevenLabs_Disconnected,
DECLARE_DYNAMIC_MULTICAST_DELEGATE_TwoParams(FOnPS_AI_ConvAgent_ElevenLabs_Disconnected,
int32, StatusCode, const FString&, Reason);
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_Agent_ElevenLabs_Error,
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_ConvAgent_ElevenLabs_Error,
const FString&, ErrorMessage);
/** Fired when a PCM audio chunk arrives from the agent. Raw bytes, 16-bit signed 16kHz mono. */
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_Agent_ElevenLabs_AudioReceived,
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_ConvAgent_ElevenLabs_AudioReceived,
const TArray<uint8>&, PCMData);
/** Fired for user or agent transcript segments. */
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_Agent_ElevenLabs_Transcript,
const FPS_AI_Agent_TranscriptSegment_ElevenLabs&, Segment);
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_ConvAgent_ElevenLabs_Transcript,
const FPS_AI_ConvAgent_TranscriptSegment_ElevenLabs&, Segment);
/** Fired with the final text response from the agent. */
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_Agent_ElevenLabs_AgentResponse,
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_ConvAgent_ElevenLabs_AgentResponse,
const FString&, ResponseText);
/** Fired when the agent interrupts the user. */
DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnPS_AI_Agent_ElevenLabs_Interrupted);
DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnPS_AI_ConvAgent_ElevenLabs_Interrupted);
/**
* Fired when the server starts generating a response (first agent_chat_response_part received).
* This fires BEFORE audio arrives — useful to detect that the server is processing
* the previous turn while the client may have restarted listening (auto-restart scenario).
*/
DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnPS_AI_Agent_ElevenLabs_ResponseStarted);
DECLARE_DYNAMIC_MULTICAST_DELEGATE(FOnPS_AI_ConvAgent_ElevenLabs_ResponseStarted);
/** Fired for every agent_chat_response_part — streams the LLM text as it is generated.
* PartialText is the text fragment from this individual part (NOT accumulated). */
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_Agent_ElevenLabs_ResponsePart,
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_ConvAgent_ElevenLabs_ResponsePart,
const FString&, PartialText);
/** Fired when the server sends a client_tool_call — the agent wants the client to execute a tool. */
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_Agent_ElevenLabs_ClientToolCall,
const FPS_AI_Agent_ClientToolCall_ElevenLabs&, ToolCall);
DECLARE_DYNAMIC_MULTICAST_DELEGATE_OneParam(FOnPS_AI_ConvAgent_ElevenLabs_ClientToolCall,
const FPS_AI_ConvAgent_ClientToolCall_ElevenLabs&, ToolCall);
// ─────────────────────────────────────────────────────────────────────────────
// WebSocket Proxy
// Manages the lifecycle of a single ElevenLabs Conversational AI WebSocket session.
// Instantiate via UPS_AI_Agent_Conv_ElevenLabsComponent (the component manages
// Instantiate via UPS_AI_ConvAgent_ElevenLabsComponent (the component manages
// one proxy at a time), or create manually through Blueprints.
// ─────────────────────────────────────────────────────────────────────────────
UCLASS(BlueprintType, Blueprintable)
class PS_AI_CONVAGENT_API UPS_AI_Agent_WebSocket_ElevenLabsProxy : public UObject
class PS_AI_CONVAGENT_API UPS_AI_ConvAgent_WebSocket_ElevenLabsProxy : public UObject
{
GENERATED_BODY()
@ -68,48 +68,48 @@ public:
// ── Events ────────────────────────────────────────────────────────────────
/** Called once the WebSocket handshake succeeds and the agent sends its initiation metadata. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_Connected OnConnected;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_Connected OnConnected;
/** Called when the WebSocket closes (graceful or remote). */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_Disconnected OnDisconnected;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_Disconnected OnDisconnected;
/** Called on any connection or protocol error. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_Error OnError;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_Error OnError;
/** Raw PCM audio coming from the agent — feed this into your audio component. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_AudioReceived OnAudioReceived;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_AudioReceived OnAudioReceived;
/** User or agent transcript (may be tentative while the conversation is ongoing). */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_Transcript OnTranscript;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_Transcript OnTranscript;
/** Final text response from the agent (complements audio). */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_AgentResponse OnAgentResponse;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_AgentResponse OnAgentResponse;
/** The agent was interrupted by new user speech. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_Interrupted OnInterrupted;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_Interrupted OnInterrupted;
/**
* Fired on the first agent_chat_response_part per turn — i.e. the moment the server
* starts generating. Fires well before audio. The component uses this to stop the
* microphone if it was restarted before the server finished processing the previous turn.
*/
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_ResponseStarted OnAgentResponseStarted;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_ResponseStarted OnAgentResponseStarted;
/** Fired for every agent_chat_response_part with the streaming text fragment. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_ResponsePart OnAgentResponsePart;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_ResponsePart OnAgentResponsePart;
/** Fired when the agent invokes a client tool. Handle the call and reply with SendClientToolResult. */
UPROPERTY(BlueprintAssignable, Category = "PS AI Agent|ElevenLabs|Events")
FOnPS_AI_Agent_ElevenLabs_ClientToolCall OnClientToolCall;
UPROPERTY(BlueprintAssignable, Category = "PS AI ConvAgent|ElevenLabs|Events")
FOnPS_AI_ConvAgent_ElevenLabs_ClientToolCall OnClientToolCall;
// ── Lifecycle ─────────────────────────────────────────────────────────────
@ -120,22 +120,22 @@ public:
* @param AgentID ElevenLabs agent ID. Overrides the project-level default when non-empty.
* @param APIKey API key. Overrides the project-level default when non-empty.
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void Connect(const FString& AgentID = TEXT(""), const FString& APIKey = TEXT(""));
/**
* Gracefully close the WebSocket connection.
* OnDisconnected will fire after the server acknowledges.
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void Disconnect();
/** Current connection state. */
UFUNCTION(BlueprintPure, Category = "PS AI Agent|ElevenLabs")
EPS_AI_Agent_ConnectionState_ElevenLabs GetConnectionState() const { return ConnectionState; }
UFUNCTION(BlueprintPure, Category = "PS AI ConvAgent|ElevenLabs")
EPS_AI_ConvAgent_ConnectionState_ElevenLabs GetConnectionState() const { return ConnectionState; }
UFUNCTION(BlueprintPure, Category = "PS AI Agent|ElevenLabs")
bool IsConnected() const { return ConnectionState == EPS_AI_Agent_ConnectionState_ElevenLabs::Connected; }
UFUNCTION(BlueprintPure, Category = "PS AI ConvAgent|ElevenLabs")
bool IsConnected() const { return ConnectionState == EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Connected; }
// ── Audio sending ─────────────────────────────────────────────────────────
@ -147,7 +147,7 @@ public:
*
* @param PCMData Raw PCM bytes (16-bit LE, 16kHz, mono).
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void SendAudioChunk(const TArray<uint8>& PCMData);
// ── Turn control (only relevant in Client turn mode) ──────────────────────
@ -157,7 +157,7 @@ public:
* Sends a { "type": "user_activity" } message to the server.
* Call this periodically while the user is speaking (e.g. every audio chunk).
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void SendUserTurnStart();
/**
@ -165,7 +165,7 @@ public:
* No explicit API message — simply stop sending user_activity.
* The server detects silence and hands the turn to the agent.
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void SendUserTurnEnd();
/**
@ -173,11 +173,11 @@ public:
* Useful for testing or text-only interaction.
* Sends: { "type": "user_message", "text": "..." }
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void SendTextMessage(const FString& Text);
/** Ask the agent to stop the current utterance. */
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void SendInterrupt();
/**
@ -188,13 +188,13 @@ public:
* @param Result A string result to return to the agent.
* @param bIsError True if the tool execution failed.
*/
UFUNCTION(BlueprintCallable, Category = "PS AI Agent|ElevenLabs")
UFUNCTION(BlueprintCallable, Category = "PS AI ConvAgent|ElevenLabs")
void SendClientToolResult(const FString& ToolCallId, const FString& Result, bool bIsError = false);
// ── Info ──────────────────────────────────────────────────────────────────
UFUNCTION(BlueprintPure, Category = "PS AI Agent|ElevenLabs")
const FPS_AI_Agent_ConversationInfo_ElevenLabs& GetConversationInfo() const { return ConversationInfo; }
UFUNCTION(BlueprintPure, Category = "PS AI ConvAgent|ElevenLabs")
const FPS_AI_ConvAgent_ConversationInfo_ElevenLabs& GetConversationInfo() const { return ConversationInfo; }
// ─────────────────────────────────────────────────────────────────────────
// Internal
@ -222,8 +222,8 @@ private:
FString BuildWebSocketURL(const FString& AgentID, const FString& APIKey) const;
TSharedPtr<IWebSocket> WebSocket;
EPS_AI_Agent_ConnectionState_ElevenLabs ConnectionState = EPS_AI_Agent_ConnectionState_ElevenLabs::Disconnected;
FPS_AI_Agent_ConversationInfo_ElevenLabs ConversationInfo;
EPS_AI_ConvAgent_ConnectionState_ElevenLabs ConnectionState = EPS_AI_ConvAgent_ConnectionState_ElevenLabs::Disconnected;
FPS_AI_ConvAgent_ConversationInfo_ElevenLabs ConversationInfo;
// Serializes WebSocket->Send() calls — needed because SendAudioChunk can now be
// called from the WASAPI background thread while SendJsonMessage runs on game thread.
@ -258,9 +258,9 @@ private:
int32 LastInterruptEventId = 0;
public:
// Set by UPS_AI_Agent_Conv_ElevenLabsComponent before calling Connect().
// Set by UPS_AI_ConvAgent_ElevenLabsComponent before calling Connect().
// Controls turn_timeout in conversation_initiation_client_data.
EPS_AI_Agent_TurnMode_ElevenLabs TurnMode = EPS_AI_Agent_TurnMode_ElevenLabs::Server;
EPS_AI_ConvAgent_TurnMode_ElevenLabs TurnMode = EPS_AI_ConvAgent_TurnMode_ElevenLabs::Server;
// Speculative turn: start LLM generation during silence before full turn confidence.
bool bSpeculativeTurn = true;

2
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/PS_AI_ConvAgentEditor.Build.cs
View File

@ -19,7 +19,7 @@ public class PS_AI_ConvAgentEditor : ModuleRules
// AnimGraph editor node base class
"AnimGraph",
"BlueprintGraph",
// Runtime module containing FAnimNode_PS_AI_Agent_LipSync
// Runtime module containing FAnimNode_PS_AI_ConvAgent_LipSync
"PS_AI_ConvAgent",
});
}

32
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/AnimGraphNode_PS_AI_Agent_FacialExpression.cpp
View File

@ -1,32 +0,0 @@
// Copyright ASTERION. All Rights Reserved.
#include "AnimGraphNode_PS_AI_Agent_FacialExpression.h"
#define LOCTEXT_NAMESPACE "AnimNode_PS_AI_Agent_FacialExpression"
FText UAnimGraphNode_PS_AI_Agent_FacialExpression::GetNodeTitle(ENodeTitleType::Type TitleType) const
{
return LOCTEXT("NodeTitle", "PS AI Agent Facial Expression");
}
FText UAnimGraphNode_PS_AI_Agent_FacialExpression::GetTooltipText() const
{
return LOCTEXT("Tooltip",
"Injects emotion expression curves from the PS AI Agent Facial Expression component.\n\n"
"Place this node BEFORE the PS AI Agent Lip Sync node in the MetaHuman Face AnimBP.\n"
"It outputs CTRL_expressions_* curves for eyes, eyebrows, cheeks, and mouth mood.\n"
"The Lip Sync node placed after will override mouth-area curves during speech.");
}
FString UAnimGraphNode_PS_AI_Agent_FacialExpression::GetNodeCategory() const
{
return TEXT("PS AI Agent");
}
FLinearColor UAnimGraphNode_PS_AI_Agent_FacialExpression::GetNodeTitleColor() const
{
// Warm amber to distinguish from Lip Sync (teal)
return FLinearColor(0.8f, 0.5f, 0.1f, 1.0f);
}
#undef LOCTEXT_NAMESPACE

32
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/AnimGraphNode_PS_AI_Agent_LipSync.cpp
View File

@ -1,32 +0,0 @@
// Copyright ASTERION. All Rights Reserved.
#include "AnimGraphNode_PS_AI_Agent_LipSync.h"
#define LOCTEXT_NAMESPACE "AnimNode_PS_AI_Agent_LipSync"
FText UAnimGraphNode_PS_AI_Agent_LipSync::GetNodeTitle(ENodeTitleType::Type TitleType) const
{
return LOCTEXT("NodeTitle", "PS AI Agent Lip Sync");
}
FText UAnimGraphNode_PS_AI_Agent_LipSync::GetTooltipText() const
{
return LOCTEXT("Tooltip",
"Injects lip sync animation curves from the PS AI Agent Lip Sync component.\n\n"
"Place this node BEFORE the Control Rig in the MetaHuman Face AnimBP.\n"
"It auto-discovers the component and outputs CTRL_expressions_* curves\n"
"that the MetaHuman Control Rig uses to drive facial bones.");
}
FString UAnimGraphNode_PS_AI_Agent_LipSync::GetNodeCategory() const
{
return TEXT("PS AI Agent");
}
FLinearColor UAnimGraphNode_PS_AI_Agent_LipSync::GetNodeTitleColor() const
{
// PS AI Agent teal color
return FLinearColor(0.1f, 0.7f, 0.6f, 1.0f);
}
#undef LOCTEXT_NAMESPACE

33
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/AnimGraphNode_PS_AI_Agent_Posture.cpp
View File

@ -1,33 +0,0 @@
// Copyright ASTERION. All Rights Reserved.
#include "AnimGraphNode_PS_AI_Agent_Posture.h"
#define LOCTEXT_NAMESPACE "AnimNode_PS_AI_Agent_Posture"
FText UAnimGraphNode_PS_AI_Agent_Posture::GetNodeTitle(ENodeTitleType::Type TitleType) const
{
return LOCTEXT("NodeTitle", "PS AI Agent Posture");
}
FText UAnimGraphNode_PS_AI_Agent_Posture::GetTooltipText() const
{
return LOCTEXT("Tooltip",
"Injects head rotation and eye gaze curves from the PS AI Agent Posture component.\n\n"
"Place this node AFTER the PS AI Agent Facial Expression node and\n"
"BEFORE the PS AI Agent Lip Sync node in the MetaHuman Face AnimBP.\n\n"
"The component distributes look-at rotation across body (actor yaw),\n"
"head (bone rotation), and eyes (ARKit curves) for a natural look-at effect.");
}
FString UAnimGraphNode_PS_AI_Agent_Posture::GetNodeCategory() const
{
return TEXT("PS AI Agent");
}
FLinearColor UAnimGraphNode_PS_AI_Agent_Posture::GetNodeTitleColor() const
{
// Cool blue to distinguish from Facial Expression (amber) and Lip Sync (teal)
return FLinearColor(0.2f, 0.4f, 0.9f, 1.0f);
}
#undef LOCTEXT_NAMESPACE

32
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/AnimGraphNode_PS_AI_ConvAgent_FacialExpression.cpp Normal file
View File

@ -0,0 +1,32 @@
// Copyright ASTERION. All Rights Reserved.
#include "AnimGraphNode_PS_AI_ConvAgent_FacialExpression.h"
#define LOCTEXT_NAMESPACE "AnimNode_PS_AI_ConvAgent_FacialExpression"
FText UAnimGraphNode_PS_AI_ConvAgent_FacialExpression::GetNodeTitle(ENodeTitleType::Type TitleType) const
{
return LOCTEXT("NodeTitle", "PS AI ConvAgent Facial Expression");
}
FText UAnimGraphNode_PS_AI_ConvAgent_FacialExpression::GetTooltipText() const
{
return LOCTEXT("Tooltip",
"Injects emotion expression curves from the PS AI ConvAgent Facial Expression component.\n\n"
"Place this node BEFORE the PS AI ConvAgent Lip Sync node in the MetaHuman Face AnimBP.\n"
"It outputs CTRL_expressions_* curves for eyes, eyebrows, cheeks, and mouth mood.\n"
"The Lip Sync node placed after will override mouth-area curves during speech.");
}
FString UAnimGraphNode_PS_AI_ConvAgent_FacialExpression::GetNodeCategory() const
{
return TEXT("PS AI ConvAgent");
}
FLinearColor UAnimGraphNode_PS_AI_ConvAgent_FacialExpression::GetNodeTitleColor() const
{
// Warm amber to distinguish from Lip Sync (teal)
return FLinearColor(0.8f, 0.5f, 0.1f, 1.0f);
}
#undef LOCTEXT_NAMESPACE

32
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/AnimGraphNode_PS_AI_ConvAgent_LipSync.cpp Normal file
View File

@ -0,0 +1,32 @@
// Copyright ASTERION. All Rights Reserved.
#include "AnimGraphNode_PS_AI_ConvAgent_LipSync.h"
#define LOCTEXT_NAMESPACE "AnimNode_PS_AI_ConvAgent_LipSync"
FText UAnimGraphNode_PS_AI_ConvAgent_LipSync::GetNodeTitle(ENodeTitleType::Type TitleType) const
{
return LOCTEXT("NodeTitle", "PS AI ConvAgent Lip Sync");
}
FText UAnimGraphNode_PS_AI_ConvAgent_LipSync::GetTooltipText() const
{
return LOCTEXT("Tooltip",
"Injects lip sync animation curves from the PS AI ConvAgent Lip Sync component.\n\n"
"Place this node BEFORE the Control Rig in the MetaHuman Face AnimBP.\n"
"It auto-discovers the component and outputs CTRL_expressions_* curves\n"
"that the MetaHuman Control Rig uses to drive facial bones.");
}
FString UAnimGraphNode_PS_AI_ConvAgent_LipSync::GetNodeCategory() const
{
return TEXT("PS AI ConvAgent");
}
FLinearColor UAnimGraphNode_PS_AI_ConvAgent_LipSync::GetNodeTitleColor() const
{
// PS AI ConvAgent teal color
return FLinearColor(0.1f, 0.7f, 0.6f, 1.0f);
}
#undef LOCTEXT_NAMESPACE

33
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/AnimGraphNode_PS_AI_ConvAgent_Posture.cpp Normal file
View File

@ -0,0 +1,33 @@
// Copyright ASTERION. All Rights Reserved.
#include "AnimGraphNode_PS_AI_ConvAgent_Posture.h"
#define LOCTEXT_NAMESPACE "AnimNode_PS_AI_ConvAgent_Posture"
FText UAnimGraphNode_PS_AI_ConvAgent_Posture::GetNodeTitle(ENodeTitleType::Type TitleType) const
{
return LOCTEXT("NodeTitle", "PS AI ConvAgent Posture");
}
FText UAnimGraphNode_PS_AI_ConvAgent_Posture::GetTooltipText() const
{
return LOCTEXT("Tooltip",
"Injects head rotation and eye gaze curves from the PS AI ConvAgent Posture component.\n\n"
"Place this node AFTER the PS AI ConvAgent Facial Expression node and\n"
"BEFORE the PS AI ConvAgent Lip Sync node in the MetaHuman Face AnimBP.\n\n"
"The component distributes look-at rotation across body (actor yaw),\n"
"head (bone rotation), and eyes (ARKit curves) for a natural look-at effect.");
}
FString UAnimGraphNode_PS_AI_ConvAgent_Posture::GetNodeCategory() const
{
return TEXT("PS AI ConvAgent");
}
FLinearColor UAnimGraphNode_PS_AI_ConvAgent_Posture::GetNodeTitleColor() const
{
// Cool blue to distinguish from Facial Expression (amber) and Lip Sync (teal)
return FLinearColor(0.2f, 0.4f, 0.9f, 1.0f);
}
#undef LOCTEXT_NAMESPACE

29
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/PS_AI_Agent_EmotionPoseMapFactory.cpp
View File

@ -1,29 +0,0 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_Agent_EmotionPoseMapFactory.h"
#include "PS_AI_Agent_EmotionPoseMap.h"
#include "AssetTypeCategories.h"
UPS_AI_Agent_EmotionPoseMapFactory::UPS_AI_Agent_EmotionPoseMapFactory()
{
SupportedClass = UPS_AI_Agent_EmotionPoseMap::StaticClass();
bCreateNew = true;
bEditAfterNew = true;
}
UObject* UPS_AI_Agent_EmotionPoseMapFactory::FactoryCreateNew(
UClass* Class, UObject* InParent, FName Name, EObjectFlags Flags,
UObject* Context, FFeedbackContext* Warn)
{
return NewObject<UPS_AI_Agent_EmotionPoseMap>(InParent, Class, Name, Flags);
}
FText UPS_AI_Agent_EmotionPoseMapFactory::GetDisplayName() const
{
return FText::FromString(TEXT("PS AI Agent Emotion Pose Map"));
}
uint32 UPS_AI_Agent_EmotionPoseMapFactory::GetMenuCategories() const
{
return EAssetTypeCategories::Misc;
}

29
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/PS_AI_Agent_LipSyncPoseMapFactory.cpp
View File

@ -1,29 +0,0 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_Agent_LipSyncPoseMapFactory.h"
#include "PS_AI_Agent_LipSyncPoseMap.h"
#include "AssetTypeCategories.h"
UPS_AI_Agent_LipSyncPoseMapFactory::UPS_AI_Agent_LipSyncPoseMapFactory()
{
SupportedClass = UPS_AI_Agent_LipSyncPoseMap::StaticClass();
bCreateNew = true;
bEditAfterNew = true;
}
UObject* UPS_AI_Agent_LipSyncPoseMapFactory::FactoryCreateNew(
UClass* Class, UObject* InParent, FName Name, EObjectFlags Flags,
UObject* Context, FFeedbackContext* Warn)
{
return NewObject<UPS_AI_Agent_LipSyncPoseMap>(InParent, Class, Name, Flags);
}
FText UPS_AI_Agent_LipSyncPoseMapFactory::GetDisplayName() const
{
return FText::FromString(TEXT("PS AI Agent Lip Sync Pose Map"));
}
uint32 UPS_AI_Agent_LipSyncPoseMapFactory::GetMenuCategories() const
{
return EAssetTypeCategories::Misc;
}

2
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/PS_AI_ConvAgentEditorModule.cpp
View File

@ -4,7 +4,7 @@
/**
* Editor module for PS_AI_ConvAgent plugin.
* Provides AnimGraph node(s) for the PS AI Agent Lip Sync system.
* Provides AnimGraph node(s) for the PS AI ConvAgent Lip Sync system.
*/
class FPS_AI_ConvAgentEditorModule : public IModuleInterface
{

29
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/PS_AI_ConvAgent_EmotionPoseMapFactory.cpp Normal file
View File

@ -0,0 +1,29 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_ConvAgent_EmotionPoseMapFactory.h"
#include "PS_AI_ConvAgent_EmotionPoseMap.h"
#include "AssetTypeCategories.h"
UPS_AI_ConvAgent_EmotionPoseMapFactory::UPS_AI_ConvAgent_EmotionPoseMapFactory()
{
SupportedClass = UPS_AI_ConvAgent_EmotionPoseMap::StaticClass();
bCreateNew = true;
bEditAfterNew = true;
}
UObject* UPS_AI_ConvAgent_EmotionPoseMapFactory::FactoryCreateNew(
UClass* Class, UObject* InParent, FName Name, EObjectFlags Flags,
UObject* Context, FFeedbackContext* Warn)
{
return NewObject<UPS_AI_ConvAgent_EmotionPoseMap>(InParent, Class, Name, Flags);
}
FText UPS_AI_ConvAgent_EmotionPoseMapFactory::GetDisplayName() const
{
return FText::FromString(TEXT("PS AI ConvAgent Emotion Pose Map"));
}
uint32 UPS_AI_ConvAgent_EmotionPoseMapFactory::GetMenuCategories() const
{
return EAssetTypeCategories::Misc;
}

8
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/PS_AI_Agent_LipSyncPoseMapFactory.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/PS_AI_ConvAgent_EmotionPoseMapFactory.h
View File

@ -4,19 +4,19 @@
#include "CoreMinimal.h"
#include "Factories/Factory.h"
#include "PS_AI_Agent_LipSyncPoseMapFactory.generated.h"
#include "PS_AI_ConvAgent_EmotionPoseMapFactory.generated.h"
/**
* Factory that lets users create PS_AI_Agent_LipSyncPoseMap assets
* Factory that lets users create PS_AI_ConvAgent_EmotionPoseMap assets
* directly from the Content Browser (right-click → Miscellaneous).
*/
UCLASS()
class UPS_AI_Agent_LipSyncPoseMapFactory : public UFactory
class UPS_AI_ConvAgent_EmotionPoseMapFactory : public UFactory
{
GENERATED_BODY()
public:
UPS_AI_Agent_LipSyncPoseMapFactory();
UPS_AI_ConvAgent_EmotionPoseMapFactory();
virtual UObject* FactoryCreateNew(UClass* Class, UObject* InParent,
FName Name, EObjectFlags Flags, UObject* Context,

29
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/PS_AI_ConvAgent_LipSyncPoseMapFactory.cpp Normal file
View File

@ -0,0 +1,29 @@
// Copyright ASTERION. All Rights Reserved.
#include "PS_AI_ConvAgent_LipSyncPoseMapFactory.h"
#include "PS_AI_ConvAgent_LipSyncPoseMap.h"
#include "AssetTypeCategories.h"
UPS_AI_ConvAgent_LipSyncPoseMapFactory::UPS_AI_ConvAgent_LipSyncPoseMapFactory()
{
SupportedClass = UPS_AI_ConvAgent_LipSyncPoseMap::StaticClass();
bCreateNew = true;
bEditAfterNew = true;
}
UObject* UPS_AI_ConvAgent_LipSyncPoseMapFactory::FactoryCreateNew(
UClass* Class, UObject* InParent, FName Name, EObjectFlags Flags,
UObject* Context, FFeedbackContext* Warn)
{
return NewObject<UPS_AI_ConvAgent_LipSyncPoseMap>(InParent, Class, Name, Flags);
}
FText UPS_AI_ConvAgent_LipSyncPoseMapFactory::GetDisplayName() const
{
return FText::FromString(TEXT("PS AI ConvAgent Lip Sync Pose Map"));
}
uint32 UPS_AI_ConvAgent_LipSyncPoseMapFactory::GetMenuCategories() const
{
return EAssetTypeCategories::Misc;
}

8
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/PS_AI_Agent_EmotionPoseMapFactory.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Private/PS_AI_ConvAgent_LipSyncPoseMapFactory.h
View File

@ -4,19 +4,19 @@
#include "CoreMinimal.h"
#include "Factories/Factory.h"
#include "PS_AI_Agent_EmotionPoseMapFactory.generated.h"
#include "PS_AI_ConvAgent_LipSyncPoseMapFactory.generated.h"
/**
* Factory that lets users create PS_AI_Agent_EmotionPoseMap assets
* Factory that lets users create PS_AI_ConvAgent_LipSyncPoseMap assets
* directly from the Content Browser (right-click → Miscellaneous).
*/
UCLASS()
class UPS_AI_Agent_EmotionPoseMapFactory : public UFactory
class UPS_AI_ConvAgent_LipSyncPoseMapFactory : public UFactory
{
GENERATED_BODY()
public:
UPS_AI_Agent_EmotionPoseMapFactory();
UPS_AI_ConvAgent_LipSyncPoseMapFactory();
virtual UObject* FactoryCreateNew(UClass* Class, UObject* InParent,
FName Name, EObjectFlags Flags, UObject* Context,

16
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Public/AnimGraphNode_PS_AI_Agent_FacialExpression.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Public/AnimGraphNode_PS_AI_ConvAgent_FacialExpression.h
View File

@ -4,24 +4,24 @@
#include "CoreMinimal.h"
#include "AnimGraphNode_Base.h"
#include "AnimNode_PS_AI_Agent_FacialExpression.h"
#include "AnimGraphNode_PS_AI_Agent_FacialExpression.generated.h"
#include "AnimNode_PS_AI_ConvAgent_FacialExpression.h"
#include "AnimGraphNode_PS_AI_ConvAgent_FacialExpression.generated.h"
/**
* AnimGraph editor node for the PS AI Agent Facial Expression AnimNode.
* AnimGraph editor node for the PS AI ConvAgent Facial Expression AnimNode.
*
* This node appears in the AnimBP graph editor under the "PS AI Agent" category.
* Place it BEFORE the PS AI Agent Lip Sync node in the MetaHuman Face AnimBP.
* It auto-discovers the PS_AI_Agent_FacialExpressionComponent on the owning Actor
* This node appears in the AnimBP graph editor under the "PS AI ConvAgent" category.
* Place it BEFORE the PS AI ConvAgent Lip Sync node in the MetaHuman Face AnimBP.
* It auto-discovers the PS_AI_ConvAgent_FacialExpressionComponent on the owning Actor
* and injects CTRL_expressions_* curves for emotion-driven facial expressions.
*/
UCLASS()
class UAnimGraphNode_PS_AI_Agent_FacialExpression : public UAnimGraphNode_Base
class UAnimGraphNode_PS_AI_ConvAgent_FacialExpression : public UAnimGraphNode_Base
{
GENERATED_BODY()
UPROPERTY(EditAnywhere, Category = "Settings")
FAnimNode_PS_AI_Agent_FacialExpression Node;
FAnimNode_PS_AI_ConvAgent_FacialExpression Node;
// UAnimGraphNode_Base interface
virtual FText GetNodeTitle(ENodeTitleType::Type TitleType) const override;

14
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Public/AnimGraphNode_PS_AI_Agent_LipSync.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Public/AnimGraphNode_PS_AI_ConvAgent_LipSync.h
View File

@ -4,24 +4,24 @@
#include "CoreMinimal.h"
#include "AnimGraphNode_Base.h"
#include "AnimNode_PS_AI_Agent_LipSync.h"
#include "AnimGraphNode_PS_AI_Agent_LipSync.generated.h"
#include "AnimNode_PS_AI_ConvAgent_LipSync.h"
#include "AnimGraphNode_PS_AI_ConvAgent_LipSync.generated.h"
/**
* AnimGraph editor node for the PS AI Agent Lip Sync AnimNode.
* AnimGraph editor node for the PS AI ConvAgent Lip Sync AnimNode.
*
* This node appears in the AnimBP graph editor under the "PS AI Agent" category.
* This node appears in the AnimBP graph editor under the "PS AI ConvAgent" category.
* Drop it into your MetaHuman Face AnimBP between the pose source and the
* Control Rig node. It auto-discovers the PS_AI_Agent_LipSyncComponent on the
* Control Rig node. It auto-discovers the PS_AI_ConvAgent_LipSyncComponent on the
* owning Actor and injects CTRL_expressions_* curves for lip sync.
*/
UCLASS()
class UAnimGraphNode_PS_AI_Agent_LipSync : public UAnimGraphNode_Base
class UAnimGraphNode_PS_AI_ConvAgent_LipSync : public UAnimGraphNode_Base
{
GENERATED_BODY()
UPROPERTY(EditAnywhere, Category = "Settings")
FAnimNode_PS_AI_Agent_LipSync Node;
FAnimNode_PS_AI_ConvAgent_LipSync Node;
// UAnimGraphNode_Base interface
virtual FText GetNodeTitle(ENodeTitleType::Type TitleType) const override;

18
Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Public/AnimGraphNode_PS_AI_Agent_Posture.h → Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgentEditor/Public/AnimGraphNode_PS_AI_ConvAgent_Posture.h
View File

@ -4,26 +4,26 @@
#include "CoreMinimal.h"
#include "AnimGraphNode_Base.h"
#include "AnimNode_PS_AI_Agent_Posture.h"
#include "AnimGraphNode_PS_AI_Agent_Posture.generated.h"
#include "AnimNode_PS_AI_ConvAgent_Posture.h"
#include "AnimGraphNode_PS_AI_ConvAgent_Posture.generated.h"
/**
* AnimGraph editor node for the PS AI Agent Posture AnimNode.
* AnimGraph editor node for the PS AI ConvAgent Posture AnimNode.
*
* This node appears in the AnimBP graph editor under the "PS AI Agent" category.
* Place it AFTER the PS AI Agent Facial Expression node and BEFORE the
* PS AI Agent Lip Sync node in the MetaHuman Face AnimBP.
* This node appears in the AnimBP graph editor under the "PS AI ConvAgent" category.
* Place it AFTER the PS AI ConvAgent Facial Expression node and BEFORE the
* PS AI ConvAgent Lip Sync node in the MetaHuman Face AnimBP.
*
* It auto-discovers the PS_AI_Agent_PostureComponent on the owning Actor
* It auto-discovers the PS_AI_ConvAgent_PostureComponent on the owning Actor
* and injects head bone rotation + ARKit eye gaze curves for look-at tracking.
*/
UCLASS()
class UAnimGraphNode_PS_AI_Agent_Posture : public UAnimGraphNode_Base
class UAnimGraphNode_PS_AI_ConvAgent_Posture : public UAnimGraphNode_Base
{
GENERATED_BODY()
UPROPERTY(EditAnywhere, Category = "Settings")
FAnimNode_PS_AI_Agent_Posture Node;
FAnimNode_PS_AI_ConvAgent_Posture Node;
// UAnimGraphNode_Base interface
virtual FText GetNodeTitle(ENodeTitleType::Type TitleType) const override;