Add PS_AI_Agent_ElevenLabs plugin (initial implementation)
Adds a new UE5.5 plugin integrating the ElevenLabs Conversational AI Agent via WebSocket. No gRPC or third-party libs required. Plugin components: - UElevenLabsSettings: API key + Agent ID in Project Settings - UElevenLabsWebSocketProxy: full WS session lifecycle, JSON message handling, ping/pong keepalive, Base64 PCM audio send/receive - UElevenLabsConversationalAgentComponent: ActorComponent for NPC voice conversation, orchestrates mic capture -> WS -> procedural audio playback - UElevenLabsMicrophoneCaptureComponent: wraps Audio::FAudioCapture, resamples to 16kHz mono, dispatches on game thread Also adds .claude/ memory files (project context, plugin notes, patterns) so Claude Code can restore full context on any machine after a git pull. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
35
.claude/MEMORY.md
Normal file
35
.claude/MEMORY.md
Normal file
@@ -0,0 +1,35 @@
|
||||
# Project Memory – PS_AI_Agent
|
||||
|
||||
> This file is committed to the repository so it is available on any machine.
|
||||
> Claude Code reads it automatically at session start (via the auto-memory system)
|
||||
> when the working directory is inside this repo.
|
||||
> **Keep it under ~180 lines** – lines beyond 200 are truncated by the system.
|
||||
|
||||
---
|
||||
|
||||
## Project Location
|
||||
- Repo root: `<repo_root>/` (wherever this is cloned)
|
||||
- UE5 project: `<repo_root>/Unreal/PS_AI_Agent/`
|
||||
- `.uproject`: `<repo_root>/Unreal/PS_AI_Agent/PS_AI_Agent.uproject`
|
||||
- Engine: **Unreal Engine 5.5** — Win64 primary target
|
||||
|
||||
## Plugins
|
||||
| Plugin | Path | Purpose |
|
||||
|--------|------|---------|
|
||||
| Convai (reference) | `<repo_root>/ConvAI/Convai/` | gRPC + protobuf streaming to Convai API. Has ElevenLabs voice type enum in `ConvaiDefinitions.h`. Used as architectural reference. |
|
||||
| **PS_AI_Agent_ElevenLabs** | `<repo_root>/Unreal/PS_AI_Agent/Plugins/PS_AI_Agent_ElevenLabs/` | Our ElevenLabs Conversational AI integration. See `.claude/elevenlabs_plugin.md` for full details. |
|
||||
|
||||
## User Preferences
|
||||
- Plugin naming: `PS_AI_Agent_<Service>` (e.g. `PS_AI_Agent_ElevenLabs`)
|
||||
- Save memory frequently during long sessions
|
||||
- Goal: ElevenLabs Conversational AI integration — simpler than Convai, no gRPC
|
||||
- Full original ask + intent: see `.claude/project_context.md`
|
||||
|
||||
## Key UE5 Plugin Patterns
|
||||
- Settings object: `UCLASS(config=Engine, defaultconfig)` inheriting `UObject`, registered via `ISettingsModule`
|
||||
- Module startup: `NewObject<USettings>(..., RF_Standalone)` + `AddToRoot()`
|
||||
- WebSocket: `FWebSocketsModule::Get().CreateWebSocket(URL, TEXT(""), Headers)`
|
||||
- Audio capture: `Audio::FAudioCapture` from the `AudioCapture` module
|
||||
- Procedural audio playback: `USoundWaveProcedural` + `OnSoundWaveProceduralUnderflow` delegate
|
||||
- Audio capture callbacks arrive on a **background thread** — always marshal to game thread with `AsyncTask(ENamedThreads::GameThread, ...)`
|
||||
- Resample mic audio to **16000 Hz mono** before sending to ElevenLabs
|
||||
61
.claude/elevenlabs_plugin.md
Normal file
61
.claude/elevenlabs_plugin.md
Normal file
@@ -0,0 +1,61 @@
|
||||
# PS_AI_Agent_ElevenLabs Plugin
|
||||
|
||||
## Location
|
||||
`Unreal/PS_AI_Agent/Plugins/PS_AI_Agent_ElevenLabs/`
|
||||
|
||||
## File Map
|
||||
```
|
||||
PS_AI_Agent_ElevenLabs.uplugin
|
||||
Source/PS_AI_Agent_ElevenLabs/
|
||||
PS_AI_Agent_ElevenLabs.Build.cs
|
||||
Public/
|
||||
PS_AI_Agent_ElevenLabs.h – FPS_AI_Agent_ElevenLabsModule + UElevenLabsSettings
|
||||
ElevenLabsDefinitions.h – Enums, structs, ElevenLabsMessageType/Audio constants
|
||||
ElevenLabsWebSocketProxy.h/.cpp – UObject managing one WS session
|
||||
ElevenLabsConversationalAgentComponent.h/.cpp – Main ActorComponent (attach to NPC)
|
||||
ElevenLabsMicrophoneCaptureComponent.h/.cpp – Mic capture, resample, dispatch to game thread
|
||||
Private/
|
||||
(implementations of the above)
|
||||
```
|
||||
|
||||
## ElevenLabs Conversational AI Protocol
|
||||
- **WebSocket URL**: `wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<ID>`
|
||||
- **Auth**: HTTP upgrade header `xi-api-key: <key>` (set in Project Settings)
|
||||
- **All frames**: JSON text (no binary frames used by the API)
|
||||
- **Audio format**: PCM 16-bit signed, 16000 Hz, mono, little-endian — Base64-encoded in JSON
|
||||
|
||||
### Client → Server messages
|
||||
| Type field value | Payload |
|
||||
|---|---|
|
||||
| *(none – key is the type)* `user_audio_chunk` | `{ "user_audio_chunk": "<base64 PCM>" }` |
|
||||
| `user_turn_start` | `{ "type": "user_turn_start" }` |
|
||||
| `user_turn_end` | `{ "type": "user_turn_end" }` |
|
||||
| `interrupt` | `{ "type": "interrupt" }` |
|
||||
| `pong` | `{ "type": "pong", "pong_event": { "event_id": N } }` |
|
||||
|
||||
### Server → Client messages (field: `type`)
|
||||
| type value | Key nested object | Notes |
|
||||
|---|---|---|
|
||||
| `conversation_initiation_metadata` | `conversation_initiation_metadata_event.conversation_id` | Marks WS ready |
|
||||
| `audio` | `audio_event.audio_base_64` | Base64 PCM from agent |
|
||||
| `transcript` | `transcript_event.{speaker, message, is_final}` | User or agent speech |
|
||||
| `agent_response` | `agent_response_event.agent_response` | Final agent text |
|
||||
| `interruption` | — | Agent stopped mid-sentence |
|
||||
| `ping` | `ping_event.event_id` | Must reply with pong |
|
||||
|
||||
## Key Design Decisions
|
||||
- **No gRPC / no ThirdParty libs** — pure UE WebSockets + HTTP, builds out of the box
|
||||
- Audio resampled in-plugin: device rate → 16000 Hz mono (linear interpolation)
|
||||
- `USoundWaveProcedural` for real-time agent audio playback (queue-driven)
|
||||
- Silence heuristic: 30 game-thread ticks (~0.5 s at 60 fps) with no new audio → agent done speaking
|
||||
- `bSignedURLMode` setting: fetch a signed WS URL from your own backend (keeps API key off client)
|
||||
- Two turn modes: `Server VAD` (ElevenLabs detects speech end) and `Client Controlled` (push-to-talk)
|
||||
|
||||
## Build Dependencies (Build.cs)
|
||||
Core, CoreUObject, Engine, InputCore, Json, JsonUtilities, WebSockets, HTTP,
|
||||
AudioMixer, AudioCaptureCore, AudioCapture, Voice, SignalProcessing
|
||||
|
||||
## Status
|
||||
- **Session 1** (2026-02-19): All source files written, registered in .uproject. Not yet compiled.
|
||||
- **TODO**: Open in UE 5.5 Editor → compile → test basic WS connection with a test agent ID.
|
||||
- **Watch out**: Verify `USoundWaveProcedural::OnSoundWaveProceduralUnderflow` delegate signature vs UE 5.5 API.
|
||||
79
.claude/project_context.md
Normal file
79
.claude/project_context.md
Normal file
@@ -0,0 +1,79 @@
|
||||
# Project Context & Original Ask
|
||||
|
||||
## What the user wants to build
|
||||
|
||||
A **UE5 plugin** that integrates the **ElevenLabs Conversational AI Agent** API into Unreal Engine 5.5,
|
||||
allowing an in-game NPC (or any Actor) to hold a real-time voice conversation with a player.
|
||||
|
||||
### The original request (paraphrased)
|
||||
> "I want to create a plugin to use ElevenLabs Conversational Agent in Unreal Engine 5.5.
|
||||
> I previously used the Convai plugin which does what I want, but I prefer ElevenLabs quality.
|
||||
> The goal is to create a plugin in the existing Unreal Project to make a first step for integration.
|
||||
> Convai AI plugin may be too big in terms of functionality for the new project, but it is the final goal.
|
||||
> You can use the Convai source code to find the right way to make the ElevenLabs version —
|
||||
> it should be very similar."
|
||||
|
||||
### Plugin name
|
||||
`PS_AI_Agent_ElevenLabs`
|
||||
|
||||
---
|
||||
|
||||
## User's mental model / intent
|
||||
|
||||
1. **Short-term**: A working first-step plugin — minimal but functional — that can:
|
||||
- Connect to ElevenLabs Conversational AI via WebSocket
|
||||
- Capture microphone audio from the player
|
||||
- Stream it to ElevenLabs in real time
|
||||
- Play back the agent's voice response
|
||||
- Surface key events (transcript, agent text, speaking state) to Blueprint
|
||||
|
||||
2. **Long-term**: Match the full feature set of Convai — character IDs, session memory,
|
||||
actions/environment context, lip-sync, etc. — but powered by ElevenLabs instead.
|
||||
|
||||
3. **Key preference**: Simpler than Convai. No gRPC, no protobuf, no ThirdParty precompiled
|
||||
libraries. ElevenLabs' Conversational AI API uses plain WebSocket + JSON, which maps
|
||||
naturally to UE's built-in `WebSockets` module.
|
||||
|
||||
---
|
||||
|
||||
## How we used Convai as a reference
|
||||
|
||||
We studied the Convai plugin source (`ConvAI/Convai/`) to understand:
|
||||
- **Module structure**: `UConvaiSettings` + `IModuleInterface` + `ISettingsModule` registration
|
||||
- **Audio capture pattern**: `Audio::FAudioCapture`, ring buffers, thread-safe dispatch to game thread
|
||||
- **Audio playback pattern**: `USoundWaveProcedural` fed from a queue
|
||||
- **Component architecture**: `UConvaiChatbotComponent` (NPC side) + `UConvaiPlayerComponent` (player side)
|
||||
- **HTTP proxy pattern**: `UConvaiAPIBaseProxy` base class for async REST calls
|
||||
- **Voice type enum**: Convai already had `EVoiceType::ElevenLabsVoices` — confirming ElevenLabs
|
||||
is a natural fit
|
||||
|
||||
We then replaced gRPC/protobuf with **WebSocket + JSON** to match the ElevenLabs API, and
|
||||
simplified the architecture to the minimum needed for a first working version.
|
||||
|
||||
---
|
||||
|
||||
## What was built (Session 1 — 2026-02-19)
|
||||
|
||||
All source files created and registered. See `.claude/elevenlabs_plugin.md` for full file map and protocol details.
|
||||
|
||||
### Components created
|
||||
| Class | Role |
|
||||
|---|---|
|
||||
| `UElevenLabsSettings` | Project Settings UI — API key, Agent ID, security options |
|
||||
| `UElevenLabsWebSocketProxy` | Manages one WS session: connect, send audio, handle all server message types |
|
||||
| `UElevenLabsConversationalAgentComponent` | ActorComponent to attach to any NPC — orchestrates mic + WS + playback |
|
||||
| `UElevenLabsMicrophoneCaptureComponent` | Wraps `Audio::FAudioCapture`, resamples to 16 kHz mono |
|
||||
|
||||
### Not yet done (next sessions)
|
||||
- Compile & test in UE 5.5 Editor
|
||||
- Verify `USoundWaveProcedural::OnSoundWaveProceduralUnderflow` delegate signature for UE 5.5
|
||||
- Add lip-sync support (future)
|
||||
- Add session memory / conversation history (future)
|
||||
- Add environment/action context support (future, matching Convai's full feature set)
|
||||
|
||||
---
|
||||
|
||||
## Notes on the ElevenLabs API
|
||||
- Docs: https://elevenlabs.io/docs/conversational-ai
|
||||
- Create agents at: https://elevenlabs.io/app/conversational-ai
|
||||
- API keys at: https://elevenlabs.io (dashboard)
|
||||
7
.claude/settings.local.json
Normal file
7
.claude/settings.local.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"permissions": {
|
||||
"allow": [
|
||||
"Bash(dir /s \"E:\\\\ASTERION\\\\GIT\\\\PS_AI_Agent\")"
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user