j.foucher
f0055e85ed
Adds a new UE5.5 plugin integrating the ElevenLabs Conversational AI Agent via WebSocket. No gRPC or third-party libs required. Plugin components: - UElevenLabsSettings: API key + Agent ID in Project Settings - UElevenLabsWebSocketProxy: full WS session lifecycle, JSON message handling, ping/pong keepalive, Base64 PCM audio send/receive - UElevenLabsConversationalAgentComponent: ActorComponent for NPC voice conversation, orchestrates mic capture -> WS -> procedural audio playback - UElevenLabsMicrophoneCaptureComponent: wraps Audio::FAudioCapture, resamples to 16kHz mono, dispatches on game thread Also adds .claude/ memory files (project context, plugin notes, patterns) so Claude Code can restore full context on any machine after a git pull. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
36 lines
2.0 KiB
Markdown
36 lines
2.0 KiB
Markdown
# Project Memory – PS_AI_Agent
|
||
|
||
> This file is committed to the repository so it is available on any machine.
|
||
> Claude Code reads it automatically at session start (via the auto-memory system)
|
||
> when the working directory is inside this repo.
|
||
> **Keep it under ~180 lines** – lines beyond 200 are truncated by the system.
|
||
|
||
---
|
||
|
||
## Project Location
|
||
- Repo root: `<repo_root>/` (wherever this is cloned)
|
||
- UE5 project: `<repo_root>/Unreal/PS_AI_Agent/`
|
||
- `.uproject`: `<repo_root>/Unreal/PS_AI_Agent/PS_AI_Agent.uproject`
|
||
- Engine: **Unreal Engine 5.5** — Win64 primary target
|
||
|
||
## Plugins
|
||
| Plugin | Path | Purpose |
|
||
|--------|------|---------|
|
||
| Convai (reference) | `<repo_root>/ConvAI/Convai/` | gRPC + protobuf streaming to Convai API. Has ElevenLabs voice type enum in `ConvaiDefinitions.h`. Used as architectural reference. |
|
||
| **PS_AI_Agent_ElevenLabs** | `<repo_root>/Unreal/PS_AI_Agent/Plugins/PS_AI_Agent_ElevenLabs/` | Our ElevenLabs Conversational AI integration. See `.claude/elevenlabs_plugin.md` for full details. |
|
||
|
||
## User Preferences
|
||
- Plugin naming: `PS_AI_Agent_<Service>` (e.g. `PS_AI_Agent_ElevenLabs`)
|
||
- Save memory frequently during long sessions
|
||
- Goal: ElevenLabs Conversational AI integration — simpler than Convai, no gRPC
|
||
- Full original ask + intent: see `.claude/project_context.md`
|
||
|
||
## Key UE5 Plugin Patterns
|
||
- Settings object: `UCLASS(config=Engine, defaultconfig)` inheriting `UObject`, registered via `ISettingsModule`
|
||
- Module startup: `NewObject<USettings>(..., RF_Standalone)` + `AddToRoot()`
|
||
- WebSocket: `FWebSocketsModule::Get().CreateWebSocket(URL, TEXT(""), Headers)`
|
||
- Audio capture: `Audio::FAudioCapture` from the `AudioCapture` module
|
||
- Procedural audio playback: `USoundWaveProcedural` + `OnSoundWaveProceduralUnderflow` delegate
|
||
- Audio capture callbacks arrive on a **background thread** — always marshal to game thread with `AsyncTask(ENamedThreads::GameThread, ...)`
|
||
- Resample mic audio to **16000 Hz mono** before sending to ElevenLabs
|