j.foucher/PS_AI_Agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
PS_AI_Agent/.claude/MEMORY.md
j.foucher f0055e85ed Add PS_AI_Agent_ElevenLabs plugin (initial implementation) 
Adds a new UE5.5 plugin integrating the ElevenLabs Conversational AI Agent
via WebSocket. No gRPC or third-party libs required.

Plugin components:
- UElevenLabsSettings: API key + Agent ID in Project Settings
- UElevenLabsWebSocketProxy: full WS session lifecycle, JSON message handling,
  ping/pong keepalive, Base64 PCM audio send/receive
- UElevenLabsConversationalAgentComponent: ActorComponent for NPC voice
  conversation, orchestrates mic capture -> WS -> procedural audio playback
- UElevenLabsMicrophoneCaptureComponent: wraps Audio::FAudioCapture,
  resamples to 16kHz mono, dispatches on game thread

Also adds .claude/ memory files (project context, plugin notes, patterns)
so Claude Code can restore full context on any machine after a git pull.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-19 12:57:48 +01:00

2.0 KiB
Raw Blame History

Project Memory – PS_AI_Agent

This file is committed to the repository so it is available on any machine. Claude Code reads it automatically at session start (via the auto-memory system) when the working directory is inside this repo. Keep it under ~180 lines – lines beyond 200 are truncated by the system.

Project Location

  • Repo root: <repo_root>/ (wherever this is cloned)
  • UE5 project: <repo_root>/Unreal/PS_AI_Agent/
  • .uproject: <repo_root>/Unreal/PS_AI_Agent/PS_AI_Agent.uproject
  • Engine: Unreal Engine 5.5 — Win64 primary target

Plugins

Plugin Path Purpose
Convai (reference) <repo_root>/ConvAI/Convai/ gRPC + protobuf streaming to Convai API. Has ElevenLabs voice type enum in ConvaiDefinitions.h. Used as architectural reference.
PS_AI_Agent_ElevenLabs <repo_root>/Unreal/PS_AI_Agent/Plugins/PS_AI_Agent_ElevenLabs/ Our ElevenLabs Conversational AI integration. See .claude/elevenlabs_plugin.md for full details.

User Preferences

  • Plugin naming: PS_AI_Agent_<Service> (e.g. PS_AI_Agent_ElevenLabs)
  • Save memory frequently during long sessions
  • Goal: ElevenLabs Conversational AI integration — simpler than Convai, no gRPC
  • Full original ask + intent: see .claude/project_context.md

Key UE5 Plugin Patterns

  • Settings object: UCLASS(config=Engine, defaultconfig) inheriting UObject, registered via ISettingsModule
  • Module startup: NewObject<USettings>(..., RF_Standalone) + AddToRoot()
  • WebSocket: FWebSocketsModule::Get().CreateWebSocket(URL, TEXT(""), Headers)
  • Audio capture: Audio::FAudioCapture from the AudioCapture module
  • Procedural audio playback: USoundWaveProcedural + OnSoundWaveProceduralUnderflow delegate
  • Audio capture callbacks arrive on a background thread — always marshal to game thread with AsyncTask(ENamedThreads::GameThread, ...)
  • Resample mic audio to 16000 Hz mono before sending to ElevenLabs