j.foucher/PS_AI_Agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

memory

Browse Source
This commit is contained in:
j.foucher
2026-02-19 15:02:55 +01:00
parent 99017f4067
commit 302337b573
2 changed files with 89 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

14
.claude/MEMORY.md
View File

@@ -43,6 +43,20 @@
## Plugin Status
- **PS_AI_Agent_ElevenLabs**: compiles cleanly on UE 5.5 Win64 (verified 2026-02-19)
- v1.1.0 — all 3 protocol bugs fixed (transcript fields, pong format, client turn mode)
- Binary WS frame handling implemented (ElevenLabs sends ALL frames as binary, not text)
- First-byte discrimination: `{` = JSON control message, else = raw PCM audio
- `SendTextMessage()` added to both WebSocketProxy and ConversationalAgentComponent
- Connection confirmed working end-to-end; audio receive path functional
## ElevenLabs WebSocket Protocol Notes
- **ALL frames are binary** — `OnRawMessage` handles everything; `OnMessage` (text) never fires
- Binary frame discrimination: peek byte[0] → `'{'` (0x7B) = JSON, else = raw PCM audio
- Fragment reassembly: accumulate into `BinaryFrameBuffer` until `BytesRemaining == 0`
- Pong: `{"type":"pong","event_id":N}` — `event_id` is **top-level**, NOT nested
- Transcript: type=`user_transcript`, key=`user_transcription_event`, field=`user_transcript`
- Client turn mode: `{"type":"user_activity"}` to signal speaking; no explicit end message
- Text input: `{"type":"user_message","text":"..."}` — agent replies with audio + text
## API Keys / Secrets
- ElevenLabs API key is set in **Project Settings → Plugins → ElevenLabs AI Agent** in the Editor