ConvAgent: cut BOTH agents' speech when the player interrupts the dialogue

On a player barge-in, the engaged agent stops (its mic feeds the player's voice to ElevenLabs, which interrupts it), but the witness agent never hears the player and kept finishing its line (e.g. '...I'll give you the cash...') mid-scene. InterruptForPlayer now calls InterruptAgent() on both CompA and CompB. Conversations stay open; the witness still receives the contextual_updates.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

.claude/CLAUDE.md

@@ -0,0 +1,95 @@
+# PS_AI_Agent — Notes pour Claude
+
+## Workflow règles
+
+**TOUJOURS demander avant de lancer un changement de code.** Avant d'appeler Edit/Write/sed sur un fichier source :
+1. Expliquer ce qu'on s'apprête à faire (quel fichier, quelle logique, quel impact)
+2. Attendre la confirmation explicite de l'utilisateur
+3. Seulement après → effectuer le changement
+
+Cela s'applique à : modifications de code (.cpp/.h), refactors, renames bulk, ajouts de nouvelles fonctions.
+Cela ne s'applique PAS à : lecture de fichiers, recherches, git status/log/diff, builds, questions diagnostiques.
+
+Exceptions où on peut agir directement :
+- L'utilisateur dit explicitement "fais-le" / "go" / "lance" / "commit"
+- L'utilisateur demande explicitement un changement précis ("renomme X en Y", "ajoute cette fonction")
+- Annulation / revert demandé par l'utilisateur
+
+## Plugins dans ce repo
+
+### PS_AI_ConvAgent
+- **Module**: `PS_AI_ConvAgent` / `PS_AI_ConvAgentEditor`
+- **API macro**: `PS_AI_CONVAGENT_API`
+- **Class prefix**: `PS_AI_ConvAgent_` (e.g. `UPS_AI_ConvAgent_PostureComponent`)
+- **ElevenLabs-specific**: suffix `_ElevenLabs` (e.g. `UPS_AI_ConvAgent_ElevenLabsComponent`)
+- **UI display**: `"PS AI ConvAgent"` (categories, DisplayName)
+- **CoreRedirects**: DefaultEngine.ini handles both ElevenLabs* and PS_AI_Agent_* → PS_AI_ConvAgent_*
+
+### PS_AI_Behavior
+- **Module**: `PS_AI_Behavior` / `PS_AI_BehaviorEditor`
+- **API macro**: `PS_AI_BEHAVIOR_API`
+- **Class prefix**: `PS_AI_Behavior_` (e.g. `UPS_AI_Behavior_PersonalityComponent`)
+- **UI display**: `"PS AI Behavior"` (categories, DisplayName)
+- **Interface**: `IPS_AI_Behavior_Interface` — implements on host project Pawns
+
+## PS_AI_Behavior — Architecture
+
+### Core Design
+- **Interface-driven**: `IPS_AI_Behavior_Interface` on Pawns decouples plugin from host project
+- **Personality profiles**: DataAsset with 5 trait axes, thresholds, target priority, speed per state
+- **NPCType enum**: Civilian, Enemy, Protector, Any (unified — no separate SplineCategory)
+- **TeamIds**: Civilian=1, Enemy=2, Protector=3, auto-assigned in OnPossess
+- **Hostile switch**: Enemy with IsBehaviorHostile=false → TeamId=1 (disguised), flips to 2 at runtime
+
+### Blackboard
+- Key `BehaviorState` = Enum type, path: `/Script/PS_AI_Behavior.EPS_AI_Behavior_State`
+- Values: Idle(0), Patrol(1), Alerted(2), Combat(3), Fleeing(4), TakingCover(5), Dead(6)
+- BB asset created manually in editor (runtime fallback exists but prefer asset)
+- SetValueAsEnum / GetValueAsEnum (not Int)
+
+### BT Pattern
+- Services on Selector: UpdateThreat + EvaluateReaction
+- Decorator Observer Aborts = Both on Combat/Flee branches
+- Spline branch as fallback (lowest priority, no condition decorator)
+- Attack task stays InProgress permanently, Decorator pulls out on state change
+
+### Spline System
+- SplinePath actors placed in level, SplineCategory = NPCType
+- SplineNetwork WorldSubsystem auto-detects junctions
+- SplineFollowerComponent on Pawns: uses Pawn's real velocity, debug green sphere
+- Junction switching: 70% probability, direction filter (dot > -0.5)
+
+### Perception
+- Senses configured in BeginPlay (NOT constructor — NewObject crashes in CDO)
+- RequestStimuliListenerUpdate() after ConfigureSenses
+- Detect ALL affiliations, filter by Hostile attitude in CalculateThreatLevel
+- Target priority from PersonalityProfile (combat targeting)
+
+### Known Gotchas (UE 5.5)
+- BB Enum picker doesn't list plugin enums — use string path
+- NTFS junctions (not symlinks) for UE5 cross-project plugin linking
+- SVN doesn't traverse junctions — robocopy for initial import
+- Live Coding fails on header changes — full rebuild required
+- BlueprintNativeEvent: UHT auto-generates _Implementation defaults, don't duplicate in .cpp
+- FGenericTeamId::NoTeam comparison: always recalculate TeamId (BP CDOs may reset to 0)
+
+### Testing Status (2026-03-26)
+- ✅ Patrol, Spline following, Perception, Threat, Flee, Combat delegation, Hostile switch
+- 🔲 Cover Points, EQS, Editor tools (SplineEdMode, CoverPoint placement)
+
+## Posture System — Diagonal Tilt Bug (à corriger)
+
+### Problème
+Le tilt diagonal (ear-to-shoulder) est PLUS VISIBLE avec la neck bone chain qu'avec le mono-bone.
+
+### Piste de fix
+Faire le swing-twist **PER-BONE** : pour chaque bone de la chaîne, composer le FractionalRot avec le bone's own rotation, swing-twist autour du bone's own tilt axis.
+
+### Fichiers concernés
+- `AnimNode_PS_AI_ConvAgent_Posture.cpp` — Evaluate_AnyThread, section multi-bone chain
+
+## Architecture Posture (ConvAgent)
+- **PS_AI_ConvAgent_PostureComponent** (game thread) : cascade Eyes→Head→Body
+- **AnimNode_PS_AI_ConvAgent_Posture** (anim thread) : applique rotation sur bone(s) + injecte curves
+- Pipeline 100% FQuat, Thread safety via FCriticalSection
+- ARKit eye curves normalisées par range fixe (40°/35°)

.claude/MEMORY.md

@@ -0,0 +1,86 @@
+# Project Memory – PS_AI_Agent
+
+> This file is committed to the repository so it is available on any machine.
+> Claude Code reads it automatically at session start (via the auto-memory system)
+> when the working directory is inside this repo.
+> **Keep it under ~180 lines** – lines beyond 200 are truncated by the system.
+
+---
+
+## Project Location
+- Repo root: `<repo_root>/` (wherever this is cloned)
+- UE5 project: `<repo_root>/Unreal/PS_AI_Agent/`
+- `.uproject`: `<repo_root>/Unreal/PS_AI_Agent/PS_AI_Agent.uproject`
+- Engine: **Unreal Engine 5.5** — Win64 primary target
+- Default test map: `/Game/TestMap.TestMap`
+
+## Plugins
+| Plugin | Path | Purpose |
+|--------|------|---------|
+| Convai (reference) | `<repo_root>/ConvAI/Convai/` | gRPC + protobuf streaming to Convai API. Used as architectural reference. |
+| **PS_AI_ConvAgent** | `<repo_root>/Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/` | Main plugin — ElevenLabs Conversational AI, posture, gaze, lip sync, facial expressions. |
+
+## User Preferences
+- Plugin naming: `PS_AI_ConvAgent` (renamed from PS_AI_Agent_ElevenLabs)
+- Save memory frequently during long sessions
+- Git remote is a **private server** — no public exposure risk
+- Full original ask + intent: see `.claude/project_context.md`
+
+## Current Branch & Work
+- **Branch**: `main`
+- **Recent merges**: `feature/multi-player-shared-agent` merged to main
+
+### Latency Debug HUD (just implemented)
+- Separate `bDebugLatency` property + CVar `ps.ai.ConvAgent.Debug.Latency`
+- All metrics anchored to `GenerationStartTime` (`agent_response_started` event)
+- Metrics: Gen>Audio (LLM+TTS), Pre-buffer, Gen>Ear (user-perceived)
+- Reset per turn in `HandleAgentResponseStarted()`
+- `DrawLatencyHUD()` separate from `DrawDebugHUD()`
+
+### Future: Server-Side Latency from ElevenLabs API
+**TODO — high-value improvement parked for later:**
+- `GET /v1/convai/conversations/{conversation_id}` returns:
+  - `conversation_turn_metrics` with `elapsed_time` per metric (STT, LLM, TTS breakdown!)
+  - `tool_latency_secs`, `step_latency_secs`, `rag_latency_secs`
+  - `time_in_call_secs` per message
+- `ping` WS event has `ping_ms` (network round-trip) — could display on HUD
+- `vad_score` WS event (0.0-1.0) — could detect real speech start client-side
+- Docs: https://elevenlabs.io/docs/api-reference/conversations/get
+
+### Multi-Player Shared Agent — Key Design
+- **Old model**: exclusive lock (one player per agent via `NetConversatingPawn`)
+- **New model**: shared array (`NetConnectedPawns`) + active speaker (`NetActiveSpeakerPawn`)
+- Speaker arbitration: server-side with `SpeakerSwitchHysteresis` (0.3s) + `SpeakerIdleTimeout` (3.0s)
+- In standalone (≤1 player): speaker arbitration bypassed, audio sent directly to WebSocket
+- Internal mic (WASAPI thread): direct WebSocket send, no game-thread state access
+- `GetCurrentBlendshapes()` thread-safe via `ThreadSafeBlendshapes` snapshot + `BlendshapeLock`
+
+## Key UE5 Plugin Patterns
+- Settings object: `UCLASS(config=Engine, defaultconfig)` inheriting `UObject`, registered via `ISettingsModule`
+- WebSocket: `FWebSocketsModule::Get().CreateWebSocket(URL, TEXT(""), Headers)`
+- Audio capture: `Audio::FAudioCapture::OpenAudioCaptureStream()` (UE 5.3+)
+  - Callback arrives on **background thread** — marshal to game thread
+- Procedural audio playback: `USoundWaveProcedural` + `OnSoundWaveProceduralUnderflow`
+- Resample mic audio to **16000 Hz mono** before sending to ElevenLabs
+- `TArray::RemoveAt(idx, count, EAllowShrinking::No)` — bool overload deprecated in UE 5.5
+
+## ElevenLabs WebSocket Protocol Notes
+- **ALL frames are binary** — bind ONLY `OnRawMessage`; NEVER bind `OnMessage` (text)
+- Binary frame discrimination: peek byte[0] → `'{'` (0x7B) = JSON, else = raw PCM audio
+- Pong: `{"type":"pong","event_id":N}` — `event_id` is **top-level**, NOT nested
+- `user_transcript` arrives AFTER `agent_response_started` in Server VAD mode
+- **MUST send `conversation_initiation_client_data` immediately after WS connect**
+
+## API Keys / Secrets
+- ElevenLabs API key: **Project Settings → Plugins → ElevenLabs AI Agent**
+- Saved to `DefaultEngine.ini` — **stripped before every commit**
+
+## Claude Memory Files in This Repo
+| File | Contents |
+|------|----------|
+| `.claude/MEMORY.md` | This file — project structure, patterns, status |
+| `.claude/elevenlabs_plugin.md` | Plugin file map, ElevenLabs WS protocol, design decisions |
+| `.claude/elevenlabs_api_reference.md` | Full ElevenLabs API reference (WS messages, REST, signed URL) |
+| `.claude/project_context.md` | Original ask, intent, short/long-term goals |
+| `.claude/session_log_2026-02-19.md` | Session record: steps, commits, technical decisions |
+| `.claude/PS_AI_Agent_ElevenLabs_Documentation.md` | User-facing Markdown reference doc |

.claude/PS_AI_Agent_ElevenLabs_Documentation.md

@@ -0,0 +1,619 @@
+# PS_AI_Agent_ElevenLabs — Plugin Documentation
+
+**Engine**: Unreal Engine 5.5
+**Plugin version**: 1.1.0
+**Status**: Beta — tested on UE 5.5 Win64, verified connection and audio pipeline
+**API**: [ElevenLabs Conversational AI](https://elevenlabs.io/docs/eleven-agents/quickstart)
+
+---
+
+## Table of Contents
+
+1. [Overview](#1-overview)
+2. [Installation](#2-installation)
+3. [Project Settings](#3-project-settings)
+4. [Quick Start (Blueprint)](#4-quick-start-blueprint)
+5. [Quick Start (C++)](#5-quick-start-c)
+6. [Components Reference](#6-components-reference)
+   - [UElevenLabsConversationalAgentComponent](#uelevenlabsconversationalagentcomponent)
+   - [UElevenLabsMicrophoneCaptureComponent](#uelevenlabsmicrophonecapturecomponent)
+   - [UElevenLabsWebSocketProxy](#uelevenlabswebsocketproxy)
+7. [Data Types Reference](#7-data-types-reference)
+8. [Turn Modes](#8-turn-modes)
+9. [Security — Signed URL Mode](#9-security--signed-url-mode)
+10. [Audio Pipeline](#10-audio-pipeline)
+11. [Common Patterns](#11-common-patterns)
+12. [Troubleshooting](#12-troubleshooting)
+13. [Changelog](#13-changelog)
+
+---
+
+## 1. Overview
+
+This plugin integrates the **ElevenLabs Conversational AI Agent** API into Unreal Engine 5.5, enabling real-time voice conversations between a player and an NPC (or any Actor).
+
+### How it works
+
+```
+Player microphone
+      │
+      ▼
+UElevenLabsMicrophoneCaptureComponent
+  • Captures from default audio device
+  • Resamples to 16 kHz mono float32
+      │
+      ▼
+UElevenLabsConversationalAgentComponent
+  • Converts float32 → int16 PCM bytes
+  • Base64-encodes and sends via WebSocket
+      │  (wss://api.elevenlabs.io/v1/convai/conversation)
+      ▼
+ElevenLabs Conversational AI Agent
+  • Transcribes speech
+  • Runs LLM
+  • Synthesizes voice (ElevenLabs TTS)
+      │
+      ▼
+UElevenLabsConversationalAgentComponent
+  • Receives raw binary PCM audio frames
+  • Feeds USoundWaveProcedural → UAudioComponent
+      │
+      ▼
+Agent voice plays from the Actor's position in the world
+```
+
+### Key properties
+- No gRPC, no third-party libraries — uses UE's built-in `WebSockets` and `AudioCapture` modules
+- Blueprint-first: all events and controls are exposed to Blueprint
+- Real-time bidirectional: audio streams in both directions simultaneously
+- Server VAD (default) or push-to-talk
+- Text input supported (no microphone needed for testing)
+
+### Wire frame protocol notes
+ElevenLabs sends **all WebSocket frames as binary** (not text frames). The plugin handles two binary frame types automatically:
+- **JSON control frames** (start with `{`) — conversation init, transcripts, agent responses, ping/pong
+- **Raw PCM audio frames** (binary) — agent speech audio, played directly via `USoundWaveProcedural`
+
+---
+
+## 2. Installation
+
+The plugin lives inside the project, not the engine, so no separate install is needed.
+
+### Verify it is enabled
+
+Open `Unreal/PS_AI_Agent/PS_AI_Agent.uproject` and confirm:
+
+```json
+{
+  "Name": "PS_AI_Agent_ElevenLabs",
+  "Enabled": true
+}
+```
+
+### First compile
+
+Open the project in the UE 5.5 Editor. It will detect the new plugin and ask to recompile — click **Yes**. Alternatively, compile from the command line:
+
+```
+"C:\Program Files\Epic Games\UE_5.5\Engine\Build\BatchFiles\Build.bat"
+    PS_AI_AgentEditor Win64 Development
+    "<repo>/Unreal/PS_AI_Agent/PS_AI_Agent.uproject"
+    -WaitMutex
+```
+
+---
+
+## 3. Project Settings
+
+Go to **Edit → Project Settings → Plugins → ElevenLabs AI Agent**.
+
+| Setting | Description | Required |
+|---|---|---|
+| **API Key** | Your ElevenLabs API key. Find it at [elevenlabs.io/app/settings/api-keys](https://elevenlabs.io/app/settings/api-keys) | Yes (unless using Signed URL Mode or a public agent) |
+| **Agent ID** | Default agent ID. Find it in the URL when editing an agent: `elevenlabs.io/app/conversational-ai/agents/<AGENT_ID>` | Yes (unless set per-component) |
+| **Signed URL Mode** | Fetch the WS URL from your own backend (keeps key off client). See [Section 9](#9-security--signed-url-mode) | No |
+| **Signed URL Endpoint** | Your backend URL returning `{ "signed_url": "wss://..." }` | Only if Signed URL Mode = true |
+| **Custom WebSocket URL** | Override the default `wss://api.elevenlabs.io/...` endpoint (debug only) | No |
+| **Verbose Logging** | Log every WebSocket frame type and first bytes to Output Log | No |
+
+> **Security note**: The API key set in Project Settings is saved to `DefaultEngine.ini`. **Never commit this file with the key in it** — strip the `[ElevenLabsSettings]` section before committing. Use Signed URL Mode for production builds.
+
+> **Finding your Agent ID**: Go to [elevenlabs.io/app/conversational-ai](https://elevenlabs.io/app/conversational-ai), click your agent, and copy the ID from the URL bar or the agent's Overview/API tab.
+
+---
+
+## 4. Quick Start (Blueprint)
+
+### Step 1 — Add the component to an NPC
+
+1. Open your NPC Blueprint (or any Actor Blueprint).
+2. In the **Components** panel, click **Add** → search for **ElevenLabs Conversational Agent**.
+3. Select the component. In the **Details** panel you can optionally set a specific **Agent ID** (overrides the project default).
+
+### Step 2 — Set Turn Mode
+
+In the component's **Details** panel:
+- **Server VAD** (default): ElevenLabs automatically detects when the player stops speaking. Microphone streams continuously once connected.
+- **Client Controlled**: You call `Start Listening` / `Stop Listening` manually (push-to-talk).
+
+### Step 3 — Wire up events in the Event Graph
+
+```
+Event BeginPlay
+    └─► [ElevenLabs Agent] Start Conversation
+
+[ElevenLabs Agent] On Agent Connected
+    └─► Print String "Connected! ConvID: " + Conversation Info → Conversation ID
+
+[ElevenLabs Agent] On Agent Text Response
+    └─► Set Text (UI widget) ← Response Text
+
+[ElevenLabs Agent] On Agent Transcript
+    └─► (optional) display live subtitles ← Segment → Text
+
+[ElevenLabs Agent] On Agent Started Speaking
+    └─► Play talking animation on NPC
+
+[ElevenLabs Agent] On Agent Stopped Speaking
+    └─► Return to idle animation
+
+[ElevenLabs Agent] On Agent Error
+    └─► Print String "Error: " + Error Message
+
+Event EndPlay
+    └─► [ElevenLabs Agent] End Conversation
+```
+
+### Step 4 — Push-to-talk (Client Controlled mode only)
+
+```
+Input Action "Talk" (Pressed)
+    └─► [ElevenLabs Agent] Start Listening
+
+Input Action "Talk" (Released)
+    └─► [ElevenLabs Agent] Stop Listening
+```
+
+### Step 5 — Testing without a microphone
+
+Once connected, use **Send Text Message** instead of speaking:
+
+```
+[ElevenLabs Agent] On Agent Connected
+    └─► [ElevenLabs Agent] Send Text Message ← "Hello, who are you?"
+```
+
+The agent will reply with audio and text exactly as if it heard you speak.
+
+---
+
+## 5. Quick Start (C++)
+
+### 1. Add the plugin to your module's Build.cs
+
+```csharp
+PrivateDependencyModuleNames.Add("PS_AI_Agent_ElevenLabs");
+```
+
+### 2. Include and use
+
+```cpp
+#include "ElevenLabsConversationalAgentComponent.h"
+#include "ElevenLabsDefinitions.h"
+
+// In your Actor's header:
+UPROPERTY(VisibleAnywhere)
+UElevenLabsConversationalAgentComponent* ElevenLabsAgent;
+
+// In the constructor:
+ElevenLabsAgent = CreateDefaultSubobject<UElevenLabsConversationalAgentComponent>(
+    TEXT("ElevenLabsAgent"));
+
+// Override Agent ID at runtime (optional):
+ElevenLabsAgent->AgentID = TEXT("your_agent_id_here");
+ElevenLabsAgent->TurnMode = EElevenLabsTurnMode::Server;
+ElevenLabsAgent->bAutoStartListening = true;
+
+// Bind events:
+ElevenLabsAgent->OnAgentConnected.AddDynamic(
+    this, &AMyNPC::HandleAgentConnected);
+ElevenLabsAgent->OnAgentTextResponse.AddDynamic(
+    this, &AMyNPC::HandleAgentResponse);
+ElevenLabsAgent->OnAgentStartedSpeaking.AddDynamic(
+    this, &AMyNPC::PlayTalkingAnimation);
+
+// Start the conversation:
+ElevenLabsAgent->StartConversation();
+
+// Send a text message (useful for testing without mic):
+ElevenLabsAgent->SendTextMessage(TEXT("Hello, who are you?"));
+
+// Later, to end:
+ElevenLabsAgent->EndConversation();
+```
+
+### 3. Callback signatures
+
+```cpp
+UFUNCTION()
+void HandleAgentConnected(const FElevenLabsConversationInfo& Info)
+{
+    UE_LOG(LogTemp, Log, TEXT("Connected, ConvID=%s"), *Info.ConversationID);
+}
+
+UFUNCTION()
+void HandleAgentResponse(const FString& ResponseText)
+{
+    // Display in UI, drive subtitles, etc.
+}
+
+UFUNCTION()
+void PlayTalkingAnimation()
+{
+    // Switch to talking anim montage
+}
+```
+
+---
+
+## 6. Components Reference
+
+### UElevenLabsConversationalAgentComponent
+
+The **main component** — attach this to any Actor that should be able to speak.
+
+**Category**: ElevenLabs
+**Inherits from**: `UActorComponent`
+
+#### Properties
+
+| Property | Type | Default | Description |
+|---|---|---|---|
+| `AgentID` | `FString` | `""` | Agent ID for this actor. Overrides the project-level default when non-empty. |
+| `TurnMode` | `EElevenLabsTurnMode` | `Server` | How speaker turns are detected. See [Section 8](#8-turn-modes). |
+| `bAutoStartListening` | `bool` | `true` | If true, starts mic capture automatically once the WebSocket is connected and ready. |
+
+#### Functions
+
+| Function | Blueprint | Description |
+|---|---|---|
+| `StartConversation()` | Callable | Opens the WebSocket connection. If `bAutoStartListening` is true, mic capture starts once `OnAgentConnected` fires. |
+| `EndConversation()` | Callable | Closes the WebSocket, stops mic, stops audio playback. |
+| `StartListening()` | Callable | Starts microphone capture and streams to ElevenLabs. In Client mode, also sends `user_activity`. |
+| `StopListening()` | Callable | Stops microphone capture. In Client mode, stops sending `user_activity`. |
+| `SendTextMessage(Text)` | Callable | Sends a text message to the agent without using the microphone. Agent replies with full audio + text. Useful for testing. |
+| `InterruptAgent()` | Callable | Stops the agent's current utterance immediately and clears the audio queue. |
+| `IsConnected()` | Pure | Returns true if the WebSocket is open and the conversation is active. |
+| `IsListening()` | Pure | Returns true if the microphone is currently capturing. |
+| `IsAgentSpeaking()` | Pure | Returns true if agent audio is currently playing. |
+| `GetConversationInfo()` | Pure | Returns `FElevenLabsConversationInfo` (ConversationID, AgentID). |
+| `GetWebSocketProxy()` | Pure | Returns the underlying `UElevenLabsWebSocketProxy` for advanced use. |
+
+#### Events
+
+| Event | Parameters | Fired when |
+|---|---|---|
+| `OnAgentConnected` | `FElevenLabsConversationInfo` | WebSocket handshake + agent initiation metadata received. Safe to call `SendTextMessage` here. |
+| `OnAgentDisconnected` | `int32 StatusCode`, `FString Reason` | WebSocket closed (graceful or remote). |
+| `OnAgentError` | `FString ErrorMessage` | Connection or protocol error. |
+| `OnAgentTranscript` | `FElevenLabsTranscriptSegment` | User speech-to-text transcript received (speaker is always `"user"`). |
+| `OnAgentTextResponse` | `FString ResponseText` | Final text response from the agent (mirrors the audio). |
+| `OnAgentStartedSpeaking` | — | First audio chunk received from the agent (audio playback begins). |
+| `OnAgentStoppedSpeaking` | — | Audio queue empty for ~0.5 s (heuristic — agent done speaking). |
+| `OnAgentInterrupted` | — | Agent speech was interrupted (by user or by `InterruptAgent()`). |
+
+---
+
+### UElevenLabsMicrophoneCaptureComponent
+
+A lightweight microphone capture component. Managed automatically by `UElevenLabsConversationalAgentComponent` — you only need to use this directly for advanced scenarios (e.g. custom audio routing).
+
+**Category**: ElevenLabs
+**Inherits from**: `UActorComponent`
+
+#### Properties
+
+| Property | Type | Default | Description |
+|---|---|---|---|
+| `VolumeMultiplier` | `float` | `1.0` | Gain applied to captured samples before resampling. Range: 0.0 – 4.0. |
+
+#### Functions
+
+| Function | Blueprint | Description |
+|---|---|---|
+| `StartCapture()` | Callable | Opens the default audio input device and begins streaming. |
+| `StopCapture()` | Callable | Stops streaming and closes the device. |
+| `IsCapturing()` | Pure | True while actively capturing. |
+
+#### Delegate
+
+`OnAudioCaptured` — fires on the **game thread** with `TArray<float>` PCM samples at 16 kHz mono. Bind to this if you want to process or forward audio manually.
+
+---
+
+### UElevenLabsWebSocketProxy
+
+Low-level WebSocket session manager. Used internally by `UElevenLabsConversationalAgentComponent`. Use this directly only if you need fine-grained protocol control.
+
+**Inherits from**: `UObject`
+**Instantiate via**: `NewObject<UElevenLabsWebSocketProxy>(Outer)`
+
+#### Key functions
+
+| Function | Description |
+|---|---|
+| `Connect(AgentID, APIKey)` | Open the WS connection. Parameters override project settings when non-empty. |
+| `Disconnect()` | Send close frame and tear down the connection. |
+| `SendAudioChunk(PCMData)` | Send raw int16 LE PCM bytes as a Base64 JSON frame. Called automatically by the agent component. |
+| `SendTextMessage(Text)` | Send `{"type":"user_message","text":"..."}`. Agent replies as if it heard speech. |
+| `SendUserTurnStart()` | Client turn mode: sends `{"type":"user_activity"}` to signal user is speaking. |
+| `SendUserTurnEnd()` | Client turn mode: stops sending `user_activity` (no explicit message — server detects silence). |
+| `SendInterrupt()` | Ask the agent to stop speaking: sends `{"type":"interrupt"}`. |
+| `GetConnectionState()` | Returns `EElevenLabsConnectionState`. |
+| `GetConversationInfo()` | Returns `FElevenLabsConversationInfo`. |
+
+---
+
+## 7. Data Types Reference
+
+### EElevenLabsConnectionState
+
+```
+Disconnected  — No active connection
+Connecting    — WebSocket handshake in progress / awaiting conversation_initiation_metadata
+Connected     — Conversation active and ready (fires OnAgentConnected)
+Error         — Connection or protocol failure
+```
+
+> Note: State remains `Connecting` until the server sends `conversation_initiation_metadata`. `OnAgentConnected` fires on transition to `Connected`.
+
+### EElevenLabsTurnMode
+
+```
+Server  — ElevenLabs Voice Activity Detection decides when the user stops speaking (recommended)
+Client  — Your code calls StartListening/StopListening to define turns (push-to-talk)
+```
+
+### FElevenLabsConversationInfo
+
+```
+ConversationID  FString  — Unique session ID assigned by ElevenLabs
+AgentID         FString  — The agent ID for this session
+```
+
+### FElevenLabsTranscriptSegment
+
+```
+Text      FString  — Transcribed text
+Speaker   FString  — "user" (agent text comes via OnAgentTextResponse, not transcript)
+bIsFinal  bool     — Always true for user transcripts (ElevenLabs sends final only)
+```
+
+---
+
+## 8. Turn Modes
+
+### Server VAD (default)
+
+ElevenLabs runs Voice Activity Detection on the server. The plugin streams microphone audio continuously and ElevenLabs decides when the user has finished speaking.
+
+**When to use**: Casual conversation, hands-free interaction, natural dialogue.
+
+```
+StartConversation()  →  mic streams continuously (if bAutoStartListening = true)
+                        ElevenLabs detects speech / silence automatically
+                        Agent replies when it detects end-of-speech
+```
+
+### Client Controlled (push-to-talk)
+
+Your code explicitly signals turn boundaries with `StartListening()` / `StopListening()`. The plugin sends `{"type":"user_activity"}` while the user is speaking; stopping it signals end of turn.
+
+**When to use**: Noisy environments, precise control, walkie-talkie style UI.
+
+```
+Input Pressed   →  StartListening()   →  streams audio + sends user_activity
+Input Released  →  StopListening()    →  stops audio (no explicit end message)
+                                         Server detects silence and hands turn to agent
+```
+
+---
+
+## 9. Security — Signed URL Mode
+
+By default, the API key is stored in Project Settings (`DefaultEngine.ini`). This is fine for development but **should not be shipped in packaged builds** as the key could be extracted.
+
+### Production setup
+
+1. Enable **Signed URL Mode** in Project Settings.
+2. Set **Signed URL Endpoint** to a URL on your own backend (e.g. `https://your-server.com/api/elevenlabs-token`).
+3. Your backend authenticates the player and calls the ElevenLabs API to generate a signed WebSocket URL, returning:
+   ```json
+   { "signed_url": "wss://api.elevenlabs.io/v1/convai/conversation?agent_id=...&token=..." }
+   ```
+4. The plugin fetches this URL before connecting — the API key never leaves your server.
+
+### Development workflow (API key in project settings)
+
+- Set the key in **Project Settings → Plugins → ElevenLabs AI Agent**
+- UE saves it to `DefaultEngine.ini` under `[/Script/PS_AI_Agent_ElevenLabs.ElevenLabsSettings]`
+- **Strip this section from `DefaultEngine.ini` before every git commit**
+- Each developer sets the key locally — it does not go in version control
+
+---
+
+## 10. Audio Pipeline
+
+### Input (player → agent)
+
+```
+Device (any sample rate, any channels)
+  ↓  FAudioCapture — UE built-in (UE 5.3+ API: OpenAudioCaptureStream)
+  ↓  Callback: const void* → cast to float32 interleaved frames
+  ↓  Downmix to mono (average all channels)
+  ↓  Resample to 16000 Hz (linear interpolation)
+  ↓  Apply VolumeMultiplier
+  ↓  Dispatch to Game Thread (AsyncTask)
+  ↓  Convert float32 → int16 signed, little-endian bytes
+  ↓  Base64 encode
+  ↓  Send as binary WebSocket frame: { "user_audio_chunk": "<base64>" }
+```
+
+### Output (agent → player)
+
+```
+Binary WebSocket frame arrives
+  ↓  Peek first byte:
+     • '{' → UTF-8 JSON: parse type field, dispatch to handler
+     • other → raw PCM audio bytes
+  ↓  [Audio path] Raw int16 LE PCM bytes at 16000 Hz mono
+  ↓  Enqueue in thread-safe AudioQueue (FCriticalSection)
+  ↓  USoundWaveProcedural::OnSoundWaveProceduralUnderflow pulls from queue
+  ↓  UAudioComponent plays from the Actor's world position (3D spatialized)
+```
+
+**Audio format** (both directions): PCM 16-bit signed, 16000 Hz, mono, little-endian.
+
+### Silence detection heuristic
+
+`OnAgentStoppedSpeaking` fires when the `AudioQueue` has been empty for **30 consecutive ticks** (~0.5 s at 60 fps). If the agent has natural pauses, increase `SilenceThresholdTicks` in the header:
+
+```cpp
+static constexpr int32 SilenceThresholdTicks = 60; // ~1.0s
+```
+
+---
+
+## 11. Common Patterns
+
+### Test the connection without a microphone
+
+```
+BeginPlay → StartConversation()
+
+OnAgentConnected → SendTextMessage("Hello, introduce yourself")
+
+OnAgentTextResponse → Print string (confirms text pipeline works)
+OnAgentStartedSpeaking → (confirms audio pipeline works)
+```
+
+### Show subtitles in UI
+
+```
+OnAgentTranscript:
+  Segment → Text  → show in player subtitle widget (speaker always "user")
+
+OnAgentTextResponse:
+  ResponseText    → show in NPC speech bubble
+```
+
+### Interrupt the agent when the player starts speaking
+
+In Server VAD mode ElevenLabs handles this automatically. For manual control:
+
+```
+OnAgentStartedSpeaking  →  set "agent is speaking" flag
+Input Action (any)      →  if agent is speaking → InterruptAgent()
+```
+
+### Multiple NPCs with different agents
+
+Each NPC Blueprint has its own `UElevenLabsConversationalAgentComponent`. Set a different `AgentID` on each component. WebSocket connections are fully independent.
+
+### Only start the conversation when the player is nearby
+
+```
+On Begin Overlap (trigger volume around NPC)
+  └─► [ElevenLabs Agent] Start Conversation
+
+On End Overlap
+  └─► [ElevenLabs Agent] End Conversation
+```
+
+### Adjust microphone volume
+
+Get the `UElevenLabsMicrophoneCaptureComponent` from the owner and set `VolumeMultiplier`:
+
+```cpp
+UElevenLabsMicrophoneCaptureComponent* Mic =
+    GetOwner()->FindComponentByClass<UElevenLabsMicrophoneCaptureComponent>();
+if (Mic) Mic->VolumeMultiplier = 2.0f;
+```
+
+---
+
+## 12. Troubleshooting
+
+### Plugin doesn't appear in Project Settings
+
+Ensure the plugin is enabled in `.uproject` and the project was recompiled after adding it.
+
+### WebSocket connection fails immediately
+
+- Check the **API Key** is set correctly in Project Settings.
+- Check the **Agent ID** exists in your ElevenLabs account (find it in the dashboard URL or via `GET /v1/convai/agents`).
+- Enable **Verbose Logging** in Project Settings and check Output Log for the exact WS URL and error.
+- Ensure port 443 (WSS) is not blocked by your firewall.
+
+### `OnAgentConnected` never fires
+
+- Connection was made but `conversation_initiation_metadata` not received yet — check Verbose Logging.
+- If you see `"Binary audio frame"` logs but no `"Conversation initiated"` — the initiation JSON frame may be arriving as a non-`{` binary frame. Check the hex prefix logged at Verbose level.
+
+### No audio from the microphone
+
+- Windows may require microphone permission. Check **Settings → Privacy → Microphone**.
+- Try setting `VolumeMultiplier` to `2.0` on the `MicrophoneCaptureComponent`.
+- Check Output Log for `"Failed to open default audio capture stream"`.
+
+### Agent audio is choppy or silent
+
+- The `USoundWaveProcedural` queue may be underflowing due to network jitter. Check latency.
+- Verify the audio format matches: plugin expects raw PCM 16-bit 16 kHz mono from the server. If ElevenLabs sends a different format (e.g. mp3_44100), audio will sound garbled — check `agent_output_audio_format` in the `conversation_initiation_metadata` via Verbose Logging.
+- Ensure no other component is using the same `UAudioComponent`.
+
+### `OnAgentStoppedSpeaking` fires too early
+
+Increase `SilenceThresholdTicks` in `ElevenLabsConversationalAgentComponent.h`:
+
+```cpp
+static constexpr int32 SilenceThresholdTicks = 60; // ~1.0s at 60fps
+```
+
+### Build error: "Plugin AudioCapture not found"
+
+Make sure the `AudioCapture` plugin is enabled. It should be auto-enabled via the `.uplugin` dependency, but you can add it manually to `.uproject`:
+
+```json
+{ "Name": "AudioCapture", "Enabled": true }
+```
+
+### `"Received unexpected binary WebSocket frame"` in the log
+
+This warning no longer appears in v1.1.0+. If you see it, you are running an older build — recompile the plugin.
+
+---
+
+## 13. Changelog
+
+### v1.1.0 — 2026-02-19
+
+**Bug fixes:**
+- **Binary WebSocket frames**: ElevenLabs sends all frames as binary (not text). All frames were previously discarded. Now correctly handled — JSON control frames decoded as UTF-8, raw PCM audio frames routed directly to the audio queue.
+- **Transcript message**: Wrong message type (`"transcript"` → `"user_transcript"`), wrong event key (`"transcript_event"` → `"user_transcription_event"`), wrong text field (`"message"` → `"user_transcript"`).
+- **Pong format**: `event_id` was nested inside a `pong_event` object; corrected to top-level field per API spec.
+- **Client turn mode**: `user_turn_start`/`user_turn_end` are not valid API messages; replaced with `user_activity` (start) and implicit silence (end).
+
+**New features:**
+- `SendTextMessage(Text)` on both `UElevenLabsConversationalAgentComponent` and `UElevenLabsWebSocketProxy` — send text to the agent without a microphone. Useful for testing.
+- Verbose logging shows binary frame hex preview and JSON frame content prefix.
+- Improved JSON parse error log now shows the first 80 characters of the failing message.
+
+### v1.0.0 — 2026-02-19
+
+Initial implementation. Plugin compiles cleanly on UE 5.5 Win64.
+
+---
+
+*Documentation updated 2026-02-19 — Plugin v1.1.0 — UE 5.5*

.claude/elevenlabs_api_reference.md

@@ -0,0 +1,463 @@
+# ElevenLabs Conversational AI – API Reference
+> Saved for Claude Code sessions. Auto-loaded via `.claude/` directory.
+> Last updated: 2026-02-19
+
+---
+
+## 1. Agent ID — Where to Find It
+
+### In the Dashboard (UI)
+1. Go to **https://elevenlabs.io/app/conversational-ai**
+2. Click on your agent to open it
+3. The **Agent ID** is shown in the agent settings page — typically in the URL bar and/or in the agent's "General" settings tab
+   - URL pattern: `https://elevenlabs.io/app/conversational-ai/agents/<AGENT_ID>`
+   - Also visible in the "API" or "Overview" tab of the agent editor (copy button available)
+
+### Via API
+```http
+GET https://api.elevenlabs.io/v1/convai/agents
+xi-api-key: YOUR_API_KEY
+```
+Returns a list of all agents with their `agent_id` strings.
+
+### Via API (single agent)
+```http
+GET https://api.elevenlabs.io/v1/convai/agents/{agent_id}
+xi-api-key: YOUR_API_KEY
+```
+
+### Agent ID Format
+- Type: `string`
+- Returned on agent creation via `POST /v1/convai/agents/create`
+- Used as URL path param and WebSocket query param throughout the API
+
+---
+
+## 2. WebSocket Conversational AI
+
+### Connection URL
+```
+wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<AGENT_ID>
+```
+
+Regional alternatives:
+| Region | URL |
+|--------|-----|
+| Default (Global) | `wss://api.elevenlabs.io/` |
+| US | `wss://api.us.elevenlabs.io/` |
+| EU | `wss://api.eu.residency.elevenlabs.io/` |
+| India | `wss://api.in.residency.elevenlabs.io/` |
+
+### Authentication
+- **Public agents**: No key required, just `agent_id` query param
+- **Private agents**: Use a **Signed URL** (see Section 4) instead of direct `agent_id`
+- **Server-side** (backend): Pass `xi-api-key` as an HTTP upgrade header
+
+```
+Headers:
+  xi-api-key: YOUR_API_KEY
+```
+
+> ⚠️ Never expose your API key client-side. For browser/mobile apps, use Signed URLs.
+
+---
+
+## 3. WebSocket Protocol — Message Reference
+
+### Audio Format
+- **Input (mic → server)**: PCM 16-bit signed, **16000 Hz**, mono, little-endian, Base64-encoded
+- **Output (server → client)**: Base64-encoded audio (format specified in `conversation_initiation_metadata`)
+
+---
+
+### Messages FROM Server (Subscribe / Receive)
+
+#### `conversation_initiation_metadata`
+Sent immediately after connection. Contains conversation ID and audio format specs.
+```json
+{
+  "type": "conversation_initiation_metadata",
+  "conversation_initiation_metadata_event": {
+    "conversation_id": "string",
+    "agent_output_audio_format": "pcm_16000 | mp3_44100 | ...",
+    "user_input_audio_format": "pcm_16000"
+  }
+}
+```
+
+#### `audio`
+Agent speech audio chunk.
+```json
+{
+  "type": "audio",
+  "audio_event": {
+    "audio_base_64": "BASE64_PCM_BYTES",
+    "event_id": 42
+  }
+}
+```
+
+#### `user_transcript`
+Transcribed text of what the user said.
+```json
+{
+  "type": "user_transcript",
+  "user_transcription_event": {
+    "user_transcript": "Hello, how are you?"
+  }
+}
+```
+
+#### `agent_response`
+The text the agent is saying (arrives in parallel with audio).
+```json
+{
+  "type": "agent_response",
+  "agent_response_event": {
+    "agent_response": "I'm doing great, thanks!"
+  }
+}
+```
+
+#### `agent_response_correction`
+Sent after an interruption — shows what was truncated.
+```json
+{
+  "type": "agent_response_correction",
+  "agent_response_correction_event": {
+    "original_agent_response": "string",
+    "corrected_agent_response": "string"
+  }
+}
+```
+
+#### `interruption`
+Signals that a specific audio event was interrupted.
+```json
+{
+  "type": "interruption",
+  "interruption_event": {
+    "event_id": 42
+  }
+}
+```
+
+#### `ping`
+Keepalive ping from server. Client must reply with `pong`.
+```json
+{
+  "type": "ping",
+  "ping_event": {
+    "event_id": 1,
+    "ping_ms": 150
+  }
+}
+```
+
+#### `client_tool_call`
+Requests the client execute a tool (custom tools integration).
+```json
+{
+  "type": "client_tool_call",
+  "client_tool_call": {
+    "tool_name": "string",
+    "tool_call_id": "string",
+    "parameters": {}
+  }
+}
+```
+
+#### `contextual_update`
+Text context added to conversation state (non-interrupting).
+```json
+{
+  "type": "contextual_update",
+  "contextual_update_event": {
+    "text": "string"
+  }
+}
+```
+
+#### `vad_score`
+Voice Activity Detection confidence score (0.0–1.0).
+```json
+{
+  "type": "vad_score",
+  "vad_score_event": {
+    "vad_score": 0.85
+  }
+}
+```
+
+#### `internal_tentative_agent_response`
+Preliminary agent text during LLM generation (not final).
+```json
+{
+  "type": "internal_tentative_agent_response",
+  "tentative_agent_response_internal_event": {
+    "tentative_agent_response": "string"
+  }
+}
+```
+
+---
+
+### Messages TO Server (Publish / Send)
+
+#### `user_audio_chunk`
+Microphone audio data. Send continuously during user speech.
+```json
+{
+  "user_audio_chunk": "BASE64_PCM_16BIT_16KHZ_MONO"
+}
+```
+Audio must be: **PCM 16-bit signed, 16000 Hz, mono, little-endian**, then Base64-encoded.
+
+#### `pong`
+Reply to server `ping` to keep connection alive.
+```json
+{
+  "type": "pong",
+  "event_id": 1
+}
+```
+
+#### `conversation_initiation_client_data`
+Override agent configuration at connection time. Send before or just after connecting.
+```json
+{
+  "type": "conversation_initiation_client_data",
+  "conversation_config_override": {
+    "agent": {
+      "prompt": { "prompt": "Custom system prompt override" },
+      "first_message": "Hello! How can I help?",
+      "language": "en"
+    },
+    "tts": {
+      "voice_id": "string",
+      "speed": 1.0,
+      "stability": 0.5,
+      "similarity_boost": 0.75
+    }
+  },
+  "dynamic_variables": {
+    "user_name": "Alice",
+    "session_id": 12345
+  }
+}
+```
+
+Config override ranges:
+- `tts.speed`: 0.7 – 1.2
+- `tts.stability`: 0.0 – 1.0
+- `tts.similarity_boost`: 0.0 – 1.0
+
+#### `client_tool_result`
+Response to a `client_tool_call` from the server.
+```json
+{
+  "type": "client_tool_result",
+  "tool_call_id": "string",
+  "result": "tool output string",
+  "is_error": false
+}
+```
+
+#### `contextual_update`
+Inject context without interrupting the conversation.
+```json
+{
+  "type": "contextual_update",
+  "text": "User just entered room 4B"
+}
+```
+
+#### `user_message`
+Send a text message (no mic audio needed).
+```json
+{
+  "type": "user_message",
+  "text": "What is the weather like?"
+}
+```
+
+#### `user_activity`
+Signal that user is active (for turn detection in client mode).
+```json
+{
+  "type": "user_activity"
+}
+```
+
+---
+
+## 4. Signed URL (Private Agents)
+
+Used for browser/mobile clients to authenticate without exposing the API key.
+
+### Flow
+1. **Backend** calls ElevenLabs API to get a temporary signed URL
+2. Backend returns signed URL to client
+3. **Client** opens WebSocket to the signed URL (no API key needed)
+
+### Get Signed URL
+```http
+GET https://api.elevenlabs.io/v1/convai/conversation/get-signed-url?agent_id=<AGENT_ID>
+xi-api-key: YOUR_API_KEY
+```
+
+Optional query params:
+- `include_conversation_id=true` — generates unique conversation ID, prevents URL reuse
+- `branch_id` — specific agent branch
+
+Response:
+```json
+{
+  "signed_url": "wss://api.elevenlabs.io/v1/convai/conversation?agent_id=...&token=..."
+}
+```
+
+Client connects to `signed_url` directly — no headers needed.
+
+---
+
+## 5. Agents REST API
+
+Base URL: `https://api.elevenlabs.io`
+Auth header: `xi-api-key: YOUR_API_KEY`
+
+### Create Agent
+```http
+POST /v1/convai/agents/create
+Content-Type: application/json
+
+{
+  "name": "My NPC Agent",
+  "conversation_config": {
+    "agent": {
+      "first_message": "Hello adventurer!",
+      "prompt": { "prompt": "You are a wise tavern keeper in a fantasy world." },
+      "language": "en"
+    }
+  }
+}
+```
+Response includes `agent_id`.
+
+### List Agents
+```http
+GET /v1/convai/agents?page_size=30&search=&sort_by=created_at&sort_direction=desc
+```
+Response:
+```json
+{
+  "agents": [
+    {
+      "agent_id": "abc123xyz",
+      "name": "My NPC Agent",
+      "created_at_unix_secs": 1708300000,
+      "last_call_time_unix_secs": null,
+      "archived": false,
+      "tags": []
+    }
+  ],
+  "has_more": false,
+  "next_cursor": null
+}
+```
+
+### Get Agent
+```http
+GET /v1/convai/agents/{agent_id}
+```
+
+### Update Agent
+```http
+PATCH /v1/convai/agents/{agent_id}
+Content-Type: application/json
+{ "name": "Updated Name", "conversation_config": { ... } }
+```
+
+### Delete Agent
+```http
+DELETE /v1/convai/agents/{agent_id}
+```
+
+---
+
+## 6. Turn Modes
+
+### Server VAD (Default / Recommended)
+- ElevenLabs server detects when user stops speaking
+- Client streams audio continuously
+- Server handles all turn-taking automatically
+
+### Client Turn Mode
+- Client explicitly signals turn boundaries
+- Send `user_activity` to indicate user is speaking
+- Use when you have your own VAD or push-to-talk UI
+
+---
+
+## 7. Audio Pipeline (UE5 Implementation Notes)
+
+```
+Microphone (FAudioCapture)
+  → float32 samples at device rate (e.g. 44100 Hz stereo)
+  → Resample to 16000 Hz mono
+  → Convert float32 → int16 little-endian
+  → Base64-encode
+  → Send as {"user_audio_chunk": "BASE64"}
+
+Server → {"type":"audio","audio_event":{"audio_base_64":"BASE64"}}
+  → Base64-decode
+  → Raw PCM bytes
+  → Push to USoundWaveProcedural
+  → UAudioComponent plays back
+```
+
+### Float32 → Int16 Conversion (C++)
+```cpp
+static TArray<uint8> FloatPCMToInt16Bytes(const TArray<float>& FloatSamples)
+{
+    TArray<uint8> Bytes;
+    Bytes.SetNumUninitialized(FloatSamples.Num() * 2);
+    for (int32 i = 0; i < FloatSamples.Num(); i++)
+    {
+        float Clamped = FMath::Clamp(FloatSamples[i], -1.f, 1.f);
+        int16 Sample = (int16)(Clamped * 32767.f);
+        Bytes[i * 2]     = (uint8)(Sample & 0xFF);        // Low byte
+        Bytes[i * 2 + 1] = (uint8)((Sample >> 8) & 0xFF); // High byte
+    }
+    return Bytes;
+}
+```
+
+---
+
+## 8. Quick Integration Checklist (UE5 Plugin)
+
+- [ ] Set `AgentID` in `UElevenLabsSettings` (Project Settings → ElevenLabs AI Agent)
+  - Or override per-component via `UElevenLabsConversationalAgentComponent::AgentID`
+- [ ] Set `API_Key` in settings (or leave empty for public agents)
+- [ ] Add `UElevenLabsConversationalAgentComponent` to your NPC actor
+- [ ] Set `TurnMode` (default: `Server` — recommended)
+- [ ] Bind to events: `OnAgentConnected`, `OnAgentTranscript`, `OnAgentTextResponse`, `OnAgentStartedSpeaking`, `OnAgentStoppedSpeaking`
+- [ ] Call `StartConversation()` to begin
+- [ ] Call `EndConversation()` when done
+
+---
+
+## 9. Key API URLs Reference
+
+| Purpose | URL |
+|---------|-----|
+| Dashboard | https://elevenlabs.io/app/conversational-ai |
+| API Keys | https://elevenlabs.io/app/settings/api-keys |
+| WebSocket endpoint | wss://api.elevenlabs.io/v1/convai/conversation |
+| Agents list | GET https://api.elevenlabs.io/v1/convai/agents |
+| Agent by ID | GET https://api.elevenlabs.io/v1/convai/agents/{agent_id} |
+| Create agent | POST https://api.elevenlabs.io/v1/convai/agents/create |
+| Signed URL | GET https://api.elevenlabs.io/v1/convai/conversation/get-signed-url |
+| WS protocol docs | https://elevenlabs.io/docs/eleven-agents/api-reference/eleven-agents/websocket |
+| Quickstart | https://elevenlabs.io/docs/eleven-agents/quickstart |

.claude/elevenlabs_plugin.md

@@ -0,0 +1,61 @@
+# PS_AI_Agent_ElevenLabs Plugin
+
+## Location
+`Unreal/PS_AI_Agent/Plugins/PS_AI_Agent_ElevenLabs/`
+
+## File Map
+```
+PS_AI_Agent_ElevenLabs.uplugin
+Source/PS_AI_Agent_ElevenLabs/
+  PS_AI_Agent_ElevenLabs.Build.cs
+  Public/
+    PS_AI_Agent_ElevenLabs.h                  – FPS_AI_Agent_ElevenLabsModule + UElevenLabsSettings
+    ElevenLabsDefinitions.h                   – Enums, structs, ElevenLabsMessageType/Audio constants
+    ElevenLabsWebSocketProxy.h/.cpp           – UObject managing one WS session
+    ElevenLabsConversationalAgentComponent.h/.cpp  – Main ActorComponent (attach to NPC)
+    ElevenLabsMicrophoneCaptureComponent.h/.cpp    – Mic capture, resample, dispatch to game thread
+  Private/
+    (implementations of the above)
+```
+
+## ElevenLabs Conversational AI Protocol
+- **WebSocket URL**: `wss://api.elevenlabs.io/v1/convai/conversation?agent_id=<ID>`
+- **Auth**: HTTP upgrade header `xi-api-key: <key>` (set in Project Settings)
+- **All frames**: JSON text (no binary frames used by the API)
+- **Audio format**: PCM 16-bit signed, 16000 Hz, mono, little-endian — Base64-encoded in JSON
+
+### Client → Server messages
+| Type field value | Payload |
+|---|---|
+| *(none – key is the type)* `user_audio_chunk` | `{ "user_audio_chunk": "<base64 PCM>" }` |
+| `user_turn_start` | `{ "type": "user_turn_start" }` |
+| `user_turn_end` | `{ "type": "user_turn_end" }` |
+| `interrupt` | `{ "type": "interrupt" }` |
+| `pong` | `{ "type": "pong", "pong_event": { "event_id": N } }` |
+
+### Server → Client messages (field: `type`)
+| type value | Key nested object | Notes |
+|---|---|---|
+| `conversation_initiation_metadata` | `conversation_initiation_metadata_event.conversation_id` | Marks WS ready |
+| `audio` | `audio_event.audio_base_64` | Base64 PCM from agent |
+| `transcript` | `transcript_event.{speaker, message, is_final}` | User or agent speech |
+| `agent_response` | `agent_response_event.agent_response` | Final agent text |
+| `interruption` | — | Agent stopped mid-sentence |
+| `ping` | `ping_event.event_id` | Must reply with pong |
+
+## Key Design Decisions
+- **No gRPC / no ThirdParty libs** — pure UE WebSockets + HTTP, builds out of the box
+- Audio resampled in-plugin: device rate → 16000 Hz mono (linear interpolation)
+- `USoundWaveProcedural` for real-time agent audio playback (queue-driven)
+- Silence heuristic: 30 game-thread ticks (~0.5 s at 60 fps) with no new audio → agent done speaking
+- `bSignedURLMode` setting: fetch a signed WS URL from your own backend (keeps API key off client)
+- Two turn modes: `Server VAD` (ElevenLabs detects speech end) and `Client Controlled` (push-to-talk)
+
+## Build Dependencies (Build.cs)
+Core, CoreUObject, Engine, InputCore, Json, JsonUtilities, WebSockets, HTTP,
+AudioMixer, AudioCaptureCore, AudioCapture, Voice, SignalProcessing
+
+## Status
+- **Session 1** (2026-02-19): All source files written, registered in .uproject. Not yet compiled.
+- **TODO**: Open in UE 5.5 Editor → compile → test basic WS connection with a test agent ID.
+- **Watch out**: Verify `USoundWaveProcedural::OnSoundWaveProceduralUnderflow` delegate signature vs UE 5.5 API.

.claude/project_context.md

@@ -0,0 +1,79 @@
+# Project Context & Original Ask
+
+## What the user wants to build
+
+A **UE5 plugin** that integrates the **ElevenLabs Conversational AI Agent** API into Unreal Engine 5.5,
+allowing an in-game NPC (or any Actor) to hold a real-time voice conversation with a player.
+
+### The original request (paraphrased)
+> "I want to create a plugin to use ElevenLabs Conversational Agent in Unreal Engine 5.5.
+> I previously used the Convai plugin which does what I want, but I prefer ElevenLabs quality.
+> The goal is to create a plugin in the existing Unreal Project to make a first step for integration.
+> Convai AI plugin may be too big in terms of functionality for the new project, but it is the final goal.
+> You can use the Convai source code to find the right way to make the ElevenLabs version —
+> it should be very similar."
+
+### Plugin name
+`PS_AI_Agent_ElevenLabs`
+
+---
+
+## User's mental model / intent
+
+1. **Short-term**: A working first-step plugin — minimal but functional — that can:
+   - Connect to ElevenLabs Conversational AI via WebSocket
+   - Capture microphone audio from the player
+   - Stream it to ElevenLabs in real time
+   - Play back the agent's voice response
+   - Surface key events (transcript, agent text, speaking state) to Blueprint
+
+2. **Long-term**: Match the full feature set of Convai — character IDs, session memory,
+   actions/environment context, lip-sync, etc. — but powered by ElevenLabs instead.
+
+3. **Key preference**: Simpler than Convai. No gRPC, no protobuf, no ThirdParty precompiled
+   libraries. ElevenLabs' Conversational AI API uses plain WebSocket + JSON, which maps
+   naturally to UE's built-in `WebSockets` module.
+
+---
+
+## How we used Convai as a reference
+
+We studied the Convai plugin source (`ConvAI/Convai/`) to understand:
+- **Module structure**: `UConvaiSettings` + `IModuleInterface` + `ISettingsModule` registration
+- **Audio capture pattern**: `Audio::FAudioCapture`, ring buffers, thread-safe dispatch to game thread
+- **Audio playback pattern**: `USoundWaveProcedural` fed from a queue
+- **Component architecture**: `UConvaiChatbotComponent` (NPC side) + `UConvaiPlayerComponent` (player side)
+- **HTTP proxy pattern**: `UConvaiAPIBaseProxy` base class for async REST calls
+- **Voice type enum**: Convai already had `EVoiceType::ElevenLabsVoices` — confirming ElevenLabs
+  is a natural fit
+
+We then replaced gRPC/protobuf with **WebSocket + JSON** to match the ElevenLabs API, and
+simplified the architecture to the minimum needed for a first working version.
+
+---
+
+## What was built (Session 1 — 2026-02-19)
+
+All source files created and registered. See `.claude/elevenlabs_plugin.md` for full file map and protocol details.
+
+### Components created
+| Class | Role |
+|---|---|
+| `UElevenLabsSettings` | Project Settings UI — API key, Agent ID, security options |
+| `UElevenLabsWebSocketProxy` | Manages one WS session: connect, send audio, handle all server message types |
+| `UElevenLabsConversationalAgentComponent` | ActorComponent to attach to any NPC — orchestrates mic + WS + playback |
+| `UElevenLabsMicrophoneCaptureComponent` | Wraps `Audio::FAudioCapture`, resamples to 16 kHz mono |
+
+### Not yet done (next sessions)
+- Compile & test in UE 5.5 Editor
+- Verify `USoundWaveProcedural::OnSoundWaveProceduralUnderflow` delegate signature for UE 5.5
+- Add lip-sync support (future)
+- Add session memory / conversation history (future)
+- Add environment/action context support (future, matching Convai's full feature set)
+
+---
+
+## Notes on the ElevenLabs API
+- Docs: https://elevenlabs.io/docs/conversational-ai
+- Create agents at: https://elevenlabs.io/app/conversational-ai
+- API keys at: https://elevenlabs.io (dashboard)

.claude/session_log_2026-02-19.md

@@ -0,0 +1,351 @@
+# Session Log — 2026-02-19
+
+**Project**: PS_AI_Agent (Unreal Engine 5.5)
+**Machine**: Desktop PC (j_foucher)
+**Working directory**: `E:\ASTERION\GIT\PS_AI_Agent`
+
+---
+
+## Conversation Summary
+
+### 1. Initial Request
+User asked to create a plugin to use the ElevenLabs Conversational AI Agent in UE5.5.
+Reference: existing Convai plugin (gRPC-based, more complex). Goal: simpler version using ElevenLabs.
+Plugin name requested: `PS_AI_Agent_ElevenLabs`.
+
+### 2. Codebase Exploration
+Explored the Convai plugin source at `ConvAI/Convai/` to understand:
+- Module/settings structure
+- AudioCapture patterns
+- HTTP proxy pattern
+- gRPC streaming architecture (to know what to replace with WebSocket)
+- Convai already had `EVoiceType::ElevenLabsVoices` — confirming the direction
+
+### 3. Plugin Created
+All source files written from scratch under:
+`Unreal/PS_AI_Agent/Plugins/PS_AI_Agent_ElevenLabs/`
+
+Files created:
+- `PS_AI_Agent_ElevenLabs.uplugin`
+- `PS_AI_Agent_ElevenLabs.Build.cs`
+- `Public/PS_AI_Agent_ElevenLabs.h` — Module + `UElevenLabsSettings`
+- `Public/ElevenLabsDefinitions.h` — Enums, structs, protocol constants
+- `Public/ElevenLabsWebSocketProxy.h` + `.cpp` — WS session manager
+- `Public/ElevenLabsConversationalAgentComponent.h` + `.cpp` — Main NPC component
+- `Public/ElevenLabsMicrophoneCaptureComponent.h` + `.cpp` — Mic capture
+- `PS_AI_Agent.uproject` — Plugin registered
+
+Commit: `f0055e8`
+
+### 4. Memory Files Created
+To allow context recovery on any machine (including laptop):
+- `.claude/MEMORY.md` — project structure + patterns (auto-loaded by Claude Code)
+- `.claude/elevenlabs_plugin.md` — plugin file map + API protocol details
+- `.claude/project_context.md` — original ask, intent, short/long-term goals
+- Local copy also at `C:\Users\j_foucher\.claude\projects\...\memory\`
+
+Commit: `f0055e8` (with plugin), updated in `4d6ae10`
+
+### 5. .gitignore Updated
+Added to existing ignores:
+- `Unreal/PS_AI_Agent/Plugins/*/Binaries/`
+- `Unreal/PS_AI_Agent/Plugins/*/Intermediate/`
+- `Unreal/PS_AI_Agent/*.sln` / `*.suo`
+- `.claude/settings.local.json`
+- `generate_pptx.py`
+
+Commit: `4d6ae10`, `b114ab0`
+
+### 6. Compile — First Attempt (Errors Found)
+Ran `Build.bat PS_AI_AgentEditor Win64 Development`. Errors:
+- `WebSockets` listed in `.uplugin` — it's a module not a plugin → removed
+- `OpenDefaultCaptureStream` doesn't exist in UE 5.5 → use `OpenAudioCaptureStream`
+- `FOnAudioCaptureFunction` callback uses `const void*` not `const float*` → fixed cast
+- `TArray::RemoveAt(0, N, false)` deprecated → use `EAllowShrinking::No`
+- `AudioCapture` is a plugin and must be in `.uplugin` Plugins array → added
+
+Commit: `bb1a857`
+
+### 7. Compile — Success
+Clean build, no warnings, no errors.
+Output: `Plugins/PS_AI_Agent_ElevenLabs/Binaries/Win64/UnrealEditor-PS_AI_Agent_ElevenLabs.dll`
+
+Memory updated with confirmed UE 5.5 API patterns. Commit: `3b98edc`
+
+### 8. Documentation — Markdown
+Full reference doc written to `.claude/PS_AI_Agent_ElevenLabs_Documentation.md`:
+- Installation, Project Settings, Quick Start (BP + C++), Components Reference,
+  Data Types, Turn Modes, Security/Signed URL, Audio Pipeline, Common Patterns, Troubleshooting.
+
+Commit: `c833ccd`
+
+### 9. Documentation — PowerPoint
+20-slide dark-themed PowerPoint generated via Python (python-pptx 1.0.2):
+- File: `PS_AI_Agent_ElevenLabs_Documentation.pptx` in repo root
+- Covers all sections with visual layout, code blocks, flow diagrams, colour-coded elements
+- Generator script `generate_pptx.py` excluded from git via .gitignore
+
+Commit: `1b72026`
+
+---
+
+## Session 2 — 2026-02-19 (continued context)
+
+### 10. API vs Implementation Cross-Check (3 bugs found and fixed)
+Cross-referenced `elevenlabs_api_reference.md` against plugin source. Found 3 protocol bugs:
+
+**Bug 1 — Transcript fields wrong:**
+- Type: `"transcript"` → `"user_transcript"`
+- Event key: `"transcript_event"` → `"user_transcription_event"`
+- Field: `"message"` → `"user_transcript"`
+
+**Bug 2 — Pong format wrong:**
+- `event_id` was nested in `pong_event{}` → must be top-level
+
+**Bug 3 — Client turn mode messages don't exist:**
+- `"user_turn_start"` / `"user_turn_end"` are not valid API types
+- Replaced: start → `"user_activity"`, end → no-op (server detects silence)
+
+Commit: `ae2c9b9`
+
+### 11. SendTextMessage Added
+User asked for text input to agent for testing (without mic).
+Added `SendTextMessage(FString)` to `UElevenLabsWebSocketProxy` and `UElevenLabsConversationalAgentComponent`.
+Sends `{"type":"user_message","text":"..."}` — agent replies with audio + text.
+
+Commit: `b489d11`
+
+### 12. Binary WebSocket Frame Fix
+User reported: `"Received unexpected binary WebSocket frame"` warnings.
+Root cause: ElevenLabs sends **ALL WebSocket frames as binary**, never text.
+`OnMessage` (text handler) never fires. `OnRawMessage` must handle everything.
+
+Fix: Implemented `OnWsBinaryMessage` with fragment reassembly (`BinaryFrameBuffer`).
+
+Commit: `669c503`
+
+### 13. JSON vs PCM Discrimination Fix
+After binary fix: `"Failed to parse WebSocket message as JSON"` errors.
+Root cause: Binary frames contain BOTH JSON control messages AND raw PCM audio.
+
+Fix: Peek at byte[0] of assembled buffer:
+- `'{'` (0x7B) → UTF-8 JSON → route to `OnWsMessage()`
+- anything else → raw PCM audio → broadcast to `OnAudioReceived`
+
+Commit: `4834567`
+
+### 14. Documentation Updated to v1.1.0
+Full rewrite of `.claude/PS_AI_Agent_ElevenLabs_Documentation.md`:
+- Added Changelog section (v1.0.0 / v1.1.0)
+- Updated audio pipeline (binary PCM path, not Base64 JSON)
+- Added `SendTextMessage` to all function tables and examples
+- Corrected turn mode docs, transcript docs, `OnAgentConnected` timing
+- New troubleshooting entries
+
+Commit: `e464cfe`
+
+### 15. Test Blueprint Asset Updated
+`test_AI_Actor.uasset` updated in UE Editor.
+
+Commit: `99017f4`
+
+---
+
+## Git History (this session)
+
+| Hash | Message |
+|------|---------|
+| `f0055e8` | Add PS_AI_Agent_ElevenLabs plugin (initial implementation) |
+| `4d6ae10` | Update .gitignore: exclude plugin build artifacts and local Claude settings |
+| `b114ab0` | Broaden .gitignore: use glob for all plugin Binaries/Intermediate |
+| `bb1a857` | Fix compile errors in PS_AI_Agent_ElevenLabs plugin |
+| `3b98edc` | Update memory: document confirmed UE 5.5 API patterns and plugin compile status |
+| `c833ccd` | Add plugin documentation for PS_AI_Agent_ElevenLabs |
+| `1b72026` | Add PowerPoint documentation and update .gitignore |
+| `bbeb429` | ElevenLabs API reference doc |
+| `dbd6161` | TestMap, test actor, DefaultEngine.ini, memory update |
+| `ae2c9b9` | Fix 3 WebSocket protocol bugs |
+| `b489d11` | Add SendTextMessage |
+| `669c503` | Fix binary WebSocket frames |
+| `4834567` | Fix JSON vs binary frame discrimination |
+| `e464cfe` | Update documentation to v1.1.0 |
+| `99017f4` | Update test_AI_Actor blueprint asset |
+
+---
+
+## Key Technical Decisions Made This Session
+
+| Decision | Reason |
+|----------|--------|
+| WebSocket instead of gRPC | ElevenLabs Conversational AI uses WS/JSON; no ThirdParty libs needed |
+| `AudioCapture` in `.uplugin` Plugins array | It's an engine plugin, not a module — UBT requires it declared |
+| `WebSockets` in Build.cs only | It's a module (no `.uplugin` file), declaring it in `.uplugin` causes build error |
+| `FOnAudioCaptureFunction` uses `const void*` | UE 5.3+ API change — must cast to `float*` inside callback |
+| `EAllowShrinking::No` | Bool overload of `RemoveAt` deprecated in UE 5.5 |
+| `USoundWaveProcedural` for playback | Allows pushing raw PCM bytes at runtime without file I/O |
+| Silence threshold = 30 ticks | ~0.5s at 60fps heuristic to detect agent finished speaking |
+| Binary frame handling | ElevenLabs sends ALL WS frames as binary; peek byte[0] to discriminate JSON vs PCM |
+| `user_activity` for client turn | `user_turn_start`/`user_turn_end` don't exist in ElevenLabs API |
+
+---
+
+---
+
+## Session 3 — 2026-02-19 (bug fixes from live testing)
+
+### 16. Three Runtime Bugs Fixed (v1.2.0)
+
+User reported after live testing:
+1. **AI speaks twice** — every audio response played double
+2. **Cannot speak** — mic capture didn't reach ElevenLabs
+3. **Latency** — requested `enable_intermediate_response: true`
+
+**Bug 1 Root Cause — Double Audio:**
+UE's libwebsockets backend fires **both** `OnMessage()` (text callback) **and** `OnRawMessage()` (binary callback) for the same incoming frame.
+We had bound both `WebSocket->OnMessage()` and `WebSocket->OnRawMessage()` in `Connect()`.
+Result: every audio frame was decoded and enqueued twice → played twice.
+
+Fix: **Remove `OnMessage` binding entirely.** `OnRawMessage` now handles all frames (JSON control messages peeked via first byte, raw PCM otherwise).
+
+**Bug 2 Root Cause — Mic Silent:**
+ElevenLabs requires a `conversation_initiation_client_data` message sent **immediately** after the WebSocket handshake completes. Without it, the server never enters a state where it will accept and process client audio chunks. This is a required session negotiation step, not optional.
+
+Fix: Send `conversation_initiation_client_data` in `OnWsConnected()` before any other message.
+
+**Bug 2 Secondary — Delegate Stacking:**
+`StartListening()` called `Mic->OnAudioCaptured.AddUObject(this, ...)` without first removing existing bindings. If called more than once (e.g. after reconnect), delegates stack up and audio is sent multiple times per frame.
+
+Fix: Add `Mic->OnAudioCaptured.RemoveAll(this)` before `AddUObject` in `StartListening()`.
+
+**Bug 3 — Latency:**
+Added `"enable_intermediate_response": true` inside `custom_llm_extra_body` of the `conversation_initiation_client_data` message. Also added `optimize_streaming_latency: 3` in `conversation_config_override.tts`.
+
+**Files changed:**
+- `ElevenLabsWebSocketProxy.cpp`:
+  - `Connect()`: removed `OnMessage` binding
+  - `OnWsConnected()`: now sends full `conversation_initiation_client_data` JSON
+- `ElevenLabsConversationalAgentComponent.cpp`:
+  - `StartListening()`: added `RemoveAll` guard before delegate binding
+
+---
+
+---
+
+## Session 4 — 2026-02-19 (mic still silent — push-to-talk deeper investigation)
+
+### 17. Two More Bugs Found and Fixed (v1.3.0)
+
+User confirmed Bug 1 (double audio) was fixed. Bug 2 (cannot speak) persisted.
+
+**Analysis of log:**
+- Blueprint is correct: T Pressed → StartListening, T Released → StopListening (proper push-to-talk)
+- Mic opens and closes correctly — audio capture IS happening
+- Server never responds to mic input → audio reaching ElevenLabs but being ignored
+
+**Bug A — TurnMode mismatch in conversation_initiation_client_data:**
+`OnWsConnected()` hardcoded `"mode": "server_vad"` in the init message regardless of the
+component's `TurnMode` setting. User's Blueprint uses Client turn mode (push-to-talk),
+so the server was configured for server_vad while the client sent client_vad audio signals.
+
+Fix: Read `TurnMode` field on the proxy (set from the component before `Connect()`).
+Translate `EElevenLabsTurnMode::Client` → `"client_vad"`, Server → `"server_vad"`.
+
+**Bug B — user_activity never sent continuously:**
+In client VAD mode, ElevenLabs requires `user_activity` to be sent **continuously**
+alongside every audio chunk to keep the server's VAD aware the user is speaking.
+`SendUserTurnStart()` sent it once on key press, but never again during speech.
+Server-side, without continuous `user_activity`, the server treated the audio as noise.
+
+Fix: In `SendAudioChunk()`, automatically send `user_activity` before each audio chunk
+when `TurnMode == Client`. This keeps the signal continuous for the full duration of speech.
+When the user releases T, `StopListening()` stops the mic → audio stops → `user_activity`
+stops → server detects silence and triggers the agent response.
+
+**Bug C — TurnMode not propagated to proxy:**
+`UElevenLabsConversationalAgentComponent` never told the proxy what TurnMode to use.
+Added `WebSocketProxy->TurnMode = TurnMode` before `Connect()` in `StartConversation()`.
+
+**Files changed:**
+- `ElevenLabsWebSocketProxy.h`: added `public TurnMode` field
+- `ElevenLabsWebSocketProxy.cpp`:
+  - `OnWsConnected()`: use `TurnMode` to set correct mode string in init message
+  - `SendAudioChunk()`: auto-send `user_activity` before each chunk in Client mode
+- `ElevenLabsConversationalAgentComponent.cpp`:
+  - `StartConversation()`: set `WebSocketProxy->TurnMode = TurnMode` before `Connect()`
+
+---
+
+---
+
+## Session 5 — 2026-02-19 (still can't speak — bAutoStartListening conflict)
+
+### 18. Root Cause Found and Fixed (v1.4.0)
+
+Log analysis revealed the true root cause:
+
+**Exact sequence:**
+```
+OnConnected → bAutoStartListening=true → StartListening() → bIsListening=true, mic opens
+OnAgentStoppedSpeaking → Blueprint calls StartListening() → bIsListening guard → no-op (already open)
+User presses T → StartListening() → bIsListening guard → no-op
+User releases T → StopListening() → bIsListening=false, mic CLOSES
+User presses T → StartListening() → NOW opens mic (was closed)
+User releases T → StopListening() → mic closes — but ElevenLabs never got audio
+```
+
+**Root cause:** `bAutoStartListening = true` opens the mic on connect and sets `bIsListening = true`.
+In Client/push-to-talk mode, every T-press hits the `bIsListening` guard and does nothing.
+Every T-release closes the auto-started mic. The mic was never open during actual speech.
+
+**Fix:** `HandleConnected()` now only calls `StartListening()` when `TurnMode == Server`.
+In Client mode, `bAutoStartListening` is ignored — the user controls listening via T key.
+
+**File changed:**
+- `ElevenLabsConversationalAgentComponent.cpp`:
+  - `HandleConnected()`: guard `bAutoStartListening` with `TurnMode == Server` check
+
+---
+
+---
+
+## Session 6 — 2026-02-19 (audio chunk size fix)
+
+### 19. Mic Audio Chunk Accumulation (v1.5.0)
+
+**Root cause (from diagnostic log in Session 5):**
+Log showed hundreds of `SendAudioChunk: 158 bytes (TurnMode=Client)` lines with zero server responses.
+- 158 bytes = 79 samples = ~5ms of audio at 16kHz 16-bit mono
+- WASAPI (Windows Audio Session API) fires the `FAudioCapture` callback at its internal buffer period (~5ms)
+- ElevenLabs requires a minimum chunk size for its VAD and STT to operate (~100ms / 3200 bytes)
+- Tiny 5ms fragments arrived at the server but were silently ignored → agent never responded
+
+**Fix applied:**
+Added `MicAccumulationBuffer TArray<uint8>` to `UElevenLabsConversationalAgentComponent`.
+`OnMicrophoneDataCaptured()` appends each callback's converted bytes and only calls `SendAudioChunk`
+when `>= MicChunkMinBytes` (3200 bytes = 100ms) have accumulated.
+
+`StopListening()` flushes any remaining bytes in the buffer before sending `SendUserTurnEnd()`,
+so the last partial chunk of speech is never dropped.
+
+`HandleDisconnected()` clears the buffer to prevent stale data on reconnect.
+
+**Files changed:**
+- `ElevenLabsConversationalAgentComponent.h`: added `MicAccumulationBuffer` + `MicChunkMinBytes = 3200`
+- `ElevenLabsConversationalAgentComponent.cpp`:
+  - `OnMicrophoneDataCaptured()`: accumulate → send when threshold reached
+  - `StopListening()`: flush remainder before end-of-turn signal
+  - `HandleDisconnected()`: clear accumulation buffer
+
+Commit: `91cf5b1`
+
+---
+
+## Next Steps (not done yet)
+
+- [ ] Test v1.5.0 in Editor — verify push-to-talk mic works end-to-end (should be the final fix)
+- [ ] Test `USoundWaveProcedural` underflow behaviour in practice (check for audio glitches)
+- [ ] Test `SendTextMessage` end-to-end in Blueprint
+- [ ] Add lip-sync support (future)
+- [ ] Add session memory / conversation history (future, matching Convai)
+- [ ] Add environment/action context support (future)
+- [ ] Consider Signed URL Mode backend implementation

.gitattributes

@@ -0,0 +1,30 @@
+# Force consistent line endings across all machines
+# LF in the repo, native (CRLF on Windows) in working tree
+* text=auto
+
+# Source code — always normalize
+*.cpp text
+*.h text
+*.cs text
+*.py text
+*.ini text
+*.json text
+*.md text
+*.xml text
+*.yaml text
+*.yml text
+
+# Unreal assets — binary, no normalization
+*.uasset binary
+*.umap binary
+*.uproject text
+
+# Binaries — no normalization
+*.dll binary
+*.exe binary
+*.pdb binary
+*.lib binary
+*.exp binary
+*.so binary
+*.dylib binary
+*.a binary

.gitignore

@@ -4,3 +4,24 @@ Unreal/PS_AI_Agent/Binaries/
 Unreal/PS_AI_Agent/Intermediate/
 Unreal/PS_AI_Agent/Saved/
 ConvAI/Convai/Binaries/
+
+# Plugin build artifacts (regenerated)
+Unreal/PS_AI_Agent/Plugins/*/Binaries/
+Unreal/PS_AI_Agent/Plugins/*/Intermediate/
+
+# Live Coding patch files (temporary, regenerated each session)
+*.patch_*
+
+# UE5 generated solution files
+Unreal/PS_AI_Agent/*.sln
+Unreal/PS_AI_Agent/*.suo
+
+# Claude Code local session settings (machine-specific, memory files in .claude/ are kept)
+.claude/settings.local.json
+.claude/worktrees/
+
+# Documentation generator script (dev tool, output .pptx is committed instead)
+generate_pptx.py
+~$PS_AI_Agent_ElevenLabs_Documentation.pptx
+Unreal/PS_AI_Agent/Build/
+Unreal/PS_AI_Agent/Builds/

Build_Plugin.bat

@@ -0,0 +1,65 @@
+@echo off
+setlocal
+
+:: ─── Configuration ──────────────────────────────────────────────────────────
+set UE_ROOT=C:\Program Files\Epic Games\UE_5.5
+set UPROJECT=E:\ASTERION\GIT\PS_AI_Agent\Unreal\PS_AI_Agent\PS_AI_Agent.uproject
+set PLATFORM=Win64
+set CONFIG=Development
+set TARGET=PS_AI_AgentEditor
+
+:: ─── Colors ─────────────────────────────────────────────────────────────────
+set GREEN=[92m
+set RED=[91m
+set YELLOW=[93m
+set RESET=[0m
+
+:: ─── Banner ─────────────────────────────────────────────────────────────────
+echo.
+echo %GREEN%============================================%RESET%
+echo   PS_AI_Agent - Plugin Build
+echo   Target: %TARGET% %PLATFORM% %CONFIG%
+echo %GREEN%============================================%RESET%
+echo.
+
+:: ─── Verify paths ───────────────────────────────────────────────────────────
+if not exist "%UE_ROOT%\Engine\Binaries\ThirdParty\DotNet\8.0.300\win-x64\dotnet.exe" (
+    echo %RED%ERROR: UE5.5 dotnet not found at %UE_ROOT%%RESET%
+    echo Check UE_ROOT path in this script.
+    pause
+    exit /b 1
+)
+
+if not exist "%UPROJECT%" (
+    echo %RED%ERROR: .uproject not found: %UPROJECT%%RESET%
+    pause
+    exit /b 1
+)
+
+:: ─── Build ──────────────────────────────────────────────────────────────────
+echo %YELLOW%Building %TARGET% (%CONFIG%)...%RESET%
+echo.
+
+"%UE_ROOT%\Engine\Binaries\ThirdParty\DotNet\8.0.300\win-x64\dotnet.exe" ^
+    "%UE_ROOT%\Engine\Binaries\DotNET\UnrealBuildTool\UnrealBuildTool.dll" ^
+    %TARGET% %PLATFORM% %CONFIG% ^
+    -Project="%UPROJECT%" ^
+    -WaitMutex ^
+    -FromMsBuild
+
+set BUILD_EXIT=%ERRORLEVEL%
+
+echo.
+if %BUILD_EXIT% EQU 0 (
+    echo %GREEN%============================================%RESET%
+    echo   BUILD SUCCEEDED
+    echo %GREEN%============================================%RESET%
+) else (
+    echo %RED%============================================%RESET%
+    echo   BUILD FAILED  (exit code: %BUILD_EXIT%)
+    echo %RED%============================================%RESET%
+)
+
+echo.
+pause
+exit /b %BUILD_EXIT%