Update memory: document v1.5.0 mic chunk size fix

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

.claude/session_log_2026-02-19.md

@@ -189,9 +189,160 @@ Commit: `99017f4`
 
 ---
 
+---
+
+## Session 3 — 2026-02-19 (bug fixes from live testing)
+
+### 16. Three Runtime Bugs Fixed (v1.2.0)
+
+User reported after live testing:
+1. **AI speaks twice** — every audio response played double
+2. **Cannot speak** — mic capture didn't reach ElevenLabs
+3. **Latency** — requested `enable_intermediate_response: true`
+
+**Bug 1 Root Cause — Double Audio:**
+UE's libwebsockets backend fires **both** `OnMessage()` (text callback) **and** `OnRawMessage()` (binary callback) for the same incoming frame.
+We had bound both `WebSocket->OnMessage()` and `WebSocket->OnRawMessage()` in `Connect()`.
+Result: every audio frame was decoded and enqueued twice → played twice.
+
+Fix: **Remove `OnMessage` binding entirely.** `OnRawMessage` now handles all frames (JSON control messages peeked via first byte, raw PCM otherwise).
+
+**Bug 2 Root Cause — Mic Silent:**
+ElevenLabs requires a `conversation_initiation_client_data` message sent **immediately** after the WebSocket handshake completes. Without it, the server never enters a state where it will accept and process client audio chunks. This is a required session negotiation step, not optional.
+
+Fix: Send `conversation_initiation_client_data` in `OnWsConnected()` before any other message.
+
+**Bug 2 Secondary — Delegate Stacking:**
+`StartListening()` called `Mic->OnAudioCaptured.AddUObject(this, ...)` without first removing existing bindings. If called more than once (e.g. after reconnect), delegates stack up and audio is sent multiple times per frame.
+
+Fix: Add `Mic->OnAudioCaptured.RemoveAll(this)` before `AddUObject` in `StartListening()`.
+
+**Bug 3 — Latency:**
+Added `"enable_intermediate_response": true` inside `custom_llm_extra_body` of the `conversation_initiation_client_data` message. Also added `optimize_streaming_latency: 3` in `conversation_config_override.tts`.
+
+**Files changed:**
+- `ElevenLabsWebSocketProxy.cpp`:
+  - `Connect()`: removed `OnMessage` binding
+  - `OnWsConnected()`: now sends full `conversation_initiation_client_data` JSON
+- `ElevenLabsConversationalAgentComponent.cpp`:
+  - `StartListening()`: added `RemoveAll` guard before delegate binding
+
+---
+
+---
+
+## Session 4 — 2026-02-19 (mic still silent — push-to-talk deeper investigation)
+
+### 17. Two More Bugs Found and Fixed (v1.3.0)
+
+User confirmed Bug 1 (double audio) was fixed. Bug 2 (cannot speak) persisted.
+
+**Analysis of log:**
+- Blueprint is correct: T Pressed → StartListening, T Released → StopListening (proper push-to-talk)
+- Mic opens and closes correctly — audio capture IS happening
+- Server never responds to mic input → audio reaching ElevenLabs but being ignored
+
+**Bug A — TurnMode mismatch in conversation_initiation_client_data:**
+`OnWsConnected()` hardcoded `"mode": "server_vad"` in the init message regardless of the
+component's `TurnMode` setting. User's Blueprint uses Client turn mode (push-to-talk),
+so the server was configured for server_vad while the client sent client_vad audio signals.
+
+Fix: Read `TurnMode` field on the proxy (set from the component before `Connect()`).
+Translate `EElevenLabsTurnMode::Client` → `"client_vad"`, Server → `"server_vad"`.
+
+**Bug B — user_activity never sent continuously:**
+In client VAD mode, ElevenLabs requires `user_activity` to be sent **continuously**
+alongside every audio chunk to keep the server's VAD aware the user is speaking.
+`SendUserTurnStart()` sent it once on key press, but never again during speech.
+Server-side, without continuous `user_activity`, the server treated the audio as noise.
+
+Fix: In `SendAudioChunk()`, automatically send `user_activity` before each audio chunk
+when `TurnMode == Client`. This keeps the signal continuous for the full duration of speech.
+When the user releases T, `StopListening()` stops the mic → audio stops → `user_activity`
+stops → server detects silence and triggers the agent response.
+
+**Bug C — TurnMode not propagated to proxy:**
+`UElevenLabsConversationalAgentComponent` never told the proxy what TurnMode to use.
+Added `WebSocketProxy->TurnMode = TurnMode` before `Connect()` in `StartConversation()`.
+
+**Files changed:**
+- `ElevenLabsWebSocketProxy.h`: added `public TurnMode` field
+- `ElevenLabsWebSocketProxy.cpp`:
+  - `OnWsConnected()`: use `TurnMode` to set correct mode string in init message
+  - `SendAudioChunk()`: auto-send `user_activity` before each chunk in Client mode
+- `ElevenLabsConversationalAgentComponent.cpp`:
+  - `StartConversation()`: set `WebSocketProxy->TurnMode = TurnMode` before `Connect()`
+
+---
+
+---
+
+## Session 5 — 2026-02-19 (still can't speak — bAutoStartListening conflict)
+
+### 18. Root Cause Found and Fixed (v1.4.0)
+
+Log analysis revealed the true root cause:
+
+**Exact sequence:**
+```
+OnConnected → bAutoStartListening=true → StartListening() → bIsListening=true, mic opens
+OnAgentStoppedSpeaking → Blueprint calls StartListening() → bIsListening guard → no-op (already open)
+User presses T → StartListening() → bIsListening guard → no-op
+User releases T → StopListening() → bIsListening=false, mic CLOSES
+User presses T → StartListening() → NOW opens mic (was closed)
+User releases T → StopListening() → mic closes — but ElevenLabs never got audio
+```
+
+**Root cause:** `bAutoStartListening = true` opens the mic on connect and sets `bIsListening = true`.
+In Client/push-to-talk mode, every T-press hits the `bIsListening` guard and does nothing.
+Every T-release closes the auto-started mic. The mic was never open during actual speech.
+
+**Fix:** `HandleConnected()` now only calls `StartListening()` when `TurnMode == Server`.
+In Client mode, `bAutoStartListening` is ignored — the user controls listening via T key.
+
+**File changed:**
+- `ElevenLabsConversationalAgentComponent.cpp`:
+  - `HandleConnected()`: guard `bAutoStartListening` with `TurnMode == Server` check
+
+---
+
+---
+
+## Session 6 — 2026-02-19 (audio chunk size fix)
+
+### 19. Mic Audio Chunk Accumulation (v1.5.0)
+
+**Root cause (from diagnostic log in Session 5):**
+Log showed hundreds of `SendAudioChunk: 158 bytes (TurnMode=Client)` lines with zero server responses.
+- 158 bytes = 79 samples = ~5ms of audio at 16kHz 16-bit mono
+- WASAPI (Windows Audio Session API) fires the `FAudioCapture` callback at its internal buffer period (~5ms)
+- ElevenLabs requires a minimum chunk size for its VAD and STT to operate (~100ms / 3200 bytes)
+- Tiny 5ms fragments arrived at the server but were silently ignored → agent never responded
+
+**Fix applied:**
+Added `MicAccumulationBuffer TArray<uint8>` to `UElevenLabsConversationalAgentComponent`.
+`OnMicrophoneDataCaptured()` appends each callback's converted bytes and only calls `SendAudioChunk`
+when `>= MicChunkMinBytes` (3200 bytes = 100ms) have accumulated.
+
+`StopListening()` flushes any remaining bytes in the buffer before sending `SendUserTurnEnd()`,
+so the last partial chunk of speech is never dropped.
+
+`HandleDisconnected()` clears the buffer to prevent stale data on reconnect.
+
+**Files changed:**
+- `ElevenLabsConversationalAgentComponent.h`: added `MicAccumulationBuffer` + `MicChunkMinBytes = 3200`
+- `ElevenLabsConversationalAgentComponent.cpp`:
+  - `OnMicrophoneDataCaptured()`: accumulate → send when threshold reached
+  - `StopListening()`: flush remainder before end-of-turn signal
+  - `HandleDisconnected()`: clear accumulation buffer
+
+Commit: `91cf5b1`
+
+---
+
 ## Next Steps (not done yet)
 
-- [ ] Verify mic audio actually reaches ElevenLabs (enable Verbose Logging, test in Editor)
+- [ ] Test v1.5.0 in Editor — verify push-to-talk mic works end-to-end (should be the final fix)
 - [ ] Test `USoundWaveProcedural` underflow behaviour in practice (check for audio glitches)
 - [ ] Test `SendTextMessage` end-to-end in Blueprint
 - [ ] Add lip-sync support (future)