ConvAgent: reassemble agent audio sub-chunks into one enqueue (fix pops)

After sub-chunking each ElevenLabs message into N ~0.8s Opus packets, the
client enqueued each sub-chunk separately, re-running the playback state
machine per sub-chunk (spurious Play() restart / pre-buffer race / underrun
at the seam) -> an audible pop at each ~0.8s boundary (2-3 per sentence).
Confirmed via multi-agent analysis + logs (no generation skips, no decode
truncation -> not a codec issue; the discontinuity is on the enqueue side).

- MulticastReceiveAgentAudio gains a bLastSubChunk flag; the server marks the
  final sub-chunk of each message (encode path frame-aligned so the last
  sub-chunk is well-defined; raw-PCM fallback likewise).
- The client accumulates decoded PCM across a message's sub-chunks and calls
  EnqueueAgentAudio (and lip-sync broadcast) ONCE, on the last sub-chunk.
- ClientReassemblyPCM reset in StopAgentAudio so a half-received message can't
  leak across a turn/interruption.

No added latency: a message's sub-chunks are sent back-to-back in one server
tick and arrive together. Each RPC still <=40 frames (Reliable, under the
encode/decode limits).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Private/PS_AI_ConvAgent_ElevenLabsComponent.cpp

@@ -1231,61 +1231,102 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::HandleAudioReceived(const TArray<uint
 
 		if (OpusEncoder.IsValid())
 		{
-			// Opus path: compress then send.
-			uint32 CompressedSize = static_cast<uint32>(OpusWorkBuffer.Num());
-			OpusEncoder->Encode(PCMData.GetData(), PCMData.Num(),
-				OpusWorkBuffer.GetData(), CompressedSize);
+			// Opus path: compress then send, in sub-blocks.
+			//
+			// A whole ElevenLabs message can be several seconds. Two hard limits
+			// force us to split it before each Encode()/RPC:
+			//   1. UE's Opus encoder caps ONE packet at 76 frames (~1.52s) — frames
+			//      beyond that are silently dropped at encode (VoiceCodecOpus.cpp).
+			//   2. The client decode buffer only holds ~44 frames (~0.88s) before
+			//      the engine decoder aborts with "Decompression buffer too small"
+			//      and drops the packet tail (MulticastReceiveAgentAudio_Implementation).
+			// 40 frames (25600 bytes, 0.8s) stays safely under BOTH, so nothing is
+			// truncated on server or client. It also keeps each Unreliable RPC small.
+			static constexpr int32 BytesPerFrame =
+				(PS_AI_ConvAgent_Audio_ElevenLabs::SampleRate / 50)
+				* PS_AI_ConvAgent_Audio_ElevenLabs::Channels
+				* static_cast<int32>(sizeof(int16));            // 640 at 16kHz mono
+			static constexpr int32 MaxOpusChunkBytes = 40 * BytesPerFrame; // 25600 = 0.8s
 
-			if (CompressedSize > 0)
+			// Frame-align the message length (Encode drops any sub-frame remainder
+			// anyway). A frame-aligned length guarantees every sub-chunk holds >=1
+			// whole frame and lets us reliably mark the FINAL sub-chunk (bLast), so
+			// the client reassembles one message into a single enqueue (no per-chunk
+			// playback re-arm → no pops).
+			const int32 UsableBytes = (PCMData.Num() / BytesPerFrame) * BytesPerFrame;
+
+			int32 Offset = 0;
+			while (Offset < UsableBytes)
 			{
-				TArray<uint8> CompressedData;
-				CompressedData.Append(OpusWorkBuffer.GetData(), CompressedSize);
+				const int32 BlockBytes = FMath::Min(MaxOpusChunkBytes, UsableBytes - Offset);
+				const bool bLast = (Offset + BlockBytes >= UsableBytes);
 
-				if (bNetAudioDebug)
+				uint32 CompressedSize = static_cast<uint32>(OpusWorkBuffer.Num());
+				OpusEncoder->Encode(PCMData.GetData() + Offset, BlockBytes,
+					OpusWorkBuffer.GetData(), CompressedSize);
+
+				if (CompressedSize > 0)
 				{
-					const float Ratio = (CompressedSize > 0)
-						? static_cast<float>(PCMData.Num()) / static_cast<float>(CompressedSize)
-						: 0.0f;
-					UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
-						TEXT("[NET-SRV] Opus audio chunk: %u bytes sent (raw %d → %.1fx compression)."),
-						CompressedSize, PCMData.Num(), Ratio);
+					TArray<uint8> CompressedData;
+					CompressedData.Append(OpusWorkBuffer.GetData(), CompressedSize);
+
+					if (bNetAudioDebug)
+					{
+						const float Ratio = static_cast<float>(BlockBytes) / static_cast<float>(CompressedSize);
+						UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Log,
+							TEXT("[NET-SRV] Opus audio chunk: %u bytes sent (raw %d → %.1fx compression, last=%d)."),
+							CompressedSize, BlockBytes, Ratio, bLast ? 1 : 0);
+					}
+
+					MulticastReceiveAgentAudio(CompressedData, bLast);
 				}
 
-				MulticastReceiveAgentAudio(CompressedData);
+				Offset += BlockBytes;
 			}
 		}
 		else
 		{
-			// Fallback: send raw PCM (no compression). ~32 KB/s at 16kHz 16-bit mono.
-			// Only hit if Opus init failed. Chunk small so the Unreliable RPC is
-			// NOT auto-promoted to reliable when fragmented: UE re-sends an
-			// unreliable bunch reliably once it splits into more than ~8 partial
-			// bunches, which would reintroduce the reliable-buffer overflow.
-			// ~4 KB ≈ 4 bunches → stays unreliable.
-			static bool bWarnedOnce = false;
-			if (!bWarnedOnce)
+			// Fallback: Opus init failed → send raw PCM (no compression, ~32 KB/s).
+			// This should not happen in the working config — it means [Voice]
+			// bEnabled=true is missing in DefaultEngine.ini. Warn on every message
+			// so it is loud, not hidden behind a one-shot flag.
+			// The RPC is now Reliable, so we must protect the reliable buffer (512
+			// bunches): each ~4 KB chunk ≈ ~4 partial bunches. Hard-cap one message
+			// at MaxFallbackBytes and drop the tail so a single message can never
+			// approach the buffer and trigger a reliable-buffer overflow (the exact
+			// cause of the old VR disconnect).
+			UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
+				TEXT("[NET-SRV] Opus encoder unavailable — sending raw PCM (no compression). "
+					 "Set [Voice] bEnabled=true in DefaultEngine.ini to restore Opus."));
+
+			static constexpr int32 MaxChunkBytes = 4000;
+			static constexpr int32 MaxFallbackBytes = 320000; // ~80 reliable chunks/message, well under the 512-bunch buffer
+
+			const int32 SendableBytes = FMath::Min(PCMData.Num(), MaxFallbackBytes);
+			if (PCMData.Num() > MaxFallbackBytes)
 			{
-				bWarnedOnce = true;
 				UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
-					TEXT("[NET-SRV] Opus encoder unavailable — sending raw PCM in chunks (no compression)."));
+					TEXT("[NET-SRV] Raw-PCM fallback capped at %d bytes — dropping %d-byte tail "
+						 "to protect the reliable buffer."),
+					MaxFallbackBytes, PCMData.Num() - MaxFallbackBytes);
 			}
 
-			static constexpr int32 MaxChunkBytes = 4000; // small enough to stay unreliable (avoid partial-bunch reliable promotion)
 			int32 Offset = 0;
-			while (Offset < PCMData.Num())
+			while (Offset < SendableBytes)
 			{
-				const int32 ChunkSize = FMath::Min(MaxChunkBytes, PCMData.Num() - Offset);
+				const int32 ChunkSize = FMath::Min(MaxChunkBytes, SendableBytes - Offset);
+				const bool bLast = (Offset + ChunkSize >= SendableBytes);
 				TArray<uint8> Chunk;
 				Chunk.Append(PCMData.GetData() + Offset, ChunkSize);
 
 				if (bNetAudioDebug)
 				{
 					UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
-						TEXT("[NET-SRV] RAW PCM chunk: %d bytes sent (NO compression — Opus disabled)."),
-						ChunkSize);
+						TEXT("[NET-SRV] RAW PCM chunk: %d bytes sent (NO compression — Opus disabled, last=%d)."),
+						ChunkSize, bLast ? 1 : 0);
 				}
 
-				MulticastReceiveAgentAudio(Chunk);
+				MulticastReceiveAgentAudio(Chunk, bLast);
 				Offset += ChunkSize;
 			}
 		}
@@ -1851,6 +1892,7 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::StopAgentAudio()
 	bool bWasSpeaking = false;
 	double Now = 0.0;
 	bPreBuffering = false;      // Clear pre-buffer state on stop.
+	ClientReassemblyPCM.Reset(); // Drop any half-received message so it can't leak into the next turn.
 	{
 		FScopeLock Lock(&AudioQueueLock);
 		AudioQueue.Empty();
@@ -2515,7 +2557,7 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::ClientConversationFailed_Implementati
 // Network: Multicast RPCs
 // ─────────────────────────────────────────────────────────────────────────────
 void UPS_AI_ConvAgent_ElevenLabsComponent::MulticastReceiveAgentAudio_Implementation(
-	const TArray<uint8>& AudioData)
+	const TArray<uint8>& AudioData, bool bLastSubChunk)
 {
 	// Server already handled playback in HandleAudioReceived.
 	if (GetOwnerRole() == ROLE_Authority) return;
@@ -2533,13 +2575,17 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::MulticastReceiveAgentAudio_Implementa
 	if (OpusDecoder.IsValid())
 	{
 		// Opus path: decode compressed audio.
-		const uint32 MaxDecompressedSize = 16000 * 2;
+		// Buffer must hold a full packet's decoded PCM. The engine decoder aborts
+		// with "Decompression buffer too small to decode voice" (dropping the
+		// packet tail) if this is smaller than the packet needs. The server caps
+		// packets at 40 frames (25600 bytes); 64KB gives ample headroom even up to
+		// the engine's 76-frame (~48640 bytes) maximum, so no packet is truncated.
+		const uint32 MaxDecompressedSize = 64 * 1024;
 		PCMBuffer.SetNumUninitialized(MaxDecompressedSize);
 		uint32 DecompressedSize = MaxDecompressedSize;
 		OpusDecoder->Decode(AudioData.GetData(), AudioData.Num(),
 			PCMBuffer.GetData(), DecompressedSize);
-		if (DecompressedSize == 0) return;
-		PCMBuffer.SetNum(DecompressedSize);
+		PCMBuffer.SetNum(DecompressedSize);   // may be 0 — handled by reassembly below
 	}
 	else
 	{
@@ -2547,13 +2593,33 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::MulticastReceiveAgentAudio_Implementa
 		PCMBuffer = AudioData;
 	}
 
-	// Local playback.
-	EnqueueAgentAudio(PCMBuffer);
-
-	// Feed lip-sync (within LOD or speaker).
-	if (bIsSpeaker || LipSyncLODDistance <= 0.f || Dist <= LipSyncLODDistance)
+	// Reassemble one ElevenLabs message from its N sub-chunks, then enqueue ONCE.
+	// The server splits a message into sub-chunks (to respect the Opus 76-frame /
+	// client decode-buffer limits), sent back-to-back in the same tick. Enqueuing
+	// each sub-chunk separately re-ran the playback state machine → a pop at each
+	// ~0.8s seam. Accumulating and enqueuing once per message plays it contiguously.
+	if (PCMBuffer.Num() > 0)
 	{
-		OnAgentAudioData.Broadcast(PCMBuffer);
+		ClientReassemblyPCM.Append(PCMBuffer);
+	}
+
+	if (!bLastSubChunk)
+	{
+		return;   // wait for the remaining sub-chunks of this message
+	}
+
+	if (ClientReassemblyPCM.Num() > 0)
+	{
+		// Local playback (one contiguous block per message).
+		EnqueueAgentAudio(ClientReassemblyPCM);
+
+		// Feed lip-sync (within LOD or speaker).
+		if (bIsSpeaker || LipSyncLODDistance <= 0.f || Dist <= LipSyncLODDistance)
+		{
+			OnAgentAudioData.Broadcast(ClientReassemblyPCM);
+		}
+
+		ClientReassemblyPCM.Reset();
 	}
 }
 

Unreal/PS_AI_Agent/Plugins/PS_AI_ConvAgent/Source/PS_AI_ConvAgent/Public/PS_AI_ConvAgent_ElevenLabsComponent.h

@@ -485,12 +485,21 @@ public:
 	void ServerRequestInterrupt();
 
 	/** Broadcast agent audio to all clients (Opus-compressed or raw PCM fallback).
-	 *  Unreliable: realtime audio — a lost/oversized packet is dropped (minor
-	 *  glitch) rather than backing up the reliable buffer and dropping the
-	 *  connection. Keep each call small (Opus ~1-4KB) so it isn't auto-promoted
-	 *  to reliable when fragmented into partial bunches. */
-	UFUNCTION(NetMulticast, Unreliable)
-	void MulticastReceiveAgentAudio(const TArray<uint8>& AudioData);
+	 *  Reliable: agent TTS must arrive complete and IN ORDER. Opus is a stateful
+	 *  (inter-frame-predictive) codec, so a dropped or reordered packet on an
+	 *  Unreliable channel desyncs the decoder → missing words + crackle. Reliable
+	 *  guarantees ordered, lossless delivery, keeping the decoder in sync.
+	 *  This does NOT reintroduce the old reliable-buffer overflow (which dropped
+	 *  VR connections): that came from 32-152 KB raw-PCM RPCs. With Opus each
+	 *  sub-chunk is ~2 KB (2-3 partial bunches, below the 8-bunch reliable-
+	 *  promotion threshold) at ~1.25 RPC/s, so in-flight bunches stay in single
+	 *  digits vs the 512-bunch buffer. Sub-chunking in HandleAudioReceived is what
+	 *  keeps packets this small — it must stay.
+	 *  bLastSubChunk marks the final sub-chunk of one ElevenLabs message so the
+	 *  client can reassemble all sub-chunks and enqueue ONCE per message (enqueuing
+	 *  each sub-chunk separately re-ran the playback state machine → pops). */
+	UFUNCTION(NetMulticast, Reliable)
+	void MulticastReceiveAgentAudio(const TArray<uint8>& AudioData, bool bLastSubChunk);
 
 	/** Notify all clients that the agent started speaking (first audio chunk). */
 	UFUNCTION(NetMulticast, Reliable)
@@ -798,6 +807,11 @@ private:
 	int32 AudioQueueReadOffset = 0;
 	FCriticalSection AudioQueueLock;
 
+	// Client-only: accumulates decoded PCM across the N sub-chunks of one
+	// ElevenLabs message; flushed to EnqueueAgentAudio() once, on the last
+	// sub-chunk (bLastSubChunk). Game-thread only. Reset in StopAgentAudio().
+	TArray<uint8> ClientReassemblyPCM;
+
 	// Pre-buffer state: delay playback start to absorb TTS inter-chunk gaps.
 	bool bPreBuffering = false;
 	double PreBufferStartTime = 0.0;