ConvAgent: agent audio multicast back to Reliable (fix Opus crackle/gaps)

Sending Opus over Unreliable multicast caused missing words and crackling:
Opus is a stateful inter-frame codec, so a dropped or reordered packet
desyncs the decoder and corrupts all following frames until the stream
resets. Unreliable was originally adopted only to stop a reliable-buffer
overflow — but that overflow came from 32-152KB raw-PCM RPCs (pre-Opus).
With Opus + sub-chunking each RPC is ~2KB (2-3 partial bunches) at
~1.25 RPC/s, so the 512-bunch reliable buffer stays in single digits and
cannot overflow. Reliable restores ordered, lossless delivery.

- MulticastReceiveAgentAudio: Unreliable -> Reliable.
- Keep the 40-frame (0.8s, ~2KB) sub-chunking — it is what keeps packets
  small enough for Reliable to be safe.
- Harden the raw-PCM fallback (only reached if Opus init fails): hard-cap
  one message at 320KB and drop the tail so it can never overflow the
  reliable buffer, and warn on every fallback (config bug: [Voice]).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-01 14:31:02 +02:00
parent 2ca5643f32
commit 5b3a11a733
2 changed files with 34 additions and 19 deletions

View File

@@ -1278,24 +1278,33 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::HandleAudioReceived(const TArray<uint
} }
else else
{ {
// Fallback: send raw PCM (no compression). ~32 KB/s at 16kHz 16-bit mono. // Fallback: Opus init failed → send raw PCM (no compression, ~32 KB/s).
// Only hit if Opus init failed. Chunk small so the Unreliable RPC is // This should not happen in the working config — it means [Voice]
// NOT auto-promoted to reliable when fragmented: UE re-sends an // bEnabled=true is missing in DefaultEngine.ini. Warn on every message
// unreliable bunch reliably once it splits into more than ~8 partial // so it is loud, not hidden behind a one-shot flag.
// bunches, which would reintroduce the reliable-buffer overflow. // The RPC is now Reliable, so we must protect the reliable buffer (512
// ~4 KB ≈ 4 bunches → stays unreliable. // bunches): each ~4 KB chunk~4 partial bunches. Hard-cap one message
static bool bWarnedOnce = false; // at MaxFallbackBytes and drop the tail so a single message can never
if (!bWarnedOnce) // approach the buffer and trigger a reliable-buffer overflow (the exact
{ // cause of the old VR disconnect).
bWarnedOnce = true; UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning, TEXT("[NET-SRV] Opus encoder unavailable — sending raw PCM (no compression). "
TEXT("[NET-SRV] Opus encoder unavailable — sending raw PCM in chunks (no compression).")); "Set [Voice] bEnabled=true in DefaultEngine.ini to restore Opus."));
}
static constexpr int32 MaxChunkBytes = 4000; // small enough to stay unreliable (avoid partial-bunch reliable promotion) static constexpr int32 MaxChunkBytes = 4000;
static constexpr int32 MaxFallbackBytes = 320000; // ~80 reliable chunks/message, well under the 512-bunch buffer
int32 Offset = 0; int32 Offset = 0;
while (Offset < PCMData.Num()) while (Offset < PCMData.Num())
{ {
if (Offset >= MaxFallbackBytes)
{
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
TEXT("[NET-SRV] Raw-PCM fallback capped at %d bytes — dropping %d-byte tail "
"to protect the reliable buffer."),
MaxFallbackBytes, PCMData.Num() - Offset);
break;
}
const int32 ChunkSize = FMath::Min(MaxChunkBytes, PCMData.Num() - Offset); const int32 ChunkSize = FMath::Min(MaxChunkBytes, PCMData.Num() - Offset);
TArray<uint8> Chunk; TArray<uint8> Chunk;
Chunk.Append(PCMData.GetData() + Offset, ChunkSize); Chunk.Append(PCMData.GetData() + Offset, ChunkSize);

View File

@@ -485,11 +485,17 @@ public:
void ServerRequestInterrupt(); void ServerRequestInterrupt();
/** Broadcast agent audio to all clients (Opus-compressed or raw PCM fallback). /** Broadcast agent audio to all clients (Opus-compressed or raw PCM fallback).
* Unreliable: realtime audio — a lost/oversized packet is dropped (minor * Reliable: agent TTS must arrive complete and IN ORDER. Opus is a stateful
* glitch) rather than backing up the reliable buffer and dropping the * (inter-frame-predictive) codec, so a dropped or reordered packet on an
* connection. Keep each call small (Opus ~1-4KB) so it isn't auto-promoted * Unreliable channel desyncs the decoder → missing words + crackle. Reliable
* to reliable when fragmented into partial bunches. */ * guarantees ordered, lossless delivery, keeping the decoder in sync.
UFUNCTION(NetMulticast, Unreliable) * This does NOT reintroduce the old reliable-buffer overflow (which dropped
* VR connections): that came from 32-152 KB raw-PCM RPCs. With Opus each
* sub-chunk is ~2 KB (2-3 partial bunches, below the 8-bunch reliable-
* promotion threshold) at ~1.25 RPC/s, so in-flight bunches stay in single
* digits vs the 512-bunch buffer. Sub-chunking in HandleAudioReceived is what
* keeps packets this small — it must stay. */
UFUNCTION(NetMulticast, Reliable)
void MulticastReceiveAgentAudio(const TArray<uint8>& AudioData); void MulticastReceiveAgentAudio(const TArray<uint8>& AudioData);
/** Notify all clients that the agent started speaking (first audio chunk). */ /** Notify all clients that the agent started speaking (first audio chunk). */