ConvAgent: agent audio multicast back to Reliable (fix Opus crackle/gaps)
Sending Opus over Unreliable multicast caused missing words and crackling: Opus is a stateful inter-frame codec, so a dropped or reordered packet desyncs the decoder and corrupts all following frames until the stream resets. Unreliable was originally adopted only to stop a reliable-buffer overflow — but that overflow came from 32-152KB raw-PCM RPCs (pre-Opus). With Opus + sub-chunking each RPC is ~2KB (2-3 partial bunches) at ~1.25 RPC/s, so the 512-bunch reliable buffer stays in single digits and cannot overflow. Reliable restores ordered, lossless delivery. - MulticastReceiveAgentAudio: Unreliable -> Reliable. - Keep the 40-frame (0.8s, ~2KB) sub-chunking — it is what keeps packets small enough for Reliable to be safe. - Harden the raw-PCM fallback (only reached if Opus init fails): hard-cap one message at 320KB and drop the tail so it can never overflow the reliable buffer, and warn on every fallback (config bug: [Voice]). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -1278,24 +1278,33 @@ void UPS_AI_ConvAgent_ElevenLabsComponent::HandleAudioReceived(const TArray<uint
|
|||||||
}
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
// Fallback: send raw PCM (no compression). ~32 KB/s at 16kHz 16-bit mono.
|
// Fallback: Opus init failed → send raw PCM (no compression, ~32 KB/s).
|
||||||
// Only hit if Opus init failed. Chunk small so the Unreliable RPC is
|
// This should not happen in the working config — it means [Voice]
|
||||||
// NOT auto-promoted to reliable when fragmented: UE re-sends an
|
// bEnabled=true is missing in DefaultEngine.ini. Warn on every message
|
||||||
// unreliable bunch reliably once it splits into more than ~8 partial
|
// so it is loud, not hidden behind a one-shot flag.
|
||||||
// bunches, which would reintroduce the reliable-buffer overflow.
|
// The RPC is now Reliable, so we must protect the reliable buffer (512
|
||||||
// ~4 KB ≈ 4 bunches → stays unreliable.
|
// bunches): each ~4 KB chunk ≈ ~4 partial bunches. Hard-cap one message
|
||||||
static bool bWarnedOnce = false;
|
// at MaxFallbackBytes and drop the tail so a single message can never
|
||||||
if (!bWarnedOnce)
|
// approach the buffer and trigger a reliable-buffer overflow (the exact
|
||||||
{
|
// cause of the old VR disconnect).
|
||||||
bWarnedOnce = true;
|
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
|
||||||
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
|
TEXT("[NET-SRV] Opus encoder unavailable — sending raw PCM (no compression). "
|
||||||
TEXT("[NET-SRV] Opus encoder unavailable — sending raw PCM in chunks (no compression)."));
|
"Set [Voice] bEnabled=true in DefaultEngine.ini to restore Opus."));
|
||||||
}
|
|
||||||
|
|
||||||
static constexpr int32 MaxChunkBytes = 4000; // small enough to stay unreliable (avoid partial-bunch reliable promotion)
|
static constexpr int32 MaxChunkBytes = 4000;
|
||||||
|
static constexpr int32 MaxFallbackBytes = 320000; // ~80 reliable chunks/message, well under the 512-bunch buffer
|
||||||
int32 Offset = 0;
|
int32 Offset = 0;
|
||||||
while (Offset < PCMData.Num())
|
while (Offset < PCMData.Num())
|
||||||
{
|
{
|
||||||
|
if (Offset >= MaxFallbackBytes)
|
||||||
|
{
|
||||||
|
UE_LOG(LogPS_AI_ConvAgent_ElevenLabs, Warning,
|
||||||
|
TEXT("[NET-SRV] Raw-PCM fallback capped at %d bytes — dropping %d-byte tail "
|
||||||
|
"to protect the reliable buffer."),
|
||||||
|
MaxFallbackBytes, PCMData.Num() - Offset);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
|
||||||
const int32 ChunkSize = FMath::Min(MaxChunkBytes, PCMData.Num() - Offset);
|
const int32 ChunkSize = FMath::Min(MaxChunkBytes, PCMData.Num() - Offset);
|
||||||
TArray<uint8> Chunk;
|
TArray<uint8> Chunk;
|
||||||
Chunk.Append(PCMData.GetData() + Offset, ChunkSize);
|
Chunk.Append(PCMData.GetData() + Offset, ChunkSize);
|
||||||
|
|||||||
@@ -485,11 +485,17 @@ public:
|
|||||||
void ServerRequestInterrupt();
|
void ServerRequestInterrupt();
|
||||||
|
|
||||||
/** Broadcast agent audio to all clients (Opus-compressed or raw PCM fallback).
|
/** Broadcast agent audio to all clients (Opus-compressed or raw PCM fallback).
|
||||||
* Unreliable: realtime audio — a lost/oversized packet is dropped (minor
|
* Reliable: agent TTS must arrive complete and IN ORDER. Opus is a stateful
|
||||||
* glitch) rather than backing up the reliable buffer and dropping the
|
* (inter-frame-predictive) codec, so a dropped or reordered packet on an
|
||||||
* connection. Keep each call small (Opus ~1-4KB) so it isn't auto-promoted
|
* Unreliable channel desyncs the decoder → missing words + crackle. Reliable
|
||||||
* to reliable when fragmented into partial bunches. */
|
* guarantees ordered, lossless delivery, keeping the decoder in sync.
|
||||||
UFUNCTION(NetMulticast, Unreliable)
|
* This does NOT reintroduce the old reliable-buffer overflow (which dropped
|
||||||
|
* VR connections): that came from 32-152 KB raw-PCM RPCs. With Opus each
|
||||||
|
* sub-chunk is ~2 KB (2-3 partial bunches, below the 8-bunch reliable-
|
||||||
|
* promotion threshold) at ~1.25 RPC/s, so in-flight bunches stay in single
|
||||||
|
* digits vs the 512-bunch buffer. Sub-chunking in HandleAudioReceived is what
|
||||||
|
* keeps packets this small — it must stay. */
|
||||||
|
UFUNCTION(NetMulticast, Reliable)
|
||||||
void MulticastReceiveAgentAudio(const TArray<uint8>& AudioData);
|
void MulticastReceiveAgentAudio(const TArray<uint8>& AudioData);
|
||||||
|
|
||||||
/** Notify all clients that the agent started speaking (first audio chunk). */
|
/** Notify all clients that the agent started speaking (first audio chunk). */
|
||||||
|
|||||||
Reference in New Issue
Block a user