Engineering voice agents: Latency, quality, and scale — Rishabh Bhargava, Together AI

Summary

Rishabh Bhargava of Together AI explains the engineering trade-offs involved in building voice agents that are responsive, capable, natural-sounding, and reliable at production scale. He presents the cascading pipeline—speech-to-text (STT), an LLM, and text-to-speech (TTS)—as the prevailing production architecture, then discusses emerging speech-to-speech systems.

Why Voice Agents Are Challenging

Voice interactions are expected to happen in real time. People may notice responses taking more than about 500 ms, and delays of a second or more can make a conversation feel unusable. Agents also need to handle complex instructions and tool calls, speak naturally, and remain reliable across many concurrent calls. These requirements must be met together, not independently.

Pipeline Architecture and Component Requirements

Speech-to-Text

STT converts incoming audio into text for the LLM. Accuracy matters because errors in names, medicines, or other key terms can propagate through the rest of the system. Bhargava cites roughly 6% word error rate on open benchmarks and a P90 transcript-completion target around 100 ms for some systems.

Turn detection remains difficult: the agent must distinguish a pause from the end of a user’s turn to avoid speaking over them.

Streaming-native STT is an important direction. Older approaches, such as Whisper, were trained on long audio clips and often need chunking and stitching to support real-time use. Newer streaming architectures use limited look-ahead and cached activations to reduce repeated computation.

LLM

The LLM handles conversation logic, generates responses, and decides whether to call tools. Time to first token (TTFT) is a key latency measure; Bhargava gives 200–300 ms as a useful target range.

This puts practical pressure on model size: roughly 8–30 billion parameters is presented as a workable range. Larger models may exceed the latency budget, while smaller ones can struggle with intelligence and tool use.

Text-to-Speech

TTS turns generated text into audio. Important measures include time to first audio and real-time factor (RTF)—the amount of audio generated per unit of processing time. An RTF below 1 helps avoid buffering.

Voice quality is difficult to judge with a single metric, so listening to samples is important. Other considerations include pronunciation, language coverage, voice variety, and control over emotion.

System-Level Engineering

Latency and cost must be budgeted across the full system. In the described setup, the LLM typically accounts for the largest share, followed by TTS and then STT.

Network latency can be significant even when model inference itself is optimized. Bhargava illustrates that moving model calls from about 75 ms of network latency to around 5 ms through co-location can reduce latency by roughly 30% in an already optimized setup.

Scaling adds further complexity. Systems may need to scale up aggressively to prevent queues, while scaling down must account for long-lived, stateful conversations that cannot simply be terminated. Global deployment can reduce the distance to users and support data-residency requirements. Detailed observability is important because small delays at each stage accumulate.

Emerging Approach: Speech-to-Speech

A single speech-to-speech model could simplify orchestration and preserve vocal cues—such as tone, emotion, and hesitation—that may be lost when audio is converted to text. It could also support full-duplex interaction, allowing the model to speak while listening, provide conversational backchannels, and handle interruptions more naturally.

However, Bhargava says these models have often been less reliable at instruction following and tool calling, which has led teams to fall back to pipeline systems. He expects the approach to improve, but describes substantial engineering work as still ahead.

Production Patterns and Evaluation

• Tool-call evaluation: Assess whether the right tool was selected, whether the call structure is valid, and whether the result is correct. The required correctness level depends on the use case. Fine-tuning smaller models on domain-specific data can improve tool calling without sacrificing latency.
• Guardrails and classifiers: These can be placed before the main LLM to route requests, or after generation to check a response before it reaches TTS. They add latency, so teams need clear latency budgets and independent scaling. Catching an unsafe or unauthorized response before it is spoken is especially important.
• Thinker–talker pattern: A small, fast LLM manages the conversation and can give a brief holding response, then make one tool call to a larger, more capable model for complex requests. This keeps the common conversational path fast while delegating harder work.
• Speech-to-speech observability: Some systems may run transcription alongside speech generation to retain an auditable record. Evaluation may need to shift from separate text and audio component tests toward longer, full-duplex conversation-level tests.

Reviews, Guides, or Tutorials

This is a technical conference presentation and Q&A, not a product review or step-by-step tutorial. It provides practical design guidance and performance targets for voice-agent systems.

Main Speaker and Sources

• Rishabh Bhargava — Together AI; presenter and primary source of the technical discussion.
• Audience members — Asked questions about tool-call evaluation, model co-location, guardrails, and observability.