Building Voice Agents With Real-Time Transcription and Low Latency
Text chatbot latency budgets don't transfer to voice. A two-second pause feels perfectly normal in a chat window and feels completely broken on a phone call — callers start talking again, assume the line dropped, or just hang up. We treat 800 milliseconds as the hard ceiling for perceived response latency in every voice agent project now, which forced a very different architecture than our text-based agents use.
The naive pipeline — wait for the caller to finish speaking, transcribe the full utterance, send it to the LLM, generate a full response, then synthesize speech — blows through that budget easily, often landing at three or four seconds end to end. We moved to streaming transcription that starts processing speech as it arrives rather than waiting for a full pause, shaving critical time off the front end of the pipeline.
Response generation needed to start before transcription fully finished, which meant accepting some risk of acting on a slightly incomplete transcript. We use a confidence-based cutoff on the speech-to-text stream — once transcript confidence and pause length cross a threshold, generation starts speculatively, and if the caller keeps talking, we cancel and restart rather than committing to an answer built on a cut-off sentence.
Text-to-speech synthesis latency mattered more than we expected going in. Waiting for the full LLM response before starting synthesis added a visible delay, so we moved to streaming synthesis that begins speaking the first sentence as soon as it's generated rather than waiting for the complete response, which cut perceived response time by close to half on longer answers.
Interruption handling — letting a caller talk over the agent and having it stop immediately rather than finishing its sentence — turned out to be one of the most important features for making the agent feel natural rather than robotic, and skipping it in an early version was the single most common complaint in user testing, ahead of any accuracy or transcription quality issue we measured.
