My AI Could Talk. It Couldn't Listen. That Was the Whole Problem.

When I first built Retone ai voice agent and called it, I was genuinely impressed. The responses were smart. The voice sounded natural. It understood what I was asking and gave good answers. For the first few calls I thought I'd nailed it.

But after a while something started bothering me. The conversations felt off. Not wrong exactly. Just hollow. Like talking to someone who was technically saying all the right things but wasn't really there. I couldn't figure out what was missing.

One afternoon I was working on it in a cafe, stuck on this feeling that the agent was 90% of the way to human but that last 10% was a wall. So I closed my laptop, closed my eyes, and just listened. To the cafe. To the people around me having conversations.

And it hit me. Every conversation around me had this invisible layer I'd never really noticed. People don't just take turns talking. While one person speaks the other is constantly producing tiny signals. "Mm-hmm." "Oh no." "Yeah." "Right." Not interrupting. Not adding anything substantial. Just a steady rhythm of sounds that say "I'm here, I'm listening, keep going."

My agent didn't do any of that. While the caller talked, it sat in complete silence. No reactions. No acknowledgment. Nothing. It waited for them to finish making sounds and then produced a response. Technically correct. Conversationally dead.

The hardest thing my agent had to learn wasn't talking. It was listening.

This is the hardest layer because the caller is actively talking. Any acknowledgment needs to land at a natural pause, not cut them off mid-sentence, match the tone of what they're saying, and play on a separate audio channel so the system doesn't think the agent is taking its turn.

The speech-to-text engine sends partial transcription results every 100-200 milliseconds as the caller speaks. The system monitors the gaps between these updates. A short gap, under 300ms, means the caller is mid-sentence. Words are still flowing. A longer gap, 500-700ms, means they paused. Collecting their thoughts. Taking a breath. That's the window.

The pause threshold isn't fixed. It's configurable on a 1-10 sensitivity scale. At low sensitivity, the system only triggers on long pauses of 700ms or more. At high sensitivity, it catches shorter breaths at 300ms. There's also a minimum gap between consecutive acknowledgments. Low sensitivity waits 8 seconds between them. High sensitivity, 3 seconds. Without this limit the agent says "mm-hmm" after every sentence, which sounds less like a listener and more like a parrot.

And nothing triggers until the caller has been speaking for at least 2 seconds. If someone just said "yes," they don't need acknowledgment. They need a response.

When someone describes a problem, the right response is "I see" or "oh." When someone is listing things, it's "okay" or "mm-hmm." When someone is excited, it's "oh nice." When someone is upset, it's "I understand" or "I'm sorry about that."

The system detects the caller's intent using two layers. The first is keyword matching. Instant, sub-millisecond. It scans the partial transcript for indicators across 12 intent categories.

The second layer is a model fallback for ambiguous cases where keywords aren't enough. A fast model classifies the intent in the background, debounced so it only fires when the transcript has grown by at least 3 new words since the last classification.

Once the intent is determined, the system randomly picks from the mapped sounds. This randomness matters. Hearing the exact same "mm-hmm" on every pause sounds robotic. Variety creates the illusion of a real person reacting naturally.

The audio plays on a dedicated background track, separate from the primary speech channel. This is critical. The agent's turn detection only monitors the primary channel. If acknowledgments played there, the system would think the agent is "speaking" and would cut off the caller's turn. On a separate track, the caller keeps talking uninterrupted.

All 11 sounds are pre-synthesised at call start using the agent's configured voice. Generated concurrently during initialisation and cached. When a back-channel triggers, there's no text-to-speech delay. The audio plays immediately.

The caller finishes a sentence. There's a gap before the model produces its response. Typically 200-500ms minimum, often much longer. In that silence, the caller wonders if the agent heard them.

The fast pre-response system fills this gap with a contextually appropriate filler. "Got it." "Sure." "Oh no, sorry about that." Spoken in under 100ms while the main model generates its full response in parallel.

The filler selection runs on two dimensions. Message type and sentiment.

Message type is a simple rule cascade. Contains "can you repeat" or "say that again?" That's a clarification request. Starts with "hi" or "hello?" That's a greeting. Starts with "yes" or "correct?" That's agreement. Contains numbers or is 5+ words? Information. And so on.

Sentiment scans for positive indicators ("thanks," "great," "awesome," "appreciate") and negative ones ("problem," "frustrated," "terrible," "broken"). Whichever count is higher wins.

The matching priority means a negative-sentiment information message hits "Oh no, sorry about that" rather than "Got it." Sentiment wins over message type because empathy is more important than accuracy. Saying "Got it" when someone just told you something went wrong sounds dismissive.

These rules are fully configurable. The agent creator can customise which phrases map to which conditions through the frontend. The defaults sound like a real receptionist, but a law firm might want more formal language while a surf school might want something more casual.

Real people don't acknowledge everything. Sometimes they just listen. Sometimes they make a small noise. Sometimes they say nothing and let the silence do the work.

The system replicates this by randomly skipping 30% of filler responses. The system rolls a random number, and if it's under 0.30, no filler is spoken. The caller goes straight to the full response after the normal processing delay.

This makes the agent feel unpredictable in a human way. Consistent acknowledgment of every single statement is a dead giveaway of automation. Real conversations have rhythm and variation. Sometimes you say "got it." Sometimes you just let the person keep going.

But there are exceptions where the skip is overridden.

Greetings are never skipped. If the caller says "Hi," the agent always acknowledges immediately. Silence after a greeting is deeply awkward.

Negative sentiment is never skipped. If the caller expresses frustration, worry, or dissatisfaction, the agent always responds with empathy. Silence after "I'm really worried about this" would feel dismissive.

And then there are empathy triggers. These detect genuine distress. Medical concerns, bereavement, anxiety, financial worry. When triggered, the system responds immediately with the appropriate tone and never skips.

"Can you hear me?" will always get "Yep, I can hear you!" regardless of conversation context. "What's your name?" will always get "I'm [agent name]! How can I help?" There's no reasoning to do. Calling a language model for these is pure waste.

The instant reply system maintains a library of pattern-response pairs across 8 categories. Connection checks, identity questions, small talk, politeness, jokes, capability questions, confirmations, and farewells. Each message runs through a three-stage matching cascade.

Each category has multiple response options, randomly selected to prevent repetition. Some use template variables. "I'm {agent_name}! How can I help?" becomes "I'm Sarah! How can I help?"

When an instant reply fires, the model is never called. The response is spoken in under 50ms. On a typical 5-minute call, instant replies handle 3-5 exchanges. Connection checks, thank yous, identity questions. That's 3-5 model calls saved, and each one answered faster than any model could even begin generating.

Let me walk through a real scenario showing all three layers working together.

Look at the timing of a single exchange without these layers versus with them.

The perceived latency drops from "uncomfortably long" to "instant." The actual model inference time hasn't changed at all. The caller just never sits in silence while it happens.

Pre-synthesising 11 sounds at call start adds about 200-400ms to initialisation. On a call that lasts 5 minutes this is invisible. On a 15-second automated callback it's a noticeable delay before the agent first speaks.

The 30% skip rate is tuned for general receptionist conversations. A therapist bot would want a lower skip rate, maybe 10%. An automated survey would want a higher one, maybe 50%. The right number depends on the use case, and getting it wrong makes the agent feel either robotic (too much acknowledgment) or cold (too little).

The keyword-based intent classifier handles most cases but misses sarcasm and subtle emotional shifts. Someone saying "Oh that's just great" sarcastically will get matched as positive emotion. The model fallback catches some of these, but not all. For most business calls this doesn't matter. For sensitive contexts like healthcare or counselling it could.

And the empathy pattern exclusions are never truly complete. Every new edge case discovered in production means another exclusion rule. "I'm dying to try that" shouldn't trigger a medical concern pattern, but without an explicit exclusion for "dying to," it will. This is an ongoing maintenance cost.

That afternoon in the cafe changed how I think about conversational AI entirely. The agent's responses were already good. The voice was already natural. The intelligence was already there. But none of it mattered because the caller didn't feel heard.

Conversation isn't just about what you say. It's about the constant stream of signals that tell the other person you're present, you're engaged, and you care about what they're saying.

A good response after dead silence feels like talking to a machine that happens to be smart. An "mm-hmm" at the right moment followed by a good response feels like talking to a person. The model didn't change. The intelligence didn't change. The only thing that changed was whether the caller felt heard.