JOURNAL

Testing voice AI with Pipecat and Gemini Live: latency, sound and conversation quality

A practical voice AI test guide: configuration, latency timelines, tutor/interview prompts, 20-minute scenarios, distortion and echo diagnosis, logging and LingoLive results.

Voice AI response timeline: speech end, accepted turn end, received audio and first non-silent playback

Read with AI

Choose content to copy and paste into your AI assistant. Nothing is sent automatically. CMS content is converted to Markdown; original Markdown is used when available.

While building LingoLive, I ran into a familiar voice AI problem: the connection worked and the agent spoke, yet users could still experience distorted sound, slow responses or an agent hearing its own speaker output. A short demo could not tell me whether a person could comfortably learn or practise an interview through a longer session.

I now evaluate three things separately: transport availability, audio comfort and conversation usefulness. This is the practical method I am using with Pipecat and Gemini Live, including the limits of what our current tests establish.

In this guide

A connected call is only the first layer

Our short production checks connected both WebSocket and WebRTC, received audio and closed sessions. Successful WebRTC trials reported zero lost packets. None of that establishes whether a laptop speaker rattles, a child understands a question or a tutor consistently follows a parent’s translation preference.

Configuration tests check code contracts. Synthetic live sessions exercise the actual provider and application. Physical-device sessions exercise the microphone, speaker, headset and operating system. A mobile viewport is still a desktop browser with a different layout; it is not an iPhone or Android audio test.

Define where latency starts and ends

I distinguish the end of meaningful learner speech, the accepted turn end, the first returned audio bytes and the first non-silent output. Measuring only a model’s response misses endpointing and browser playback queues.

Voice response timeline showing learner speech end, accepted turn end, received audio and first non-silent playback
Received bytes and sound heard by a person are different events.

Pipecat service metrics help locate slow stages. Browser observations answer the separate question of how long the learner waits. I report sample counts, missing responses, median, p95 and maximum. Small samples deserve their individual values; a timeout must not disappear from the report to improve an average.

Start with the deployed architecture and configuration

Direct Gemini captures audio in the browser, converts it for the provider, receives output and schedules playback. Pipecat adds a server agent that manages a pipeline and transport. Our Pipecat path also uses Gemini Live. This comparison therefore concerns orchestration, transport, turn handling and playback; it is not a comparison between unrelated models.

For each run, save the commit, observed model, voice, SDK versions, transport, device, browser, effective prompt hash and turn policy. Our 13 October 2026 deployment reported gemini-3.8-live, Pipecat Python 1.12.0, client-js 1.13.1 and WebSocket transport 1.7.2. These identify the tested deployment; check availability and compatibility in your own provider account before reusing them.

On a phone, swipe tables horizontally to view every column.

ConfigurationCaptureCommon hidden failure
Input audioActual rate, channels and PCM/codecRelabelling the rate without resampling
Output audioDeclared rate, decoder and playback queueWrong playback speed or a growing queue
Turn endingVAD, silence window and manual endCutting off a hesitant learner
MicrophoneDevice and actual getSettings()Requested echo cancellation was not applied
PromptFinal prompt after profile compositionAge or translation preference disappeared
ServerCPU/RSS, event loop, sessions and regionWorker overload mistaken for model latency

Change one variable at a time, using the same input and task. Changing model, voice, VAD and transport together may improve a demo but leaves you unable to explain the improvement.

Work through a latency example before tuning

Suppose meaningful speech ends at 10,000 ms, the turn is accepted at 10,650 ms, audio arrives at 11,250 ms and output begins at 11,420 ms. These are illustrative values, not LingoLive observations. The person waits 1,420 ms; a metric starting at accepted turn end reports only 770 ms. Endpointing accounts for 650 ms of the wait.

endpointing = 10650 - 10000 = 650 ms
turn-to-playback = 11420 - 10650 = 770 ms
bytes-to-playback = 11420 - 11250 = 170 ms
user-perceived-gap = 11420 - 10000 = 1420 ms

Reducing the silence window might improve this number while interrupting a child who is thinking. Measure turn fragmentation alongside latency: how many accepted turns did one intended utterance become? Use a single monotonic browser clock, such as performance.now(), for browser intervals. Correlate server logs with a random turn ID rather than subtracting unsynchronized clocks.

A “bot speaking” callback establishes SDK state, not acoustic sound from a speaker. A loopback/reference recording is needed for acoustic timing. Keep no-response turns in the eligible-turn count, report timeout and missing-sample counts, and label successful-turn percentiles accordingly. A 15-second timeout must not become a zero-millisecond success.

Give each part of a 20-minute session a job

A silent long session mostly establishes that a connection survives. This proposed physical-device schedule exercises behavior as well. It is a protocol to run, not a claim that all its steps occurred in our synthetic 15-minute runs.

TimeActionEvidence
0–3 minStart, normal answer and opening speechConnection timing, audible quality, speaker echo
3–6 minSoft speech and a 1–2 second mid-answer pauseRecognition and turn fragmentation
6–9 minWrong answer, confusion and help requestTranslation, hint, attempt count and truthful praise
9–12 minInterrupt output, then remain silentQueue cancellation and false interruption
12–15 minConstrain connectivity; offline for 20 secondsUI, timeout, microphone and recovery route
15–18 minRestore and continue/restartContext retention and quota behavior
18–20 minEnd, read report and start againStopped microphone, closed server session, grounded report

The minimum matrix is laptop/speakers, laptop/headset, phone/speakers and phone/headset. Record OS, browser, headset type, volume and mic position. Treat Bluetooth as an additional route if supported. Children can take breaks; a checklist should never force them through a tiring session.

Isolate distortion and self-conversation

Keep the digital output and the sound heard on the device conceptually separate. Clean digital output with distortion only at high speaker volume points toward a device/gain investigation. Distorted digital output calls for checks of resampling, clipping and scheduling. These are diagnostic branches, not a conclusion about the cause of a particular complaint.

SymptomHypothesisDiscriminating test
Wrong pitch or speedRate/resampling mismatchPlay a reference at the declared rate; compare duration
Distortion on loud soundsClipping, gain or physical speakerInspect clipped samples and compare a headset
Clicks at chunk boundariesResampler continuity or schedulingCorrelate clicks with boundaries and underruns
Agent answers while user is silentEcho or false triggeringCompare headset/speaker and mic/transcript events
Latency grows over timeQueue, resource load, context or providerCorrelate interval latency with queue and resources
rms = sqrt(mean(x * x))
rmsDbFS = 20 * log10(rms)  # rms must be positive
clipFraction = count(abs(x) >= 0.999) / sampleCount
lossInterval = deltaLost / (deltaLost + deltaReceived)

For normalized PCM, these describe signal energy and clipping near the numerical boundary. Listen to the affected interval; a high isolated peak is not enough to diagnose distortion. For WebRTC, calculate loss from interval counter changes and convert jitter seconds to milliseconds. Missing fields remain missing. See the W3C statistics definitions.

For self-conversation, keep the learner silent during an entire tutor response. A learner transcript matching the spoken output is evidence worth investigating. Repeat with a headset. A child-oriented manual-turn mode can close input while the tutor speaks and reopen it for the child, with the trade-off that natural interruption is unavailable. An adult interview mode needs barge-in, echo handling and cancellation of queued old output.

Turn a tutor prompt into observable rules

“Be friendly and improve pronunciation” leaves the model too many decisions. Define the target, help behavior, language policy and stopping rule, then evaluate each observed reply. The following is an adaptable example, not a guarantee of compliance.

Role: beginner English tutor for a young child.
Ask one short question, then wait.
If translation=always: follow each English question
  with a short Vietnamese sentence of the same meaning.
When confused: simplify; do not introduce a new task.
Pronunciation: one target, one cue, one retry at a time.
Count only actual attempts to say the target.
Help, silence and skip requests are not attempts.
After at most 3 attempts: encourage and move on.
Praise effort; praise correctness only with evidence.
If audio is unclear: say so; do not invent a sound error.
End with an observed strength and one next activity.

For a hard attempt limit, application state is stronger than asking the model to remember a counter throughout a long conversation: target → attempt → hint → retry → mastered/skip. Let the model express support while state governs whether another retry is allowed. The transcript “lion” cannot establish that the child produced /l/ accurately; audio review must permit uncertainty.

ScenarioAcceptable behaviorFailure
“I don’t understand”Simple Vietnamese explanation and one easier questionThree additional English questions
Wrong animal guessAcknowledge effort and offer a useful hint“Perfect!” and record a correct answer
Third target attempt completedEncourage, change target or allow skipRequire a fourth repetition
Unclear microphone audioAsk again or offer a choice; acknowledge uncertaintyAssert a specific phonetic error

Evaluate a voice interviewer on interview behavior

An interview candidate needs time for a substantial answer and follow-ups grounded in that answer. Fast output is unhelpful if every thinking pause ends the turn. Keep one question at a time, but evaluate an answer length and silence policy appropriate for adults.

If the candidate says, “We had a checkout incident after a release. I coordinated rollback and added a canary,” a relevant follow-up asks about impact, the candidate’s responsibility or the rollback decision. Moving straight to “What are your strengths?” loses that thread. A report must not invent “50% less downtime” when no such result was stated.

Role: interviewer for a product/engineering lead.
Ask one question and let the candidate finish.
Follow up on a concrete claim in the last answer.
Ask for evidence when an impact is unspecified.
Do not invent responsibilities, dates or results.
Clarify a question without supplying its answer.
End with observed examples, one strength,
  up to three improvements and a short practice task.
Label advice as advice; do not give a hiring decision.

Score role relevance, grounded follow-up, completion of the candidate’s turn, evidence-based feedback and separation of communication from domain knowledge. Have two listeners score clarity, naturalness and listening effort independently. A 1–5 internal rubric supports configuration comparisons; it is not a standardized MOS study.

Separate network impairment from recovery semantics

Our short rerun used 400 ms latency, 32,000 bytes/second download, 16,000 bytes/second upload and a 20-second offline period. Units matter. CDP browser simulation does not cover every loss pattern, Wi-Fi roam or physical route change. For WebRTC, confirm the media path is impaired rather than assuming a slower HTTP request establishes that.

We verified a same-origin request failed while offline, then separately closed the native socket to exercise lifecycle handling. The first checks offline request behavior; the second injects a socket closure. Together they do not establish an actual cable/Wi-Fi outage test.

After restoration, label the outcome as same-context resume, fresh start or reload. Pipecat recovered through a fresh start in the observed run; Direct needed a reload/new session. Automatic history resumption has not been established. Also check microphone state, quota reservations and orphan server sessions.

Create logs that explain the next failure

“Connection failed” loses the information a developer needs. Keep a timestamped event chain and group dashboard results by engine, transport, version, network and outcome. Routine logs can carry metadata; authorized recordings/transcripts should have separate access and retention rules, particularly for children.

{
  "event": "voice.turn.audio_started",
  "runId": "random-test-run", "turnId": "turn-04",
  "engine": "pipecat", "transport": "websocket",
  "turnEndMs": 10650, "firstAudioMs": 11420,
  "responseGapMs": 770, "outcome": "response",
  "promptHash": "hash-of-effective-prompt",
  "synthetic": true
}
// Illustrative schema, not a collected production log.

Distinguish speech end, accepted turn, received audio, output start, interruption, socket closure, restart readiness and microphone stop. Add error stage and classification without logging API keys, bearer tokens, signed URLs or private learner data. A metric becomes actionable when its measurement boundary is explicit.

Read the LingoLive results at their actual scope

EvidenceObservationSupported conclusion
Two 900-second sessions; seven synthetic turns eachResponse diagnostics; no observed render errorsThe application path survived these synthetic sessions
Input attenuated to 8%Received a responseAmplitude tolerance observed; not a whisper benchmark
Vietnamese explanation requestedProvided helpThat help request worked
translation=alwaysLater questions omitted translationsProfile compliance failed
Post-fix native fault run; 180 seconds per modeFresh session data, no page error, stopped tracksShort lifecycle smoke passed
Physical phone and audio routesNot fully exercisedPhysical sound quality remains open

The seven Pipecat diagnostics were 492, 137, 1,791, 842, 1,085, 684 and 378 ms; Direct yielded 686, 858, 1,477, 1,089, 934, 590 and 651 ms. These expose variation but cannot rank candidates: different turn-end callbacks, tiny samples and an initial proxy harness make them unsuitable for a controlled comparison. Long-run network impairment evidence was excluded. The native rerun lasted 180 seconds, not 15 minutes.

The fatal-error path revealed a genuine race: both the application and SDK disconnected the recorder. A shared disconnect promise per one-use client prevents duplicate teardown. Regression checks, all 275 unit tests and the production build passed at that verification. The coaching failure remains a separate failure.

Run the reusable application harness

The repository includes Playwright scenarios, a fixture generator and a report template. Prepare isolated synthetic accounts, a manifest and WAV files; keep tokens private. These commands target LingoLive’s harness, not every voice application. Live tests consume provider resources, so use bounded duration/spend and the explicit opt-in.

$env:VOICE_QA_ALLOW_PAID='1'
$env:VOICE_QA_DURATION_SECONDS='180'
$env:VOICE_QA_FIXTURE='D:/private/qa-account-fixture.json'
node scripts/voice-qa/soak.cjs

# Once smoke is valid, run the long conversation
$env:VOICE_QA_DURATION_SECONDS='900'
node scripts/voice-qa/soak.cjs

Inspect results even after exit 0. Verify actual response audio, not merely control frames, and verify account identity before paid navigation. End clients, check server/mic cleanup, delete synthetic Auth/Firestore/Storage data and remove token files from every receiving host. Publish aggregates and failure classifications rather than private fixtures.

Release targets need enough evidence

My initial LingoLive targets include a response-gap median of at most 1.5 seconds and p95 of at most 3 seconds over at least 100 comparable turns, alongside timeout, recovery, microphone and coaching gates. They are proposed product targets to validate with a pilot, not vendor promises. Three successful starts cannot establish 99% reliability.

The repository now includes metric definitions, scenarios, a physical-device protocol, a live-test harness with paid-test opt-in and a report template for developers and agents. Results stay explicitly passed, failed, untested or inconclusive. Missing evidence is never converted into a pass.

WebSocket remains our default and WebRTC remains optional. Physical phone/headset sessions, room acoustics and real children’s pronunciation still need testing. I want to know what the product helps with, where it fails and what needs work before expanding it.

You can explore LingoLive or read the reusable test steps above. Start with a concrete conversation and a clear measurement definition; model selection becomes more useful once the desired outcome is explicit.

Discussion

Comments are reviewed before publication. Your email is kept private.

← Back to allĐọc tiếng Việt