Every STT evaluation starts with WER. That is the right instinct: word error rate is the primary accuracy metric, and it should be the first thing you measure. But enterprises that stop there consistently discover, three to six months into production, that a system with acceptable WER is failing in ways the benchmark never predicted.

The failures follow a pattern. The quality monitoring team cannot tell which transcript segments belong to the agent and which to the customer. The automated review workflow is flagging calls for human review at five times the expected rate because the confidence scores are uncalibrated. The live agent assist feature is lagging by 800 milliseconds, long enough for the agent to have already moved past the point where the prompt would have been useful.

None of these failures show up in a WER evaluation. All of them are predictable if you test for the right metrics before you commit to a vendor.

This guide covers the three STT metrics that enterprise buyers routinely underweight: speaker diarization accuracy, confidence score calibration, and transcription latency. For each one, it explains what the metric measures, why it matters for Indian contact center and BFSI deployments specifically, what good looks like, and how to test for it.

Speaker Diarization: Who Said What

What It Is

Speaker diarization is the process of segmenting a transcript by speaker. It answers the question "who spoke when?" rather than just "what was said?" In a two-party contact center call, diarization separates agent speech from customer speech. In a three-party interaction involving a supervisor, diarization assigns each segment to the correct speaker.

Diarization is a separate component from transcription. A system can achieve excellent WER, accurately transcribing every word, while completely misattributing those words to the wrong speaker. WER measurement does not capture this. You can have a perfect transcript with every word correctly transcribed but every word attributed to the wrong person.

For a plain-English definition of diarization and its relationship to other STT components, see our STT Glossary: Every Term You'll Encounter When Evaluating a Speech-to-Text API.

Why It Matters in Indian Contact Centers

Diarization is not a nice-to-have feature for Indian enterprise STT deployments. It is a prerequisite for three of the most common use cases.

Quality monitoring. Automated quality monitoring scores agent performance on language used, script adherence, empathy markers, and compliance disclosures. All of these metrics require knowing which speech segments belong to the agent. If diarization attributes 20% of customer speech to the agent, the quality scores will be wrong in ways that neither the system nor the reviewer will immediately detect.

Compliance review in BFSI. RBI-regulated entities must maintain accurate records of what agents said during customer interactions. In insurance, IRDAI requires that specific disclosures be made by the agent. These requirements are meaningless if the transcript cannot reliably distinguish agent speech from customer speech. For a full breakdown of regulatory compliance requirements for STT in financial services, see Speech Recognition for BFSI: The Accuracy, Compliance, and Language Requirements That Matter.

Collections call analytics. In collections, the ratio of agent talk time to customer talk time, the point in the call where payment commitment is made, and the exact language of consent confirmations all require accurate speaker attribution. A diarization error that attributes a customer's payment refusal to the agent creates a false compliance record.

The Indian-Specific Diarization Challenge

Standard diarization systems are tested primarily on clean, two-party English calls. Indian contact center audio presents three conditions that degrade diarization accuracy beyond what standard benchmarks predict.

Code-switching mid-sentence. When both agent and customer switch between Hindi and English within utterances, diarization systems that use language as a speaker signal break down. The system may incorrectly treat a language switch as a speaker switch. For a full treatment of code-switching and its effects on STT system components, see Why Hinglish Breaks Most STT APIs: The Code-Switching Problem in Indian Voice AI.

Overlapping speech and backchannels. Indian conversational norms include more frequent overlapping speech and backchannel responses, "haan," "theek hai," "okay" — than the call recordings most diarization systems were trained on. Backchannel responses from the listening party while the other speaker continues create short, overlapping segments that many diarization systems mishandle.

Variable audio quality between speakers. In a typical Indian BFSI collections call, the agent is on a VoIP connection from a contact center and the customer is on a mobile phone in a noisy environment. The acoustic profiles of the two speakers are radically different. Systems that use acoustic profile consistency to track speakers can mistake audio quality changes for speaker changes.

How to Test Diarization Before You Deploy

Pull 50 to 100 two-party calls from your production environment with known ground truth: calls where you know which speaker is the agent and which is the customer. Run each vendor's diarization on these calls and calculate the percentage of words correctly attributed to the right speaker.

Report two numbers: diarization error rate overall, and diarization error rate specifically at speaker transition points, where errors are most consequential. A system with acceptable overall diarization accuracy but high error rates at transitions will produce transcripts where the first and last few words of each speaker turn are misattributed.

If your deployment involves three-party calls, test on three-party audio. Diarization accuracy degrades with speaker count, and a system that handles two-party calls well may fail significantly on three-party interactions.

‍

Confidence Scores: Are They Telling You Anything Useful?

What They Are

A confidence score is a number between 0 and 1 that the STT system assigns to each word or phrase in a transcript, indicating how certain the model is about that transcription. A score of 0.95 means high certainty. A score of 0.35 means the model is guessing.

Confidence scores are the mechanism that allows enterprises to build intelligent human-in-the-loop workflows. Rather than sending every transcript to human review, which is expensive, or sending no transcripts to human review, which misses errors, a calibrated confidence score lets you route only the low-certainty segments for verification. In BFSI, this is the mechanism for ensuring that consent confirmations, disclosure statements, and payment amounts are verified on calls where the STT system was uncertain.

The Calibration Problem

Here is what most vendors do not tell you about confidence scores: they are frequently uncalibrated. An uncalibrated confidence score does not reliably predict transcription accuracy. A system might assign a score of 0.8 to segments that are wrong 40% of the time, and a score of 0.6 to segments that are wrong only 15% of the time. The number exists but conveys no reliable information about actual error probability.

Uncalibrated confidence scores are worse than no confidence scores, because they give the engineering team false assurance that the human review routing is working when it is not. Calls with high-confidence wrong transcriptions go through unreviewed. Calls with low-confidence correct transcriptions get flagged unnecessarily.

The calibration problem is particularly acute in Indian language and code-switched audio, where STT systems are operating further from their training distribution. Models tend to be overconfident on audio that sounds superficially familiar but contains unfamiliar vocabulary, and underconfident on audio with dialectal features they have not encountered. The result is confidence scores that are systematically miscalibrated for the audio conditions that matter most in Indian enterprise deployments.

What Good Calibration Looks Like

A well-calibrated confidence score has a predictable relationship between score value and actual accuracy. Segments scored between 0.9 and 1.0 should have WER close to zero. Segments scored between 0.5 and 0.7 should show meaningfully higher WER. Segments scored below 0.5 should have high error rates.

The practical test: take a sample of your test set transcripts with confidence scores. Bin them into score ranges: above 0.9, 0.7 to 0.9, 0.5 to 0.7, below 0.5. Calculate WER separately for each bin. A calibrated system shows monotonically increasing WER as confidence decreases. A system where the WER in the 0.7 to 0.9 bin is higher than in the 0.5 to 0.7 bin has an inverted calibration and its confidence scores cannot be trusted for routing decisions.

Why This Matters for Downstream Automation

Every downstream automation that uses STT output is implicitly relying on confidence score calibration. Intent detection models that receive uncertain transcripts need to know they are uncertain. CRM auto-population workflows that fill amount fields from transcript content need to know when the amount transcription was low-confidence. Compliance flagging systems that check for mandatory disclosures need to escalate segments where the disclosure transcription was uncertain.

If confidence scores are uncalibrated, all of these downstream workflows are making decisions based on noise. The cost is invisible until a compliance audit, a CRM data quality review, or a customer dispute surfaces the systematic errors.

For a full breakdown of how to evaluate STT systems on confidence score calibration as part of a structured vendor comparison, see How to Benchmark a Speech-to-Text API on Indian Languages Before You Sign Anything.

‍

Latency: The Metric That Only Matters When It Is Wrong

What It Measures

Transcription latency is the time between a word being spoken and that word appearing in the transcript. For batch transcription, latency is the time between submitting a completed audio file and receiving the transcript. For real-time streaming transcription, latency is the delay between live speech and the corresponding text output.

Latency matters only for real-time use cases. For post-call analytics, quality monitoring, and compliance review, a few seconds of additional processing time is irrelevant. For live agent assist, voice bots, and real-time compliance monitoring, latency is a hard constraint that determines whether the application is usable at all.

For a full breakdown of which contact center use cases require real-time transcription and which are better served by batch, see Real-Time vs Batch Transcription: Which STT Mode Does Your Contact Center Actually Need.

P95 Latency: The Right Metric

Average latency is a misleading metric for production STT deployments. A system with 150ms average latency and 2,000ms P99 latency will produce perceptible lag spikes in roughly one in a hundred requests. At contact center scale, one in a hundred requests is not a rare event. It is a daily occurrence across thousands of concurrent calls.

P95 latency, the latency value below which 95% of all requests complete, is the right metric for production evaluation. It captures the tail behaviour that determines whether a real-time application feels responsive under production load. For live agent assist, P95 latency under 200ms is the threshold below which the system feels immediate. Above 300ms P95, agents begin to perceive lag. Above 500ms, the assist feature becomes counterproductive.

Gnani STT API achieves P95 latency under 200ms in streaming mode, measured on production telephony audio under concurrent load.

The Concurrency Problem

Single-request latency tests are almost always misleading. A system that achieves 120ms latency on a single isolated request may show 600ms P95 latency when handling 500 concurrent streaming connections under peak contact center load.

The relevant test is P95 latency measured under a concurrency level that matches your peak call volume. If your contact center handles 300 simultaneous calls at peak, your latency evaluation should measure P95 under 300 concurrent streaming connections, not under a single test request.

Most vendors will not proactively offer to run this test. You need to request it explicitly and specify the concurrency level that matches your production environment. A vendor that declines or cannot provide P95 latency data under realistic concurrency is telling you something about how they have engineered their infrastructure.

How Latency Interacts With Noise and Code-Switching

Audio quality and code-switching affect latency in a specific way that is relevant to Indian enterprise deployments. When acoustic model confidence is low, which happens more frequently on noisy audio and code-switched speech, the decoder spends more time exploring alternative transcription paths before committing to output. This increases processing time, which increases latency.

The practical consequence is that P95 latency on clean English audio does not predict P95 latency on noisy Hindi-English code-mixed telephony audio. Systems that perform within latency thresholds on clean audio may exceed those thresholds on the audio profile of Indian BFSI contact centers. Latency evaluation must be run on the audio type that matches your deployment, not on clean benchmark audio.

For a full analysis of how noise degrades STT system performance on Indian telephony audio, see Speech Recognition in Noisy and Rural India: Why Your STT Fails Where It Matters Most.

The End-to-End Latency Stack

For voice bot deployments, STT latency is one component in a larger end-to-end latency stack. The full stack from customer speech to bot response includes: audio capture and streaming, STT transcription latency, NLU inference latency, response generation latency, TTS synthesis latency, and audio playback initiation. Each component adds to the total.

If the end-to-end target for a voice bot response is under 1,500ms, and NLU inference takes 200ms, TTS synthesis takes 400ms, and audio overhead accounts for 200ms, the STT budget is approximately 700ms. A P95 STT latency of 200ms gives comfortable margin. A P95 of 500ms consumes most of the budget and leaves no room for NLU or TTS variability.

Plan the end-to-end latency budget before specifying STT latency requirements. STT is not the only variable, and optimising STT latency at the cost of transcription accuracy will degrade the voice bot's ability to understand responses correctly.

‍

Building a Three-Metric Evaluation

WER, diarization accuracy, confidence score calibration, and latency should be evaluated together, not sequentially. Here is a practical structure for a combined evaluation.

Start with WER on your production audio, segmented by language and audio condition. This establishes your accuracy baseline per vendor. Do not proceed to the other metrics with vendors that cannot meet your WER threshold. There is no point evaluating diarization accuracy on a system whose transcription accuracy disqualifies it.

For vendors that meet the WER threshold, run the diarization evaluation: 50 to 100 production calls with known speaker ground truth. Calculate overall diarization error rate and transition-point error rate separately.

Run the confidence calibration test: bin transcripts by confidence score range and calculate WER per bin. Check for monotonic WER increase as confidence decreases. Any inversion disqualifies the confidence scores for routing workflows.

Run the latency test under realistic concurrency. Match the concurrency level to your peak production call volume. Measure P95, not average. For real-time use cases, apply the 200ms P95 threshold as a hard filter.

Build your vendor comparison matrix with all four dimensions. Weight them according to your deployment priorities. A post-call analytics deployment weights WER and diarization most heavily. A live agent assist deployment weights latency alongside WER. A compliance-grade BFSI deployment weights all four roughly equally because failures on any dimension create regulatory exposure.

For the complete multi-metric evaluation methodology including jiwer-based WER calculation, noise stress-testing, and vendor comparison matrix structure, see How to Benchmark a Speech-to-Text API on Indian Languages Before You Sign Anything.

‍

Frequently Asked Questions

1. ‍What is speaker diarization and why does it matter for contact centers? Speaker diarization segments a transcript by speaker, identifying who said what in a multi-party conversation. In a contact center call, it separates agent speech from customer speech. Without accurate diarization, automated quality monitoring, compliance review, and agent coaching workflows cannot distinguish between what the agent said and what the customer said, making their outputs unreliable or meaningless.‍
2. What is a good diarization error rate for a two-party contact center call? On clean two-party audio, well-configured diarization systems can achieve error rates below 5%. On Indian contact center telephony audio with code-switching and variable audio quality between speakers, error rates of 8 to 15% are more realistic. The key evaluation is transition-point accuracy: errors at speaker boundaries have more downstream impact than errors in the middle of a speaker turn.‍
3. How do I know if an STT system's confidence scores are calibrated? Bin a sample of transcripts by confidence score range and calculate WER per bin. A calibrated system shows monotonically increasing WER as confidence decreases. If a bin with confidence scores between 0.7 and 0.9 shows higher WER than a bin with scores between 0.5 and 0.7, the calibration is inverted and the scores cannot be used reliably for human review routing.‍
4. What P95 latency do I need for live agent assist? P95 latency under 200ms is the production threshold for live agent assist to feel responsive. Above 300ms P95, agents begin to perceive lag. The latency must be measured under your peak concurrency level, not on single isolated test requests. A system that achieves 150ms on a single request may show 600ms P95 under 300 concurrent streaming connections.‍
5. Does diarization accuracy degrade on noisy audio? Yes, significantly. Diarization systems that track speakers using acoustic profile consistency are more likely to confuse speaker changes with audio quality changes on noisy telephony audio. Background noise, codec distortion, and the acoustic difference between a VoIP agent line and a mobile customer connection all degrade diarization accuracy. Always test diarization on audio that matches your production environment, not on clean studio recordings.‍
6. Is diarization the same as speaker identification? No. Diarization segments a transcript by speaker without identifying who the speakers are: it labels segments as Speaker 1 and Speaker 2. Speaker identification matches segments to known identities using voiceprint data. In BFSI deployments, speaker identification using voiceprints is classified as biometric data collection under India's DPDP Act and requires explicit consent. Diarization alone does not create this requirement.

‍