Real-time accent & voice conversion for call agents

Offshore and multilingual contact centres struggle with mutual intelligibility: customers mishear agents, agents are judged on accent, and recruitment is limited to a few language markets. Streaming speech-to-speech conversion models rewrite the agent's outgoing audio live (about 200 ms of added latency is the order of magnitude vendors report), changing accent or translating while keeping the agent's own timbre, combined with noise suppression. It is an ethically contested use case: the technology is mature enough to deploy, but the transparency and workforce-ethics questions are the real design constraints.

Problem & value

• Today: BPOs train agents for accent neutralisation, script clarity and "voice and accent" coaching, and screen recruits on accent. Customers repeat themselves, handle time grows, and escalations to "native" agents follow.
• AI value: better first-call intelligibility, wider recruiting pool (agents in more locations or language markets), less accent-related training and attrition, and faster onboarding for new language lines.
• Evidence: Sanas is the market reference, vendor-reported use by 30k+ agents and 12 of the top 20 BPOs, a $65M Series B (announced Feb 2025) and a real-time translation launch (2025) [1]. Independent, peer-reviewed outcome data on handle time or CSAT is scarce; treat vendor claims as unverified and pilot with your own A/B measurement.
• Contested: critics argue that accent conversion reinforces accent bias instead of challenging it, and that customers are not told the voice is synthetic.

Variants

• Accent normalisation (same language): e.g. Indian, Filipino or Nigerian English to a General American, British or Irish target for English-language support.
• Real-time speech translation: agent speaks one language, customer hears another in the agent's timbre (overlaps SPEECH-003; here the agent is the sole converted party).
• Inbound-audio conversion ("hear the customer clearly"): convert the customer's accent for the agent's ear; this processes the customer's voice and therefore needs a stricter legal basis and notice.
• Noise and background-voice suppression only: the low-risk sub-variant, see SPEECH-008.
• Private use / gaming: voice changers for creators; out of scope here.

How it works

1. agent mic
2. VAD + noise suppression
3. streaming content/prosody encoder (accent-invariant)
4. target-accent decoder / neural vocoder with agent speaker embedding (timbre kept)
5. (optional) streaming translation head
6. softphone / SBC / CCaaS media path
7. customer ear; parallel tap
8. recording, ASR, analytics, disclosure marker

• Core model: a low-latency streaming speech-to-speech network (encoder-decoder with a speaker embedding) trained on paired or synthetic-paired accented/target speech. Chunked inference, a ~100-300 ms budget, and GPU or on-device NPU runtime are required.
• Deployment point: on the agent's PC as a virtual audio device (keeps audio local and reduces network hops) or in the cloud as a media-server hop (SIP/WebRTC).
• Human-in-the-loop: the agent can toggle conversion off, supervisors review quality samples, and a fallback to raw audio kicks in when latency exceeds budget.
• Failure handling: conversion errors can change phonemes and so the meaning of numbers, names and addresses; critical entities should be confirmed in text/SMS.

Data preparation

Little classic data preparation is needed because the product works on the live audio stream. What matters is the environment: the telephony platform (softphone, CCaaS, SIP trunk), headsets and network quality, and the consent and information given to agents and callers.

Typical gaps: unknown latency budget, inconsistent headsets, no list of which queues and sites are in scope, unclear agent consent, no recorded baseline of customer satisfaction and handling time. - Pilot recordings: collect before/after samples of volunteer agents (with consent) as a test set with domain experts (PREP-018). - Baseline: export handling time, quality scores and complaints from the CCaaS or CRM (PREP-005). - Voice data: voice is personal and possibly biometric data; define retention, no model training on agent voices without a basis (PREP-025). - Tools: the vendor's test mode, softphone logs, MOS or latency measurement.

Effort share is low (about 10-15 %); most work is integration, consent and change management.

Data preparation

Little classic data preparation is needed because the product works on the live audio stream. What matters is the environment: the telephony platform (softphone, CCaaS, SIP trunk), headsets and network quality, and the consent and information given to agents and callers.

Typical gaps: unknown latency budget, inconsistent headsets, no list of which queues and sites are in scope, unclear agent consent, no recorded baseline of customer satisfaction and handling time. - Pilot recordings: collect before/after samples of volunteer agents (with consent) as a test set with domain experts (PREP-018). - Baseline: export handling time, quality scores and complaints from the CCaaS or CRM (PREP-005). - Voice data: voice is personal and possibly biometric data; define retention, no model training on agent voices without a basis (PREP-025). - Tools: the vendor's test mode, softphone logs, MOS or latency measurement.

Effort share is low (about 10-15 %); most work is integration, consent and change management.

Data preparation

Little classic data preparation is needed because the product works on the live audio stream. What matters is the environment: the telephony platform (softphone, CCaaS, SIP trunk), headsets and network quality, and the consent and information given to agents and callers.

Typical gaps: unknown latency budget, inconsistent headsets, no list of which queues and sites are in scope, unclear agent consent, no recorded baseline of customer satisfaction and handling time. - Pilot recordings: collect before/after samples of volunteer agents (with consent) as a test set with domain experts (PREP-018). - Baseline: export handling time, quality scores and complaints from the CCaaS or CRM (PREP-005). - Voice data: voice is personal and possibly biometric data; define retention, no model training on agent voices without a basis (PREP-025). - Tools: the vendor's test mode, softphone logs, MOS or latency measurement.

Effort share is low (about 10-15 %); most work is integration, consent and change management.

Implementation paths

Which path for whom

• Enterprise: Path A. Real-time voice conversion needs very low latency and tuned models; mature vendor products integrate into the softphone or headset, so buying is faster and safer.
• Developer: Path B. Cloud voice and streaming APIs allow own prototypes; self-hosting needs HW3-class hardware for little gain.

Which path for whom

• Enterprise: Path A. Real-time voice conversion needs very low latency and tuned models; mature vendor products integrate into the softphone or headset, so buying is faster and safer.
• Developer: Path B. Cloud voice and streaming APIs allow own prototypes; self-hosting needs HW3-class hardware for little gain.

Which path for whom

• Enterprise: Path A. Real-time voice conversion needs very low latency and tuned models; mature vendor products integrate into the softphone or headset, so buying is faster and safer.
• Developer: Path B. Cloud voice and streaming APIs allow own prototypes; self-hosting needs HW3-class hardware for little gain.

Path A — Buy (recommended)

• Specialised vendors: Sanas (US, accent translation and real-time translation) [1]; Krisp (accent conversion plus noise cancellation, US/Armenia-founded) [2]; Tomato.ai (accent softening), and comparable offerings from CCaaS/telephony vendors such as Genesys, NICE and Five9 via partners. EU-based options are limited; check data-processing location and the sub-processor list explicitly.
• Choose A when: you need production quality and agent-desktop integration within weeks to months; building a sub-300 ms voice-conversion stack is rarely worth it.
• Check: end-to-end latency on your network, supported accents and languages, whether audio leaves the EU, retention of voice embeddings, a disclosure-marker feature, and works-council-ready documentation.

Path B — Build on cloud APIs

• Cloud APIs mostly provide pieces, not accent conversion: realtime speech-to-speech APIs such as OpenAI Realtime and Google Gemini Live, and ElevenLabs voice and agents APIs, which generate a voice rather than preserving the agent's timbre with low latency. Mistral's Voxtral offers EU-provider speech understanding.
• Accent conversion via a cascade (ASR → text → TTS in cloned voice) adds noticeable latency and loses prosody; it suits asynchronous or low-interactivity use, not natural conversation.

Path C — Self-host (open source)

• Building blocks: open voice-conversion tools such as RVC and OpenVoice, Seed-VC (zero-shot, streaming-capable), speaker-embedding encoders, Whisper or Parakeet for monitoring ASR, RNNoise/DeepFilterNet for denoising, and Kyutai's open streaming speech models (Moshi, Unmute) as a research base from an EU lab [3].
• Hardware: HW3 for a shared inference server; per-agent on-device inference needs a recent NPU or GPU laptop. Quality, stability and latency under load are typically below commercial products.
• Ops effort: high; real-time media engineering (jitter, packet loss, barge-in) is the hard part.

Path D — Custom training

Worth it only for a vendor or a very large BPO targeting specific accent pairs or low-resource languages. Requires consented, high-quality, labelled recordings of the agent population (hundreds of hours) and matched target-accent speech; all training data is biometric-adjacent personal data and needs explicit consent plus a deletion process.

State of the art (2026)

• Streaming speech-to-speech models with about 200 ms added latency are production-grade for major English accents; timbre preservation is good, prosody and emotion preservation is partial.
• Translation in the speaker's voice has moved from demo to product (e.g. Sanas, 2025) [1], but accuracy and turn-taking in noisy telephony (8 kHz narrowband) lag clean audio.
• Limits and failure modes: mis-rendered proper nouns, digits and codes; artefacts in overlapping speech; reduced accuracy for strongly accented or code-switching speech and for accents outside the training data; added latency that breaks conversational rhythm; uncanny or robotic output on poor lines; downstream ASR/analytics seeing converted rather than original audio.
• Bias: the target accent is a value judgement; performance differs between speaker groups and should be measured per accent.

Shared resources & combinations

• Same stack (speech-to-speech + tts + streaming): SPEECH-007 (voice cloning and dubbing, offline), SPEECH-003 (real-time interpreting), CONV-002 (voice agents), SPEECH-008 (denoising).
• Contact-centre stack: SPEECH-002 (call analytics, run on original and converted audio), SPEECH-010 (voice biometrics; conversion can break voice authentication on either side).
• Counter-use: the same technology powers voice-fraud and impersonation; see ANOM-025.

Effort, cost & KPIs

• Path A: weeks for a pilot with 20-50 agents; per-agent monthly licences; main cost is integration, network testing and change management.
• Path C/D: many months and a specialised real-time audio team; GPU servers.
• KPIs: added end-to-end latency (p95), mean opinion score and word error rate of converted versus raw audio, customer intelligibility ("could you understand the agent"), repeat-request rate, average handle time, CSAT/NPS, escalation rate, agent satisfaction and attrition, complaints about undisclosed synthetic voice.

People & change

Involve management and the department lead (contact centre), the works council and HR before any pilot, then end users, IT, legal-compliance and the DPO. This use case alters how employees sound and is sensitive for discrimination and transparency reasons, so involvement must be early and honest.

Works council: yes. The system processes agent voice and activity and is objectively suited to monitor behaviour (§87(1) no. 6 BetrVG, per BAG); the council must also be informed about AI-supported work processes (§90(1) no. 3) and may engage an expert (§80(3)). Agree a works agreement covering purpose, voluntariness, no performance evaluation and no recording (GUIDE-004). Public sector: Personalrat.

DPIA is likely (voice data of employees, systematic processing); see GUIDE-005. Art. 50 transparency: check whether callers must be told about synthetic voice elements. AI literacy measures: training for agents and team leads (GUIDE-006). Pilot with volunteers first, let agents opt out, and communicate clearly that the goal is understandability, not hiding identity (GUIDE-008). Keep humans in control: the agent can switch conversion off at any time.

People & change

Involve management and the department lead (contact centre), the works council and HR before any pilot, then end users, IT, legal-compliance and the DPO. This use case alters how employees sound and is sensitive for discrimination and transparency reasons, so involvement must be early and honest.

Works council: yes. The system processes agent voice and activity and is objectively suited to monitor behaviour (§87(1) no. 6 BetrVG, per BAG); the council must also be informed about AI-supported work processes (§90(1) no. 3) and may engage an expert (§80(3)). Agree a works agreement covering purpose, voluntariness, no performance evaluation and no recording (GUIDE-004). Public sector: Personalrat.

DPIA is likely (voice data of employees, systematic processing); see GUIDE-005. Art. 50 transparency: check whether callers must be told about synthetic voice elements. AI literacy measures: training for agents and team leads (GUIDE-006). Pilot with volunteers first, let agents opt out, and communicate clearly that the goal is understandability, not hiding identity (GUIDE-008). Keep humans in control: the agent can switch conversion off at any time.

People & change

Involve management and the department lead (contact centre), the works council and HR before any pilot, then end users, IT, legal-compliance and the DPO. This use case alters how employees sound and is sensitive for discrimination and transparency reasons, so involvement must be early and honest.

Works council: yes. The system processes agent voice and activity and is objectively suited to monitor behaviour (§87(1) no. 6 BetrVG, per BAG); the council must also be informed about AI-supported work processes (§90(1) no. 3) and may engage an expert (§80(3)). Agree a works agreement covering purpose, voluntariness, no performance evaluation and no recording (GUIDE-004). Public sector: Personalrat.

DPIA is likely (voice data of employees, systematic processing); see GUIDE-005. Art. 50 transparency: check whether callers must be told about synthetic voice elements. AI literacy measures: training for agents and team leads (GUIDE-006). Pilot with volunteers first, let agents opt out, and communicate clearly that the goal is understandability, not hiding identity (GUIDE-008). Keep humans in control: the agent can switch conversion off at any time.

Risks, regulation & compliance

• EU AI Act (Reg. 2024/1689): a typical deployment is limited risk with Art. 50 transparency duties. Providers of systems generating synthetic audio must mark outputs in a machine-readable way (Art. 50(2)); deployers of audio that resembles a real person and could appear authentic ("deep fake") must disclose it (Art. 50(4)); persons must be told when interacting with AI (Art. 50(1)). Whether accent conversion of a real agent counts as a "deep fake" is unsettled; the cautious path is a short disclosure to customers and a documented legal assessment [4].
• Art. 5 prohibition (flag): do not combine the voice stream with emotion recognition of employees at the workplace (Art. 5(1)(f)); this applies to sentiment or stress scoring of agents from voice. Customer-emotion analytics also need careful assessment.
• High-risk edge (Annex III, employment): using conversion or its quality scores for agent evaluation, promotion or termination could make the system high-risk.
• GDPR: voice is personal data; voice embeddings used to identify a person may be biometric data (Art. 9) if processed for unique identification, so avoid persistent voiceprints, use a consent-independent legal basis only after a DPIA, and inform customers of recording and third-party processing. Inbound-customer-audio conversion needs its own legal basis.
• Labour law: in Germany a works council has co-determination rights over technical systems monitoring performance (BetrVG s. 87(1) no. 6); agents must be able to opt out without penalty.
• Consumer/ethics: discrimination and accent bias concerns, deception of customers, and fraud misuse; Telephone fraud rules and national consumer law may require clear identification.
• Mitigations: disclosure to callers, on-device processing, no voiceprint retention, agent opt-out, bias testing per accent, DPIA, and logging of conversion state per call.

Getting started

1. Define the real problem (intelligibility versus recruitment) and baseline AHT, repeat-request rate and CSAT.
2. Run the legal and works-council assessment: DPIA, Art. 50 disclosure wording, agent opt-out.
3. Pilot 2 vendors on your own telephony path (8 kHz, VPN, softphone) and measure p95 latency and entity errors.
4. A/B test with and without conversion; sample converted calls for digits, names and addresses.
5. Roll out with a customer disclosure, a text confirmation for critical data, and quarterly bias review.

Sources

1. Sanas company and press pages (accent translation, 2025 funding, real-time translation) — https://www.sanas.ai
2. Krisp AI Accent Conversion — https://krisp.ai/accent-conversion/
3. Kyutai — Moshi and Unmute open speech models — https://github.com/kyutai-labs/moshi
4. Regulation (EU) 2024/1689 (AI Act), Art. 5 and Art. 50 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
5. European Commission — AI Act regulatory framework — https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
6. Seed-VC open-source voice conversion — https://github.com/Plachtaa/seed-vc