Spoken language & dialect identification (e.g. asylum origin assessment)

Language identification (LID) tells you which language is spoken; dialect identification (DID) goes one level deeper, for example separating Gulf, Levantine, Egyptian and Maghrebi Arabic. Businesses use LID to route calls and media to the right queue or recogniser. Migration authorities use DID as one advisory indicator when a person's claimed origin is unclear. The first use is routine engineering. The second is an Annex III high-risk AI system with serious fundamental-rights and accuracy concerns.

Problem & value

• Routing today: contact centres and media archives get audio in dozens of languages. Callers press menu keys or agents guess, and mislabelled audio is sent to the wrong ASR model or interpreter. Automatic LID fixes this in the first seconds of a call.
• Asylum procedures today: case officers often lack reliable ways to check a claimed region of origin. Language analysis by human experts is scarce and slow. Germany's BAMF has used a dialect-identification assistant (DIAS) and a name-transliteration tool (TraLitA) since 2017 as decision support for case officers, alongside human language experts [1][2].
• AI value: consistent triage, faster routing, less waiting for experts. No credible, sourced figure shows that DID alone can establish a person's origin, and none is claimed here.
• Core limit: how someone speaks reflects where they grew up, where they lived afterwards, their social network and their education. It is not a proof of nationality or of persecution risk.

Variants

• Call and media routing (low risk): LID on IVR, call-centre and broadcast audio; chooses ASR model, interpreter queue or subtitle language (SPEECH-002, SPEECH-004, SPEECH-003).
• Multilingual ASR front end: Whisper-class models detect the language first, then decode.
• Asylum and migration triage (high-risk): DID on interview audio with a case-officer-facing indicator and name transliteration from Arabic, Cyrillic and other scripts.
• Linguistic research and archives: dialect atlases, endangered-language documentation, folklore collections.
• Private use: language-learning apps that detect which variety a learner speaks; no special concerns beyond voice-data privacy.

How it works

1. audio capture (interview, call, stream)
2. quality gate (duration, SNR, speech activity detection, single-speaker segment)
3. feature extraction / speech foundation model embeddings (wav2vec 2.0 / XLS-R / MMS / Whisper encoder)
4. LID classifier (language set)
5. DID classifier (dialect set within the language)
6. calibration + open-set rejection ("unknown / insufficient evidence")
7. advisory report: top-k hypotheses, confidence, audio quality, known limits
8. human review (case officer + qualified language expert) ← mandatory in asylum use

• Models: an encoder pretrained on many languages produces embeddings. A small classification head, or a fine-tuned encoder, outputs language or dialect scores. Spoken DID usually combines acoustic and phonotactic cues with an ASR transcript, because lexical choice is a strong dialect signal.
• Calibration and rejection: scores must be calibrated on held-out data resembling real recordings (telephone, interpreter-mediated, noisy). Short, noisy or mixed-dialect audio must produce "inconclusive", not a forced guess.
• Transliteration (relates to LANG-009): rule-based and neural tools map names from non-Latin scripts to standardised Latin spellings, returning several variants so registers can be searched consistently.
• Human-in-the-loop: the indicator is shown beside, not instead of, the interview, documents and expert opinion. Officers must be able to see the confidence and the limitations, and to disregard the result.

Data preparation

Data: labelled speech samples per language and dialect, plus realistic test recordings (telephone, interpreter-mediated, noisy). In an authority or company these sit in case files, call-recording systems, media archives, file shares and sometimes on tapes. This use case has the highest data demand of the three.

Typical gaps: few labelled samples for rare dialects, labels set by non-experts or inconsistent between annotators, mismatch between clean training audio and real field recordings, no legal basis for reuse, and scattered storage with unclear access rights.

Preparation steps: 1. Define the language and dialect scheme with linguists before labelling (PREP-019). 2. Build annotated test sets with qualified language experts and measure annotator agreement (PREP-018). 3. Harvest existing labels from case records where lawful (PREP-020). 4. Check purpose and legal basis for reusing recordings (PREP-025) and pseudonymise exports (PREP-022). 5. Standardise audio (sample rate, channel, quality gate) and store it with metadata.

Tools: ffmpeg, Silero VAD, a labelling tool such as Label Studio, speech foundation models for embeddings.

Share of effort: often 50 percent or more, mainly expert labelling and legal clearance. See GUIDE-003.

Implementation paths

Which path for whom

• public: C. Voice data of asylum seekers and other vulnerable persons must stay under authority control, and open encoders run on a single HW1 server.
• enterprise: C. Contact centres and media archives handle customer voice data at volume, and a self-hosted classifier avoids per-minute costs and data transfer.
• developer: C. Open speech foundation models can be fine-tuned and calibrated on own data, which no off-the-shelf service offers for dialects.

Path A — Buy (off-the-shelf)

• Routing use: language detection is built into cloud speech and contact-centre platforms: Azure AI Speech (language identification), Google Cloud Speech-to-Text, Amazon Transcribe (identify language), Speechmatics (UK), Deepgram, AssemblyAI, plus contact-centre suites such as Genesys and NICE. EU options: Speechmatics (UK, with EU deployment options), Mistral (Voxtral models), self-hosted Phonexia (Czech Republic), which sells language and dialect identification to public authorities.
• Asylum and border use: no mainstream SaaS is appropriate. Public authorities buy bespoke systems or build in-house, as BAMF did [1]. Procurement must include AI Act conformity evidence.
• Choose A when: the goal is routing or ASR pre-selection among 10 to 100 major languages.

Path B — Build on cloud APIs

• Use the language-detection output of Whisper-class APIs, Azure AI Speech (EU regions), or Google Speech with EU location for the routing tier. Add an LLM only for downstream summarisation, never for origin judgments.
• Orchestrate with n8n or a message queue: detect language on the first 10 to 30 seconds, route, and log the score for audit.
• Not recommended for asylum use: voice data of vulnerable persons should not leave authority-controlled infrastructure.

Path C — Self-host (open source)

• Models: OpenAI Whisper (language detection head, 99 languages) [3]; Meta MMS (1,100+ languages, includes LID models) [4]; SpeechBrain recipes with ECAPA-TDNN trained on VoxLingua107 (107 languages) [5][6]; NVIDIA NeMo. Fine-tune an encoder such as XLS-R for dialect sets.
• Stack: PyTorch, ONNX Runtime, a small API service, an audit database. LID inference runs on CPU or edge devices; fine-tuning needs a GPU.
• Hardware: HW1 for inference, HW2 for fine-tuning a ~300M to 1B parameter encoder.
• Choose C when: audio is sensitive (always in migration use) or volumes are high.

Path D — Custom training

• Needed for any real dialect work: public data is limited. The VarDial Arabic Dialect Identification shared tasks (ADI) and the NADI series provide benchmark sets for Arabic [7], but they come from broadcast or social-media sources, not from interview conditions.
• Requires labelled speech with verified speaker provenance, balanced across gender, age and recording channel. Verified provenance is hard to obtain: self-reported origin is exactly what is in doubt in asylum cases, which makes ground truth circular.
• Consent, ethics review and a data protection impact assessment are prerequisites for any collection.

State of the art (2026)

• LID for major languages with clean audio is a solved engineering problem: foundation models detect the language within seconds with high accuracy, and Whisper-class systems do it as a by-product of ASR [3][4].
• Closely related languages and dialects remain hard. Published Arabic DID results are much weaker than LID results, drop on unseen speakers and channels, and depend heavily on training-set topic and recording conditions [7]. Treat any vendor accuracy claim as valid only for its test set.
• Failure modes: short or noisy samples; code-switching; speakers who migrated long ago or grew up in diaspora communities; interpreter-mediated or telephone audio; children; dialect continua with no sharp borders; under-represented varieties with little training data; bias by gender and age group; over-confident scores without calibration.
• Trend: self-supervised speech encoders and multilingual ASR now supply most features. The open question is not model strength but ground-truth quality and legitimate use.

Shared resources & combinations

• Same stack (asr + audio-classification): SPEECH-001 meeting transcription, SPEECH-002 call analytics, SPEECH-003 speech translation and SPEECH-004 captioning all start with language detection and can share one LID service (HW1/HW2).
• SPEECH-010 identifies the speaker, not language or dialect; do not merge the two, since combined voice profiles raise biometric concerns (VIS-016 for the face analogue).
• LANG-009 machine translation and transliteration handle names and interview text downstream.
• SPEECH-021 verbatim records can use LID to switch recognisers per speaker.

Effort, cost & KPIs

• Path A/B (routing): days to weeks; cost per audio minute is small relative to misrouted contact cost.
• Path C: 2 to 4 months for a production LID service; marginal cost is infrastructure.
• Path D (dialect, high-risk): many months to years, dominated by data collection, expert annotation, validation and conformity assessment.
• KPIs (routing): correct-queue rate, detection latency, share of "unknown" results, ASR word error rate after routing.
• KPIs (high-risk): per-dialect and per-subgroup error rates, calibration error, rejection rate, agreement with independent expert opinion, share of cases where the indicator was overridden.

People & change

Who and when: management and the department lead decide scope and the advisory-only role before anything is built. Involve the data protection officer, legal and IT security from the start, because voice data of vulnerable persons is highly sensitive. Case officers and qualified language experts, as end users, co-design the output format and the review step.

Works council / staff council: depends. The people analysed are not employees, but the system records how officers or agents use it. If it logs user activity, co-determination under §87(1) no. 6 BetrVG (or the Personalrat in the public sector) applies, as the BAG requires only objective suitability for monitoring. Inform it under §90(1) no. 3, see GUIDE-004.

Data protection: a DPIA is very likely (Art. 35 GDPR): voice features of vulnerable persons and new technology. Involve the DPO early, see GUIDE-005. Check the high-risk duties in GUIDE-014.

AI literacy: support officers with training to read confidence values and "inconclusive" results, and to disregard the indicator when it conflicts with other evidence (GUIDE-006).

Pilot and communication: run a shadow pilot where the tool output is hidden from decisions and compared with expert opinions. Typical concern: automated decisions on asylum cases. State clearly that the result is advisory and humans decide.

Risks, regulation & compliance

• EU AI Act: AI systems used by or for public authorities in migration, asylum and border control, including assessing the reliability of evidence or the origin of applicants, are listed in Annex III point 7 and are high-risk. Duties include risk management, data governance, logging, human oversight, accuracy and robustness, and registration in the EU database [8]. The routing use in commercial contact centres is generally minimal or limited risk.
• Art. 5 prohibitions to avoid: biometric categorisation that infers race, political opinions, religion or similar sensitive traits from biometric data is prohibited. A dialect classifier must not be used to infer ethnicity or religion, and must not be sold as such. Emotion recognition from voice in workplace or education settings is also prohibited. Review each deployment against Art. 5.
• GDPR: voice recordings are personal data; if used to uniquely identify, they become special-category biometric data (Art. 9). Origin and language indicators can touch on ethnic origin. A DPIA is required. Asylum data also falls under the Asylum Procedures Directive and national law. Purpose limitation and data minimisation apply to training data.
• Case law and standards: the CJEU held that methods used to assess an asylum applicant's credibility must respect fundamental rights and be based on sufficiently reliable methods (C-473/16) [9]. A dialect score is not a determination of nationality.
• Mitigations: advisory-only output, mandatory human expert review, documented error rates per subgroup, "inconclusive" as a first-class result, explanation of limits to officers, disclosure to the applicant and legal remedies, independent audit, no use of the score as sole ground for a decision, and logging for judicial review.

Getting started

1. Decide the purpose: routing (low risk) or origin assessment (Annex III high-risk, needs a legal basis).
2. For routing, run Whisper or SpeechBrain LID on 500 of your own recordings and measure the confusion matrix.
3. Add an "unknown" threshold and a logging field for scores; route low-confidence audio to humans.
4. For any authority use, start with the DPIA, the fundamental-rights impact assessment and a legal review before collecting a single training sample.
5. Evaluate per subgroup (gender, age, channel, dialect) and publish the limits to users.

Sources

1. BAMF (Federal Office for Migration and Refugees), digitisation and IT-assisted procedures, including dialect identification (DIAS) and transliteration (TraLitA) — https://www.bamf.de
2. Federal Government of Germany, answers in the Bundestag on automated language and dialect analysis in asylum procedures — https://dip.bundestag.de
3. OpenAI Whisper (language identification, 99 languages) — https://github.com/openai/whisper
4. Meta Massively Multilingual Speech (MMS) — https://github.com/facebookresearch/fairseq/tree/main/examples/mms
5. SpeechBrain — https://github.com/speechbrain/speechbrain
6. Valk and Alumäe, VoxLingua107: a dataset for spoken language recognition — https://arxiv.org/abs/2011.12998
7. VarDial / NADI Arabic dialect identification shared tasks — https://nadi.dlnlp.ai
8. Regulation (EU) 2024/1689 (AI Act), Art. 5 and Annex III — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
9. CJEU, Case C-473/16, F v Bevándorlási és Állampolgársági Hivatal — https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:62016CJ0473