Catalog / Speech & Audio / SPEECH-020

Mission-critical radio transcription & command extraction (air traffic control, maritime, BOS radio)

Transcribe controller-pilot, VTS/maritime VHF and public-safety dispatch radio, extract structured commands (callsign, level, heading) and feed them to controller tools, readback checks, safety nets and training simulators.

Parts needed to run it yourselfLanguage modelSpeechPrediction & time seriesVision

Air traffic controllers, vessel traffic services and public-safety dispatchers run their work over voice radio, and most instructions never reach a computer unless someone types or clicks them. Speech recognition tuned to noisy narrowband audio and strict phraseology can transcribe the channel, turn spoken instructions into structured data and check them against the controller's own inputs. Published SESAR results show command recognition above 85% and fewer manual inputs; the technology is moving from research into certified decision support, not autonomous control.

Problem & value

  • Today: a controller hears "Lufthansa 4 Alpha Bravo, descend flight level 120, turn left heading 250" and then must enter level and heading into the flight strip, label or CPDLC system by hand. Inputs add workload, can be late or wrong, and a pilot readback error can go unnoticed. Maritime VTS and BOS (public-safety) control rooms log radio traffic mostly by recording and listening back.
  • AI value: automatic pre-filling of controller inputs, detection of readback/hearback mismatches, searchable transcripts for incident investigation and quality review, objective workload measurement, and AI pseudo-pilots that replace human staff in training simulators.
  • Sourced evidence: the SESAR HAAWAII project (Idiap, DLR and partners with Isavia, NATS and others) reported word error rates below 5% and command recognition above 85% in operational-like tests; the ATCO2 project released a public ATC corpus and pipeline for open research [1][2][3]. Treat figures as airspace- and accent-specific, not guarantees.

Variants

  • Air traffic control (core): en-route, approach and tower; command extraction into electronic flight strips and radar labels, readback checking, safety nets (e.g. clearance conformance), controller workload estimation.
  • ATC training and simulation: ASR plus synthetic pilots (TTS, see SPEECH-006) replace "pseudo-pilot" staff; links to EDU-010.
  • Maritime: VTS centres, port authorities and GMDSS VHF channels; IMO Standard Marine Communication Phrases make the speech semi-structured, but multilingual and accent-heavy.
  • BOS / public-safety radio (TETRA, Digitalfunk): transcription and keyword search of dispatch radio for control rooms and logging; much less constrained than ATC, so accuracy is lower. Emergency call triage is covered separately in SPEECH-016 and CONV-021.

How it works

  1. VHF/UHF radio + PTT signal (or recorded audio)
  2. segmentation by push-to-talk / voice activity detection
  3. ASR for narrowband speech (acoustic model + domain language model / biasing)
  4. context biasing with live callsign list, waypoints, sector data from surveillance
  5. spoken language understanding: NER + command parser (callsign, command, value, unit)
  6. validation vs. phraseology grammar and plausibility (level, heading ranges)
  7. output: HMI proposal (pre-fill, controller confirms) | readback check | alert | transcript store
  • ASR: end-to-end transformer models (wav2vec 2.0-class, Whisper-class, conformer) fine-tuned on ATC speech replace the older Kaldi hybrid systems. The key gain comes from contextual biasing: the decoder is told which callsigns are currently in the sector, which cuts callsign errors sharply.
  • Understanding: a small rule/NER model or compact LLM maps the transcript onto a closed command schema (ICAO phraseology). Free-form LLM interpretation is deliberately avoided for operational outputs; use constrained parsing and reject out-of-grammar results.
  • Human-in-the-loop: the controller always sees the proposal and confirms (or ignores) it. Hearing and understanding the pilot stays a human task; the AI is a second reader, not the decision maker.

Data preparation

Data preparation is the largest part of this use case (about 60-70 % of effort). Source material: radio audio and push-to-talk metadata from voice recorders and logging systems, plus surveillance and flight-plan data (callsigns, waypoints, sectors) from the ATM or dispatch system. These typically live in proprietary recorders, tape or WAV archives and operational databases, often with restricted access.

Typical gaps: narrowband 8 kHz noisy audio, no transcripts, inconsistent local phraseology, missing time alignment with surveillance data, and strict access rights and retention rules. - Export and convert recordings to a standard format (PREP-005, PREP-007). - Transcribe a gold set: several hundred hours need human transcription by controllers or dispatchers; start with a few hours as an evaluation set (PREP-018, PREP-016). - Callsign, waypoint and phraseology lists for context biasing (PREP-015). - Segment by push-to-talk with VAD; label commands against a closed schema (PREP-019). - Protect: voices are personal data; apply retention limits and pseudonymisation (PREP-022).

Tools: Audacity/ffmpeg, pyannote, Label Studio, Kaldi or NeMo tooling.

Data preparation

Data preparation is the largest part of this use case (about 60-70 % of effort). Source material: radio audio and push-to-talk metadata from voice recorders and logging systems, plus surveillance and flight-plan data (callsigns, waypoints, sectors) from the ATM or dispatch system. These typically live in proprietary recorders, tape or WAV archives and operational databases, often with restricted access.

Typical gaps: narrowband 8 kHz noisy audio, no transcripts, inconsistent local phraseology, missing time alignment with surveillance data, and strict access rights and retention rules. - Export and convert recordings to a standard format (PREP-005, PREP-007). - Transcribe a gold set: several hundred hours need human transcription by controllers or dispatchers; start with a few hours as an evaluation set (PREP-018, PREP-016). - Callsign, waypoint and phraseology lists for context biasing (PREP-015). - Segment by push-to-talk with VAD; label commands against a closed schema (PREP-019). - Protect: voices are personal data; apply retention limits and pseudonymisation (PREP-022).

Tools: Audacity/ffmpeg, pyannote, Label Studio, Kaldi or NeMo tooling.

Data preparation

Data preparation is the largest part of this use case (about 60-70 % of effort). Source material: radio audio and push-to-talk metadata from voice recorders and logging systems, plus surveillance and flight-plan data (callsigns, waypoints, sectors) from the ATM or dispatch system. These typically live in proprietary recorders, tape or WAV archives and operational databases, often with restricted access.

Typical gaps: narrowband 8 kHz noisy audio, no transcripts, inconsistent local phraseology, missing time alignment with surveillance data, and strict access rights and retention rules. - Export and convert recordings to a standard format (PREP-005, PREP-007). - Transcribe a gold set: several hundred hours need human transcription by controllers or dispatchers; start with a few hours as an evaluation set (PREP-018, PREP-016). - Callsign, waypoint and phraseology lists for context biasing (PREP-015). - Segment by push-to-talk with VAD; label commands against a closed schema (PREP-019). - Protect: voices are personal data; apply retention limits and pseudonymisation (PREP-022).

Tools: Audacity/ffmpeg, pyannote, Label Studio, Kaldi or NeMo tooling.

Implementation paths

Which path for whom

  • Enterprise: Path C. Radio traffic is sensitive and operationally critical; on-premises open-source ASR on recorded audio gives control and cost stability, while operational controller positions come through the ATM vendor.
  • Developer: Path C. Whisper, NeMo and wav2vec 2.0 models fine-tuned on ATC data (e.g. ATCO2) run on HW2 and give full control over biasing and parsing.
  • Public: Path C. Authorities and control rooms need sovereign, on-premises operation; certified vendor modules (Path A) are used where an operational safety case is required.

Which path for whom

  • Enterprise: Path C. Radio traffic is sensitive and operationally critical; on-premises open-source ASR on recorded audio gives control and cost stability, while operational controller positions come through the ATM vendor.
  • Developer: Path C. Whisper, NeMo and wav2vec 2.0 models fine-tuned on ATC data (e.g. ATCO2) run on HW2 and give full control over biasing and parsing.
  • Public: Path C. Authorities and control rooms need sovereign, on-premises operation; certified vendor modules (Path A) are used where an operational safety case is required.

Which path for whom

  • Enterprise: Path C. Radio traffic is sensitive and operationally critical; on-premises open-source ASR on recorded audio gives control and cost stability, while operational controller positions come through the ATM vendor.
  • Developer: Path C. Whisper, NeMo and wav2vec 2.0 models fine-tuned on ATC data (e.g. ATCO2) run on HW2 and give full control over biasing and parsing.
  • Public: Path C. Authorities and control rooms need sovereign, on-premises operation; certified vendor modules (Path A) are used where an operational safety case is required.

Path A — Buy (for certified operational use)

  • ATM system vendors integrate speech recognition into controller working positions: Thales (FR), Frequentis (AT), Indra (ES), Leonardo (IT), Saab (SE), Raytheon / Collins and Searidge (CA, AI-based tower tools). Check roadmaps with your Air Navigation Service Provider (ANSP) such as DFS, DSNA, Austro Control, NATS or Isavia; most of these have pilots with research partners.
  • Maritime and public safety: VTS vendors (Kongsberg, Saab, Frequentis) and control-room suppliers (Frequentis, Motorola Solutions, Hexagon, Leonardo) offer recording/logging with speech-to-text modules. Check on-premises operation and EU hosting.
  • Choose A when: you are an ANSP, port authority or control-room operator and need certified, supported operational tools; building a safety-relevant stack yourself rarely pays off.

Path B — Build on cloud APIs

  • General cloud ASR (Azure AI Speech, Google Chirp, AWS Transcribe, Deepgram, Speechmatics (UK), Mistral Voxtral (FR), AssemblyAI) can transcribe recorded or low-criticality traffic, ideally with custom vocabularies/phrase lists for callsigns and waypoints. Out-of-the-box accuracy on 8 kHz noisy ATC audio is clearly worse than domain-tuned models, so benchmark on your own recordings first.

Path C — Self-host (open source)

  • Models/tools: Whisper (large-v3 / turbo) with faster-whisper or whisper.cpp, NVIDIA NeMo (Parakeet/Canary, FastConformer), wav2vec 2.0 / XLS-R via Hugging Face, Kaldi and Vosk for lightweight setups; pyannote for segmentation where PTT data is missing; spaCy or small LLMs for command parsing.
  • Hardware: HW1 for a single real-time channel on a good CPU/Jetson; HW2 (one 24-32 GB GPU) for 10-30 parallel channels plus fine-tuning small models. Redundancy and monitoring matter more than raw GPU size.

Path D — Custom training

Almost always needed for operational quality. Collect several hundred hours of transcribed local-sector audio (accents, frequencies, local phraseology), fine-tune an open model, add a domain language model and context biasing, and validate per airspace. Pre-train on ATCO2 [3][4], adapt on local data.

State of the art (2026)

  • ASR: fine-tuned end-to-end models reach word error rates in the low single digits on clean ATC test sets; generic models without adaptation are often an order of magnitude worse on real radio audio. Callsign recognition and number sequences are the hardest elements, since a single wrong digit changes the meaning. HAAWAII and later SESAR work (e.g. Digital Technologies for ATM / Solutions) show that biasing with surveillance data closes much of that gap [1][2].
  • Understanding: command extraction on standard phraseology is a mostly solved task; non-standard phraseology, code-switching between English and local language, and corrections ("disregard, descend level 100") remain failure modes.
  • Limits: poor radio quality, stuck microphones, overlapping transmissions, stress-induced speech, rare callsigns, and accents. False positives that look plausible are more dangerous than missed detections, so systems should present proposals with confidence and never act autonomously.

Shared resources & combinations

Effort, cost & KPIs

  • Path A: quarters; procurement, integration into the ATM system, safety case with the ANSP. Cost mostly in vendor licences and certification effort.
  • Path B/C: 3-6 months for a recorded-audio analytics pilot; 12-24 months for operational decision support including safety assessment.
  • KPIs: word and callsign error rate per sector, command recognition rate (command + value correct), false-alert rate per hour, latency, controller input events saved per hour, readback discrepancies found, acceptance by controllers (workload questionnaires).

People & change

Involve management, the department lead (operations or watch manager) and end users (controllers, dispatchers) from day one, as they are the experts for the gold-standard transcripts. Bring in IT, it-security, legal-compliance, the DPO and the works council before recordings are analysed. In ANSPs and public control rooms the staff council (Personalrat) is the counterpart.

Works council: yes. Analysing recorded radio traffic of identifiable staff is objectively suited to monitor behaviour and performance (§87(1) no. 6 BetrVG); also §90(1) no. 3 for AI-supported work processes. Agree explicitly that analytics serve safety and not individual performance evaluation (GUIDE-004).

DPIA is likely (voice recordings, systematic processing); AVV for any processor (GUIDE-005). AI literacy: train controllers on error patterns and limits as a support measure under Art. 4 AI Act (GUIDE-006). Start with offline analysis of recorded traffic in a small pilot group, with go/no-go criteria agreed with the safety department (GUIDE-009). Typical concern: automation replacing the controller; state clearly that the AI is a second reader and the human confirms every operational output.

People & change

Involve management, the department lead (operations or watch manager) and end users (controllers, dispatchers) from day one, as they are the experts for the gold-standard transcripts. Bring in IT, it-security, legal-compliance, the DPO and the works council before recordings are analysed. In ANSPs and public control rooms the staff council (Personalrat) is the counterpart.

Works council: yes. Analysing recorded radio traffic of identifiable staff is objectively suited to monitor behaviour and performance (§87(1) no. 6 BetrVG); also §90(1) no. 3 for AI-supported work processes. Agree explicitly that analytics serve safety and not individual performance evaluation (GUIDE-004).

DPIA is likely (voice recordings, systematic processing); AVV for any processor (GUIDE-005). AI literacy: train controllers on error patterns and limits as a support measure under Art. 4 AI Act (GUIDE-006). Start with offline analysis of recorded traffic in a small pilot group, with go/no-go criteria agreed with the safety department (GUIDE-009). Typical concern: automation replacing the controller; state clearly that the AI is a second reader and the human confirms every operational output.

People & change

Involve management, the department lead (operations or watch manager) and end users (controllers, dispatchers) from day one, as they are the experts for the gold-standard transcripts. Bring in IT, it-security, legal-compliance, the DPO and the works council before recordings are analysed. In ANSPs and public control rooms the staff council (Personalrat) is the counterpart.

Works council: yes. Analysing recorded radio traffic of identifiable staff is objectively suited to monitor behaviour and performance (§87(1) no. 6 BetrVG); also §90(1) no. 3 for AI-supported work processes. Agree explicitly that analytics serve safety and not individual performance evaluation (GUIDE-004).

DPIA is likely (voice recordings, systematic processing); AVV for any processor (GUIDE-005). AI literacy: train controllers on error patterns and limits as a support measure under Art. 4 AI Act (GUIDE-006). Start with offline analysis of recorded traffic in a small pilot group, with go/no-go criteria agreed with the safety department (GUIDE-009). Typical concern: automation replacing the controller; state clearly that the AI is a second reader and the human confirms every operational output.

Risks, regulation & compliance

  • EU AI Act (depends): Annex III lists critical-infrastructure safety components for road traffic and utilities, not air traffic. Aviation falls under Annex I Section B (Regulation (EU) 2018/1139, EASA), where only parts of the AI Act apply directly; high-risk requirements are brought in via future EASA delegated acts, so certification mainly runs through EASA/SESAR safety assurance. Public-safety radio transcription for logging is typically limited or minimal risk. Check Art. 4 AI literacy for operators [6].
  • Art. 5 prohibitions: do not use the system for emotion recognition of staff (controllers, pilots, dispatchers) in the workplace, e.g. stress or fatigue detection from voice (prohibited; narrow medical/safety exceptions are interpreted strictly). Workload measurement must use speech activity and task counts, not emotional state. Voice-based individual identification of crews or officers for performance monitoring needs a separate legal assessment.
  • GDPR: radio recordings contain personal data (voice, names, callsigns tied to crew). Legal basis (public task, safety obligations), retention limits, access control, works-council involvement (Germany: BetrVG/PersVG) and a DPIA are needed. Public-safety radio contents can include health and crime data.
  • Operational risks and mitigations: automation bias (design for confirmation, show confidence, train controllers); misrecognised numbers (double-entry check against surveillance data); degraded modes (fall back to manual entry); security of the audio chain; model drift across airspaces (monitoring and re-validation).

Getting started

  1. Define scope (post-event analytics first, or training simulator; avoid live operational use at the start) and involve safety, data protection and staff representatives.
  2. Collect and lawfully prepare 20-50 hours of local radio audio; have experts transcribe a 2-5 hour test set.
  3. Benchmark an open model (Whisper, NeMo, wav2vec 2.0) and one commercial ASR on this set; measure word, callsign and command error rates.
  4. Add callsign biasing and a closed command grammar; re-measure, then run in shadow mode with controllers before any safety case.

Sources

  1. SESAR Joint Undertaking — HAAWAII project (Highly Automated Air traffic controller Workstations with Artificial Intelligence Integration) — https://www.sesarju.eu/projects/haawaii
  2. Idiap Research Institute — HAAWAII speech recognition for air traffic control — https://www.idiap.ch/en/scientific-research/projects/haawaii
  3. ATCO2 project — Automatic understanding of air traffic control communications — https://www.atco2.org/
  4. Zuluaga-Gomez et al. — ATCO2 corpus: a large-scale dataset for research on automatic speech recognition and natural language understanding of air traffic control communications — https://arxiv.org/abs/2211.04054
  5. EASA — Artificial Intelligence Roadmap and concept paper on guidance for machine learning applications — https://www.easa.europa.eu/en/domains/research-innovation/ai
  6. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  7. OpenAI Whisper — https://github.com/openai/whisper
  8. NVIDIA NeMo ASR — https://github.com/NVIDIA/NeMo