Vocal biomarkers: voice- and cough-based health screening

Speech and cough carry signals of mood, cognition, motor control and lung and heart function. Models extract acoustic and paralinguistic features (or learned embeddings) from a short recording and estimate the likelihood of a condition or a change versus the patient's own baseline. Used as a low-cost screening and monitoring aid in telehealth and primary care. In the EU it is a medical device (MDR) and, under the AI Act, high-risk; it supports a clinician and does not diagnose alone.

Problem & value

• Today: depression and anxiety screening relies on questionnaires (PHQ-9, GAD-7) that are skipped or answered unreliably; cognitive decline and Parkinson's are found late; respiratory and heart-failure worsening is noticed only when symptoms are severe. Specialist assessment is scarce.
• AI value: a 25-second speech sample needs no hardware and can be repeated often, so it fits remote monitoring and call-based screening. Cough sound is an objective, cheap signal for TB and respiratory triage in low-resource settings.
• Evidence level (be sober): the largest peer-reviewed depression study (Kintsugi, Annals of Family Medicine 2025, about 14,000 US/Canadian adults, self-reported PHQ-9 >= 10 as reference) reported sensitivity 71.3% and specificity 73.5% [1]. That is a screening aid, not a diagnostic test, and the reference standard was a questionnaire, not a clinician. A 2025 systematic review reaches similar conclusions on heterogeneity and generalisability [2].

Variants

• Depression and anxiety screening in primary care, telepsychiatry and employee-assistance programmes (note Art. 5 limits below).
• Cognitive decline (verbal fluency, pauses, lexical features) from clinic-based or home tasks.
• Parkinson's and neurological monitoring (voice tremor, hypophonia, articulation).
• Respiratory: cough classification for COPD, asthma, pneumonia, COVID-19-era models, TB triage.
• Heart failure: voice and breathing changes as an early decompensation signal (see HLTH-007).
• Insurer and wellness use: health-risk scoring from voice is legally and ethically fraught (see Risks).
• Private use: consumer wellness apps; only wellness claims without a medical purpose avoid MDR, and the line is thin.

How it works

1. consent
2. capture (app / telehealth call / IVR, fixed prompt or free speech)
3. quality check (SNR, clipping, duration, language, device)
4. preprocessing (VAD, denoise, segmentation; optional ASR transcript)
5. features: openSMILE-style acoustic descriptors | self-supervised embeddings (wav2vec 2.0 / HuBERT / Whisper encoder / HeAR) + linguistic features from transcript
6. classifier or regressor per condition, calibrated; personal-baseline drift model for longitudinal use
7. report with uncertainty and abstain option
8. clinician review
9. confirmatory test (PHQ-9, spirometry, lab)
10. care decision

• Key components: a quality gate (most failures are bad audio), a speech or audio foundation model for embeddings, a small supervised head trained on clinically labelled recordings, and calibration per language, accent, age and device.
• Human-in-the-loop: the output is a flag for a clinician or care team; any diagnosis, therapy or safety decision (e.g. suicidality) is made by a human with a defined escalation pathway.

Data preparation

The data are short speech or cough recordings, device and noise metadata, and clinical reference labels (PHQ-9, MoCA, spirometry, NT-proBNP) plus consent records. Recordings come from smartphones, telehealth calls or IVR; labels sit in the EHR, study databases or paper questionnaires. Typical gaps: few labelled samples per condition, different microphones and noise, imbalanced cohorts, missing demographic diversity (dialects, age, languages), labels that are late or inconsistent, and voice as inherently identifying biometric data.

Preparation steps: define the consent and purpose per recording (PREP-025); standardise the recording protocol (task, length, quality check); harvest labels from existing clinical records (PREP-020); build a held-out evaluation set per device and subgroup with clinicians (PREP-018); pseudonymise audio and metadata (PREP-022); transcribe with a local ASR such as Whisper where text features are used (DATA-003). Tools: librosa or openSMILE for features, audio foundation models, Label Studio for annotation. Rough effort: 60 to 70 percent of the project; for bought products the deployer's share is mostly consent flows and integration, around 20 percent.

Implementation paths

Which path for whom

• Enterprise (A): Vocal biomarker screening is a medical-device function; providers and clinics should buy CE-marked, clinically validated products rather than validate their own models.
• Developer (C): Open audio models and small classifiers on modest hardware (HW1) let you train and validate on your own recordings, which is the route to your own device certification.
• Public (A): Public health services and statutory insurers should only deploy certified products with EU hosting; own builds only make sense inside a research setting with an ethics vote.

Path A — Buy (off-the-shelf)

• Vocal-biomarker vendors: Sonde Health (US), Canary Speech (US), Ellipsis Health (US), Kintsugi (US, depression), Winterlight Labs (Canada, cognition), Hyfe (cough tracking), Swaasa (India, cough and respiratory). EU-based: audEERING (Germany, audio AI and openSMILE), ki:elements (Germany, voice analytics for health and neurology), plus academic spin-offs. Regulatory status varies per product and market; ask for CE certificate (MDR class, intended purpose) or FDA clearance, not marketing claims.
• Choose A when: you are a provider or telehealth platform that needs a certified or clinically validated component; building your own would mean owning a medical device.

Path B — Build on cloud APIs

• Cloud ASR with word timestamps (Azure Speech, Google Cloud Speech-to-Text, Mistral Voxtral, EU regions) plus LLM analysis of transcripts for linguistic markers. Cloud audio-embedding APIs are scarce; Google's HeAR is released as open weights instead [3].
• Pure LLM scoring of mental state from transcripts is a research or wellness pattern, not a device-grade method. As soon as output is used for a medical purpose, MDR applies.
• Choose B for: research prototypes, pre-screening studies and internal tooling, not production diagnosis.

Path C — Self-host (open source)

• Feature extraction: openSMILE (eGeMAPS/ComParE feature sets), Praat/Parselmouth, librosa.
• Embeddings: wav2vec 2.0 / HuBERT / WavLM, Whisper encoder, Google HeAR (health acoustic representations for cough and breathing, open weights on Hugging Face) [3].
• Data: public cough sets such as COUGHVID (EPFL) and Coswara (IISc) for pre-training; clinical labels almost always have to be collected.
• Serving: ONNX or TFLite for on-device inference (data stays on the phone, strong GDPR benefit); server side on CPU or one small GPU. Hardware HW1 is enough; ops effort is mostly MLOps and data governance.

Path D — Custom training

Needed whenever you want your own validated claim. Requires thousands of labelled recordings from the target population with a trustworthy reference standard (clinician interview, spirometry, lab values), balanced for language, age, sex, device and recording environment, with a pre-registered validation cohort separate from training. Budget for a clinical study, ethics approval and MDR conformity assessment.

State of the art (2026)

• Foundation-model embeddings (self-supervised speech models, HeAR trained on large amounts of health-relevant audio) have replaced hand-crafted features as the default base; small heads on top reach competitive performance with little labelled data [3].
• Longitudinal, within-person baselines outperform one-shot population thresholds for monitoring.
• Evidence maturity: depression has the largest validation set (about 71-74% sensitivity and specificity in [1]); successor-model reports are preprints. Cognition, Parkinson's and cough results are mostly single-site studies with modest sample size.
• Limits and failure modes: confounding by microphone, room noise, language, accent, age, sex and medication; dataset shortcuts (the model detects the recording setup, not the disease); poor calibration across sites; small differences in prevalence collapse positive predictive value in general populations; speech changes are non-specific (tiredness, cold, intoxication).

Shared resources & combinations

• Same stack (asr, audio-classification, edge-runtime): SPEECH-001 transcription, SPEECH-010 speaker identification (identity, a different task but the same feature pipelines and the same re-identification risk), CONV-021 call triage.
• Combine with: HLTH-007 remote monitoring and wearables (fuse voice with heart rate, activity, weight for heart-failure decompensation), HLTH-004 clinical decision support, HLTH-012 deterioration early warning, HLTH-013 cardiology signals, CONV-009 wellbeing companion (voice flags should hand off to humans, never to the bot alone).
• Shared data pipeline: consent management, FHIR export, audio retention and pseudonymisation.

Effort, cost & KPIs

• Path A: weeks to integrate an SDK/API; pilot study with your own patients 3-6 months; per-assessment or per-patient pricing.
• Path B/C: 3-6 months for a research-grade prototype.
• Path D: 1-3 years including data collection, clinical validation and MDR certification; dominant cost is the clinical study and quality system.
• KPIs: sensitivity, specificity, PPV at local prevalence, AUROC with subgroup breakdown (language, sex, age, device), calibration error, rate of unusable recordings, referral-to-confirmation yield, clinician time per flag, patient adherence to repeated recording, false-reassurance incidents.

People & change

Involve management and the clinical lead for the decision and clinical responsibility, with legal/compliance for medical-device status (MDR) and liability. IT security and the DPO are required before any recording is stored: voice used to identify a person is biometric, and health inferences are special-category data (Art. 9 GDPR), so a DPIA and AVV with the vendor are needed. Affected patients must be informed and give consent; involve patient representatives in designing the information.

Works council: no for the patient-facing core, since employees are not the monitored group. It becomes relevant if the tool runs on staff calls, in occupational health or for staff wellbeing screening; then §87(1) no. 6 and §90(1) no. 3 BetrVG (or the Personalrat) apply, and such use should be avoided.

AI literacy (Art. 4 AI Act, support measures): teach clinicians what a score means, its false-positive rate, and that it is a screening flag and not a diagnosis. Start with a pilot cohort in one clinic with opt-in patients, keep the clinician as the one who acts on a result, and address concerns about surveillance and stigma openly. See GUIDE-005, GUIDE-009 and GUIDE-014.

Risks, regulation & compliance

• Medical device (MDR 2017/745): software that infers disease risk from voice for diagnosis, screening, monitoring or prognosis is medical device software; under Rule 11 it is typically class IIa and higher if it drives decisions with serious health impact. Needs a notified body, clinical evaluation, post-market surveillance [4]. A pure wellness claim avoids the MDR, but not the AI Act or GDPR if it effectively assesses health.
• EU AI Act (Regulation 2024/1689): AI that is a safety component of, or is itself, a medical device requiring third-party conformity assessment is high-risk (Art. 6(1), Annex I), with obligations for risk management, data governance, bias testing, logging, human oversight and accuracy; the application dates of the high-risk obligations are being adjusted by the 2025 Digital Omnibus proposal, so verify current timelines [5].
• Art. 5 prohibition (flag): inferring emotions of people in the workplace or in education institutions is prohibited, except for medical or safety reasons (Art. 5(1)(f)). Voice-based mood or stress screening of employees or students by employers or schools is therefore a high-risk design; do not deploy it outside a genuinely medical, clinician-led setting. Biometric categorisation inferring sensitive traits (Art. 5(1)(g)) must also be avoided.
• GDPR: voice recordings are personal data and often biometric; health inferences are Art. 9 special-category data. Need explicit consent or a clear Art. 9(2) basis, DPIA (Art. 35), purpose limitation (no reuse for insurance pricing, recruitment, credit), minimisation (on-device feature extraction, delete raw audio), and the Art. 22 limit on solely automated decisions. Voice can re-identify even after removal of names.
• European Health Data Space (EHDS): governs secondary use of health data for training and research.
• Other risks: false positives causing anxiety or stigma, false reassurance, discrimination by accent or language, function creep into insurer or employer scoring, safety escalation when a patient is flagged as at risk.
• Mitigations: intended-use statement limited to screening support, subgroup validation, abstain on poor audio, clinician review, escalation protocol, transparent information to patients, no third-party sharing.

Getting started

1. Pick one narrow clinical question and population (e.g. depression screening in primary care in one language) and define the reference standard.
2. Decide the regulatory route early: buy a CE-marked component, or plan the MDR and AI Act work if you build.
3. Run a retrospective test of a candidate tool on your own recordings, with subgroup analysis, before any live pilot.
4. Prototype with openSMILE and HeAR or wav2vec embeddings plus a logistic-regression head to understand the data and baseline.
5. Complete a DPIA, consent flow, escalation pathway and clinician workflow before collecting live patient audio.

Sources

1. Mazur et al., Evaluation of an AI-Based Voice Biomarker Tool to Detect Signals Consistent With Moderate to Severe Depression, Annals of Family Medicine 2025 — https://www.annfammed.org/content/early/2025/01/07/afm.240091
2. Speech and Voice Quality as Digital Biomarkers in Depression: A Systematic Review (PubMed) — https://pubmed.ncbi.nlm.nih.gov/40410060/
3. Google Health Acoustic Representations (HeAR) model on Hugging Face — https://huggingface.co/google/hear
4. Regulation (EU) 2017/745 (MDR) — https://eur-lex.europa.eu/eli/reg/2017/745/oj
5. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
6. openSMILE, audEERING — https://github.com/audeering/opensmile
7. Kintsugi study summary, Psychiatry.org — https://www.psychiatry.org/news-room/apa-blogs/vocal-biomarkers-for-mental-health
8. COUGHVID crowdsourced cough dataset (EPFL) — https://zenodo.org/records/4048312