Sign-language translation (text/speech to signing avatar, sign recognition)

Many Deaf people use a sign language (in Germany DGS) as their first language and read written German only with difficulty, yet most digital information exists only as text or audio. AI pipelines translate short, standardised content into sign language and render it with a 3D avatar or generated video; a second direction recognises signing and outputs text. The technology is useful today for narrow, repetitive content (travel announcements, website introductions, leaflets) and is not a replacement for human interpreters in complex, high-stakes or conversational settings.

Problem & value

• Who: Deaf and hard-of-hearing sign-language users; public bodies, transport operators, universities, pharma and consumer companies that must or want to offer accessible information.
• Today: Information is produced as pre-recorded videos with human signers (slow, costly to update), booked interpreters, or not at all. Live announcements (platform changes, delays) are normally inaccessible.
• AI value: faster updates, 24/7 availability and lower marginal cost per additional message. The market operators report volume rather than outcome: Signapse states it generates "over 5,000 BSL train announcements daily" across the UK [3] — a throughput claim, not a quality or comprehension measurement; no vendor page I reviewed published comprehension or satisfaction metrics.
• Legal drivers (Germany): BITV 2.0 §4 requires public-body websites to provide explanations in DGS and Leichte Sprache on the start page [6]; §9 BGG gives the right to use DGS in administrative procedures at the authority's expense [7]; the Barrierefreiheitsstärkungsgesetz (BFSG) implements the European Accessibility Act for products and services [8].

Variants

• Text → sign (production): website content, forms, public-transport and station announcements, digital package leaflets (SignTime SiMAX applies its avatar to digital leaflets [4]), e-learning.
• Speech → sign (live): ASR + MT + avatar for events or announcements; latency and error propagation are the limiting factors.
• Sign → text (recognition): video of a signer to text/gloss. Mobile app research such as the Horizon 2020 project SignON targeted Irish, British, Dutch, Flemish and Spanish Sign Language with an avatar called EVA [2]. Continuous, open-vocabulary recognition remains a research problem.
• Private use: apps for Deaf people to communicate with hearing contacts; learning tools for hearing relatives (quality caveats apply).
• Sign lexicon / dictionary lookup: retrieval of individual signs, a low-risk, mature building block.

How it works

1. text / speech (ASR)
2. normalisation & simplification (LANG-010 style)
3. sign-language MT (text
4. gloss/notation or directly
5. pose sequence)
6. motion synthesis (lexicon + generated transitions, facial/mouth/non-manual markers)
7. avatar rendering (3D engine / neural video)
8. published video or live stream
9. human review by Deaf native signers before publication (HITL) recognition: video
10. pose/hand/face landmarks
11. sign recognition / translation model
12. text

• Why it is hard: sign languages are full languages with their own grammar; meaning is carried simultaneously by hands, face, mouth, head and body and by use of space. Glosses are lossy and there is no standard written form; datasets are small and narrow, and results on benchmarks such as RWTH-PHOENIX-Weather-2014T (DGS, weather news) do not transfer to open domains [1].
• Approaches: classic gloss pipelines; gloss-free end-to-end models (e.g. GFSLT-VLP, SignLLM); self-supervised pretraining on video (SSVP-SLT); pose-based models [1]. Production avatars from DGS/ÖGS vendors rely on curated sign lexica plus rules for short, template-like content rather than free generation.
• HITL: a Deaf reviewer approves every template, glossary entry and leaflet; live content is restricted to a validated sentence set.

Data preparation

The data is mostly small but linguistically delicate: the source texts (website, leaflets, announcement templates, often in a CMS, Word files on file shares or PDFs), a reference glossary of terms with approved signs, and, for recognition, video of signers plus sign-language corpora. Typical gaps are inconsistent wording, long and complex sentences, missing plain-language versions, no approved sign lexicon for company terms, and video recordings without consent or annotation. Access rights to personal video data are often unclear.

Concrete steps: - Reduce source text to short, standardised, plain-language modules and freeze them as approved templates (PREP-014, PREP-016). - Build a glossary of company and domain terms with Deaf reviewers (PREP-015, PREP-013). - For recognition, collect annotated and consented video, check purpose and legal basis (PREP-025), and pseudonymise faces where possible (PREP-022). - Create a small test set with Deaf experts to judge comprehensibility (PREP-018).

Tools: a CMS export, Paperless-ngx or a DMS for source texts, an annotation tool such as ELAN for video. Data effort is high, roughly 50 to 60 percent of the project, mainly the expert review and glossary, not technology. See GUIDE-003 for a readiness check.

Implementation paths

Which path for whom

• SMB: Path A, because mature avatar and video-translation services exist and a small firm only needs a few standardised texts, not its own sign-language model.
• Enterprise: Path B, because a company with many recurring announcements or manuals benefits from an own pipeline (glossary, approval workflow) around cloud avatar APIs.
• Public: Path B on a sovereign EU or German cloud, because procurement rules and accessibility law favour controllable hosting and documented Deaf-community review.
• Developer: Path C, because open pose-estimation and sign corpora allow own research and rendering stacks, accepting the lower quality of open models.

Path A — Buy (recommended)

• Signing-avatar services: SignTime SiMAX (Austria; DGS and ÖGS, website, film, real-time and leaflet use, avatar adaptable to corporate design) [4]; Signapse (UK; BSL and ASL, API and transport solutions; DGS listed as "coming soon") [3]; Charamel (Germany) coordinates the BMBF-funded AVASAG research project on text-to-sign avatars [5]. Other products exist (e.g. Hand Talk for Libras/ASL), but I could not verify current details.
• Human-signed video from Deaf-led studios remains the benchmark for quality and is the right choice for legally required BITV start-page explanations and sensitive content.
• Choose A when: content is short and recurring, the vendor supports your sign language (DGS support is still thin), a Deaf review process is part of the contract, and the vendor can host in the EU.

Path B — Build on cloud APIs

• ASR (EU-hosted Azure, Google, Mistral/Voxtral-class or Whisper-class services) and LLMs for text normalisation and simplification, then a vendor sign-production API (e.g. Signapse SignStream for BSL/ASL [3]). There is no general-purpose cloud "text to DGS" API from the large hyperscalers that I could verify; frontier LLMs and VLMs are not reliable sign translators and should be used only for preprocessing and QA assistance.
• Choose B when: you need integration into transport/IoT announcement systems or a custom app but accept a vendor-provided sign-production core.

Path C — Self-host (open source)

• Landmark extraction: MediaPipe Holistic/BlazePose/hand tracking (Apache-2.0) [9]; the pose-format library (MIT) handles .pose files, normalisation and augmentation [10].
• Research code and data: the sign-language-processing community resource lists tasks, models and datasets [1]; datasets such as RWTH-PHOENIX-2014T, How2Sign, BOBSL and the DGS-Korpus (research licences — check terms before commercial use). sign.mt is an open-source-oriented translation demo; verify its licence and language coverage before relying on it.
• Rendering: Unity/Unreal/Three.js avatars driven by retargeted poses; pose-to-video diffusion models for photoreal output.
• Hardware: HW2 (one 24–32 GB GPU) for inference and avatar rendering of short clips; recognition on-device (HW1/edge) is possible with landmark models. Effort is dominated by data and linguistics, not compute.

Path D — Custom training

Often necessary for any sign language or domain not covered by vendors. Needs sign-language video with expert annotation (ideally several hundred hours with diverse signers), a stable lexicon, a Deaf-led annotation team and consent management. Realistic only with research partners or public funding (AVASAG [5], SignON [2] are examples). For a single organisation, build a validated sentence/template set rather than a general model.

State of the art (2026)

• Production: robust for constrained domains (transport announcements, weather, template sentences); avatars no longer look as uncanny as earlier generations but non-manual signals (face, mouth, body) remain a quality gap compared with human signers.
• Recognition/translation research: gloss-free and LLM-based approaches (SignLLM, GFSLT-VLP) are closing the gap with gloss-supervised ones; large-scale self-supervised video pretraining (SSVP-SLT) improved ASL results [1]. Evaluation is unsettled: BLEU is questioned for sign output, SignBLEU was proposed, and test-set signer overlap inflates reported scores [1].
• Limits and failure modes: hallucinated or ungrammatical signs, wrong fingerspelling of names, regional dialects, lack of context/space references, domain shift, and bias toward the few well-resourced languages (ASL, BSL, CSL, DGS weather). Live open-domain speech-to-sign is not reliable enough for medical, legal or emergency use.
• Community position: acceptance depends on Deaf involvement; Deaf organisations generally stress that avatars may complement but not replace human interpreters. I could not retrieve a current DGB or WFD statement during this review, so the organisation's latest position should be checked before procurement.

Shared resources & combinations

• Building blocks: translation, llm-general, asr, avatar, pose-estimation, vlm, edge-runtime. The avatar/rendering stack is shared with GEN-003 (presenter videos) and CONV-018 (digital humans); the pose stack with VIS-024 (pose & gesture analysis).
• Combines with: LANG-010 (Leichte Sprache — simplify first, then sign), LANG-009 (translate foreign-language content to German before signing), DOC-026 (accessible documents/websites, BITV), SPEECH-004 (live captions as a parallel channel), SPEECH-003 (speech translation upstream), CODE-017 (accessibility testing of the host site).
• Hardware: one HW2 GPU box can serve ASR, simplification LLM and rendering for the same accessibility programme.

Effort, cost & KPIs

• Path A: weeks to onboard; per-minute/per-message or subscription pricing; the main cost is Deaf review and content preparation.
• Path B: 2–4 months for integration with announcement/CMS systems.
• Path C/D: 6–24 months and a research-grade team; justified only for unsupported languages or public programmes.
• KPIs: comprehension rate measured with Deaf test users, share of content reviewed by Deaf signers, update latency (text change → signed output), coverage (pages/announcements signed), complaint and feedback rate, cost per signed minute vs. human production.

People & change

Involve management (budget, accessibility commitment) and the department lead for communication first, then IT and legal/compliance (accessibility duties, copyright of signs). The decisive stakeholder is external: Deaf associations and Deaf reviewers must co-design and approve output, otherwise acceptance fails. Customers or citizens are the end audience and should test early prototypes.

Works council: normally no, because the system translates public content and does not monitor employees (§87(1) no. 6 BetrVG); it becomes relevant only if staff are recorded for recognition. Even then §90(1) no. 3 requires timely information if work processes change. Public bodies inform the staff council. A data protection officer is needed for recognition variants because video of people is personal data; run a DPIA (GUIDE-005) if signers are filmed at scale.

AI literacy: support staff who release content with a short briefing on avatar limits and the review duty (GUIDE-006). Communicate openly that the avatar supplements, not replaces, human interpreters for complex or legally relevant content. Start with a pilot of a few frequently used texts (GUIDE-009) and keep a Deaf reviewer as final approver. Typical concern: poor quality or loss of interpreter jobs; address it with scope limits and published review results. See also GUIDE-008.

Risks, regulation & compliance

• EU AI Act: typically limited risk. The avatar video is synthetic content: Art. 50 requires providers to make AI-generated audio/video machine-detectable and deployers to disclose deepfakes; Art. 50(5) requires the information to meet accessibility requirements [11]. Check current application dates of Art. 50 after the 2026 digital-omnibus discussions. Emergency or medical information could raise context-dependent obligations; sign recognition used for biometric categorisation or emotion inference must not be built (Art. 5 prohibitions).
• GDPR: TTS-free avatar output is not personal data, but recognition input video of a signer is personal data (face, body; potentially biometric when used for identification). Sign videos used as training data need consent and licence clarity; pose skeletons can still identify people [1]. Avoid cloning a real signer's likeness without a written agreement.
• Accessibility law: BITV 2.0 §4, §9 BGG, BFSG [6][7][8]; whether an avatar satisfies a legal requirement for DGS explanations is a legal interpretation — confirm with the responsible accessibility office before replacing human-signed videos.
• Quality and trust risk: wrong signs can mislead (timetable, medication). Mitigate with restricted content scope, Deaf review, visible label "automatically generated", a feedback channel and a fallback to a human interpreter or text.

Getting started

1. Select 20–50 high-volume, low-risk messages (station announcements, FAQ intro) and define the target sign language.
2. Recruit Deaf native signers and an accessibility advisory group before choosing a technology.
3. Test 2–3 Path A vendors on your content with Deaf reviewers; require DGS/ÖGS coverage, EU hosting and an update workflow.
4. Add disclosure labels, feedback and human fallback; document the legal basis (BITV/BFSG) with your accessibility officer.
5. Measure comprehension with Deaf users after one quarter and decide whether to extend scope.

Sources

1. Sign Language Processing community resource (tasks, datasets, limitations) — https://sign-language-processing.github.io/
2. SignON project (Horizon 2020) — https://signon-project.eu/
3. Signapse (BSL/ASL avatar translation, transport customers) — https://www.signapse.ai/
4. SignTime — SiMAX signing avatar — https://www.signtime.media/
5. AVASAG research project (Charamel, BMBF-funded) — https://avasag.de/
6. BITV 2.0 §4 — https://www.gesetze-im-internet.de/bitv_2_0/__4.html
7. BGG §9 (right to sign language in administrative procedures) — https://www.gesetze-im-internet.de/bgg/__9.html
8. BMAS — Barrierefreiheitsstärkungsgesetz — https://www.bmas.de/DE/Service/Gesetze-und-Gesetzesvorhaben/barrierefreiheitsstaerkungsgesetz.html
9. Google MediaPipe — https://github.com/google-ai-edge/mediapipe
10. pose-format library — https://github.com/sign-language-processing/pose
11. EU AI Act Article 50 (transparency) — https://artificialintelligenceact.eu/article/50/