Consumer scam call & message protection (on-device scam detection)

Fraudsters reach people by phone, SMS and messenger apps with fake bank, police, parcel or "grandchild in need" stories, increasingly with AI-cloned voices. Small language and audio models running on the phone (or in the operator network) can recognise the conversation pattern of a scam while it happens and interrupt it with a warning, without sending the conversation to the cloud. Benefit: protection at the moment of manipulation, which transaction monitoring at the bank (see ANOM-022) can only catch afterwards.

Problem & value

• Today: protection is mostly reactive: number blacklists and crowd-sourced spam labels, SMS filters, awareness campaigns by police and consumer bodies (German police advice covers Enkeltrick, "falsche Polizisten", Schockanrufe and messenger fraud [5]). Scripts change faster than blacklists, numbers are spoofed or rotated, and the victim is pressured in real time.
• AI value: classify the conversation or message content and intent (urgency, secrecy, authority impersonation, request for codes/remote access/transfers) instead of only the number, locally and in real time.
• Scale of the problem: the US FTC reports that scams reach people by phone, text and email and that social-media-originated losses in 2025 were eight times the 2020 level [4]; Microsoft reports blocking about USD 4 billion of fraud attempts in a year and describes vishing used to obtain remote access (e.g. Quick Assist) [1]. EU-wide figures should be taken from national police/regulator statistics (not verified for this file).

Variants

• On-device (consumer): scam detection in the messaging and phone apps of the OS vendor, using a small on-device model such as Gemini Nano; conversation content stays on the phone.
• Network-side (operator): telecom operators analyse call-signalling and traffic (number spoofing, call bursts) and, where legally allowed, content; overlaps with ANOM-024.
• Bank-side companion: in-app "you are on a call while paying" warnings, see ANOM-022.
• Seniors & family: trusted-contact alerts, call screening that answers unknown callers with an assistant.
• SMB: vishing protection for employees (CEO-fraud, fake IT support); see SEC-011, SEC-002.
• Private use: primary audience; free OS features or small subscription apps.

How it works

1. incoming call / SMS / chat
2. cheap pre-filters: number reputation, CLI authenticity, URL reputation, known-scam fingerprints
3. on-device processing: ASR snippet (calls) or text
4. small LLM / classifier (scam tactic + intent)
5. risk score with explanation ("caller claims to be your bank and asks you to move money")
6. UX: silent label | warning banner/haptic/audio cue | block | offer to hang up
7. optional: alert trusted contact, report number (user-consented, minimised data)
8. feedback loop: user reports + vendor threat intel update models and blocklists

• Key AI components: speech recognition for streaming audio, a small LLM or encoder classifier for scam tactics, optional synthetic-voice detection (cf. SEC-011), URL/domain classifiers.
• Human-in-the-loop: the user always decides; the system warns, it does not take the call over or accuse. Bank/operator escalations go to fraud teams.
• Design rule: detect behavioural patterns (urgency, secrecy, authority, payment/code request) rather than identifying persons.

Data preparation

Scam detection needs three kinds of data: labelled scam examples (scripts, messages, URLs, numbers), threat intelligence feeds, and for in-house builds the transcripts or texts users submit. In an SMB these rarely exist in-house; the practical source is vendor threat intelligence plus your own call logs, mail gateway reports and staff "this looks fishy" tickets.

Typical gaps are few labelled examples in German, scattered reports (mail, chat, phone notes), no consistent category scheme, and transcripts that contain personal data. Preparation steps:

1. Define a simple category scheme (phishing, vishing, CEO fraud, tech support) before collecting examples (PREP-019).
2. Harvest labels from existing records such as helpdesk tickets and blocked-mail logs (PREP-020).
3. Pseudonymise transcripts and numbers before using them for evaluation or training (PREP-022).
4. Build a small, expert-reviewed test set (PREP-018) and capture user corrections afterwards (PREP-021).
5. Check purpose and legal basis, including telecom secrecy for call content (PREP-025).

Tools are modest: a spreadsheet or ticket export, Whisper for transcripts, a pseudonymisation script. Rough share of effort: about 30 percent for buy-path pilots, 40 to 50 percent when you build your own detector.

Implementation paths

Which path for whom

• private: A. Built-in OS scam protection and caller-ID apps work out of the box and need no setup; local-only apps are an option for privacy-minded users.
• enterprise: B. Banks and operators need to integrate warnings into their own apps and fraud systems, and no product fits their case and data flows.
• developer: C. On-device small models (Gemma 3n, Whisper variants via llama.cpp) keep audio and messages on the phone, which is the privacy-preserving design.
• smb: A. Companies protect staff with spam-filter and caller-ID products or operator services; building a detector is not worthwhile for a standard problem.

Path A: Buy (recommended)

• OS-native features: Google's on-device scam detection in Messages/Phone on Android, call screening and live voicemail-style features on iOS, Samsung's equivalents. These are free, device-dependent, and region/language availability varies. Check current availability for your market (not verified here).
• Third-party apps / networks: caller-ID and spam-protection providers such as Hiya, Truecaller and Pindrop (enterprise voice fraud); operator services (e.g. Deutsche Telekom, Orange, Vodafone offer network or app-based spam/scam protection; confirm current product names).
• Operators and banks buy network analytics and in-app warning SDKs; check GDPR role split, EU hosting and telecom-secrecy (TKG/ePrivacy) compliance.
• Choose A when: you are a consumer, bank or operator and want fast time to value without model operations.

Path B: Build on cloud APIs

• For apps that cannot rely on OS features (banking apps, messengers, contact-centre tooling): classify user-submitted or consented text/transcripts with an EU-hosted LLM API (Mistral, Azure OpenAI EU regions, Claude or Gemini via EU regions), plus managed speech-to-text.
• Sending live call audio to the cloud is legally delicate (telecom secrecy, consent of all parties); prefer text snippets, user-initiated "check this message" flows and on-device pre-filtering.
• Useful for "forward suspicious message for analysis" services run by consumer organisations, banks or public bodies.

Path C: Self-host (open source)

• On-device/edge models: Gemma 3n E4B (open weights, text+audio+image input, about 4B effective size) [3], small Qwen/Llama/Phi models, Whisper variants via faster-whisper [7] or whisper.cpp; run with llama.cpp (Android/iOS/desktop, 4–8 bit quantisation) [6], ONNX Runtime or ExecuTorch.
• Server side (operators/banks): same models on one workstation GPU for text triage; Kaldi/Whisper streaming ASR for call-centre use cases.
• Hardware: HW1: modern phone NPU or CPU for 1–4B quantised models; HW2 for network-side batch/streaming analysis.
• Effort: high on the mobile engineering side (battery, latency, background execution limits of iOS/Android).

Path D: Custom training

• Worthwhile for operators, banks and security vendors: fine-tune a small classifier on labelled scam scripts, transcripts and messages in the local language (German dialects, Turkish, Russian, etc.), and a spoof/deepfake voice detector. Needs thousands of labelled, consented or synthetic examples plus continuously refreshed adversarial data, since scammers adapt. Distillation of a large model into a 1–3B on-device model is the common route.

State of the art (2026)

• On-device LLMs are now routine: Google exposes Gemini Nano through AICore and ML Kit GenAI APIs, including prompt and speech-recognition APIs on supported phones (Pixel 9–11, Galaxy S25/S26 and others) [2]; OS vendors use such models for scam detection of messages and calls with conversation content kept local, as announced by Google (details not re-verified here).
• Open multimodal edge models such as Gemma 3n accept audio directly and run in a ~4B-class memory footprint [3], making open implementations realistic.
• Threat shift: AI-generated storefronts, fake job offers, voice cloning and remote-access tech-support scams are documented by Microsoft [1]; defence combines product friction (warnings before granting remote access) with detection [1].
• Limits and failure modes: false positives (legitimate bank, doctor, school calls) erode trust; false negatives for novel scripts; dialect/accent ASR errors; adversaries probing the detector; background-execution and battery limits; coverage only on newer devices; calls over end-to-end encrypted messengers are usually inaccessible to operators. Voice-clone detection is an arms race and should not be sole evidence.
• Public benchmarks for scam-call detection are scarce; quote vendor numbers only with independent verification.

Shared resources & combinations

• Same building blocks (asr, llm-small, edge-runtime, text-classifier): PERS-019 child online safety (on-device protection), SEC-002 phishing and email security (text/URL classifiers), SEC-011 deepfake and voice-clone detection, CONV-002 voice agents (same ASR/streaming stack), ANOM-015 provenance of synthetic content.
• Downstream/upstream: ANOM-022 (bank-side APP-scam warnings, payee/mule intelligence), ANOM-024 (telecom fraud signals such as SIM-swap, Wangiri), ANOM-004 (platform fake accounts), ANOM-014 (identity forgery).
• Hardware: phone NPUs for the consumer layer; one HW2 GPU server can run streaming ASR plus a small LLM for an operator's or bank's analysis pipeline; threat-intelligence database shared across all of these.

Effort, cost & KPIs

• Path A: hours (enable features) for consumers; weeks for banks/operators to integrate vendor SDK or network service.
• Path B: 2–4 months for a "check this message" service or in-app warnings.
• Path C/D: 6–12+ months for a production mobile on-device detector with multilingual models, plus continuous threat-intel operations.
• Cost drivers: labelled multilingual data, mobile engineering and device matrix, threat-intel upkeep, support for victims.
• KPIs: scam detection rate and false-positive rate per 1,000 calls/messages, warning-to-abandon rate (users who stop after a warning), prevented-loss estimates (carefully defined), time-to-new-script coverage, user trust/opt-in rate, battery/latency overhead.

People & change

Who and when. Management decides scope and budget first. IT and IT security integrate and secure the solution, legal/compliance clears telecom secrecy (TKG) and consent for call content, and the DPO is involved from the start. Affected staff and, for banks and operators, customer-facing teams join the pilot before rollout.

Works council. Depends. For consumer products, none. If the company deploys call or message screening for its own employees, co-determination under §87(1) no. 6 BetrVG is likely, because the system logs calls and messages and is objectively suited to monitoring behaviour (BAG). In addition, §90(1) no. 3 requires timely information about AI-supported work processes. Public bodies involve the Personalrat.

Data protection. Messages and call audio are personal data. Conclude an AVV (Art. 28 GDPR) with every vendor; a DPIA (Art. 35 GDPR) is likely for large-scale analysis of communication content.

AI literacy. Support staff with a short briefing on what the tool flags, why false alarms occur and how to report misses (GUIDE-006).

Communication and control. Announce what is analysed and what is not, start with a volunteer pilot group, and let users always override a warning. Typical concerns are surveillance and false blocks; address them by on-device processing, no storage of content and a clear complaint route (GUIDE-004, GUIDE-008).

Risks, regulation & compliance

• EU AI Act: typically minimal risk: a consumer warning tool is not in Annex III and does not decide on access to services. Be careful that the system does not infer emotions from voice in workplace/education contexts (Art. 5(1)(f) prohibition; for other contexts emotion recognition is high-risk/Art. 50(3) transparency). Voice-clone detection and any AI-generated warning voice/text may trigger Art. 50 transparency duties [8]. AI literacy (Art. 4) applies to deploying organisations.
• GDPR / ePrivacy / telecom secrecy: call audio and message text are personal data and communications content. On-device processing without exfiltration is the privacy-preferred design; server-side analysis needs a clear legal basis, consent of the user (and, for call recording or analysis, the other party under national law, e.g. §201 StGB in Germany), data minimisation, and strict retention. Operators are bound by telecommunications secrecy (TDDDG/TKG in Germany).
• Consumer harm: wrongful blocking of legitimate calls (emergency, medical); mitigate with explainable warnings and easy override, never silent deletion of important messages.
• Mitigations: local processing, opt-in, transparent explanations, redress channel, red-team tests with scam transcripts, accessibility for older users.

Getting started

1. Inventory threats in your target group (e.g. use national police/consumer-agency reports [5]) and pick 5–10 scam scripts as test cases in your language.
2. Consumers/families: enable OS-native scam detection and spam protection, and agree a family code word; organisations: evaluate 2–3 vendor options (Path A) on real call/message samples.
3. For a custom service, prototype on text first: LLM-based classifier on 500 labelled scam/legit messages, measure false positives.
4. Add the audio path only after legal review (consent, telecom secrecy); test Gemma 3n or Whisper-small quantised on target phones for latency and battery.
5. Connect warnings to downstream controls (bank step-up, trusted contact) and track KPIs monthly.

Sources

1. Microsoft Security — Cyber Signals Issue 9: AI-powered deception — https://www.microsoft.com/en-us/security/blog/2025/04/16/cyber-signals-issue-9-ai-powered-deception-emerging-fraud-threats-and-countermeasures/
2. Google — ML Kit GenAI APIs (on-device Gemini Nano, AICore) — https://developers.google.com/ml-kit/genai
3. Google DeepMind — Gemma 3n E4B-it model card — https://huggingface.co/google/gemma-3n-E4B-it
4. US FTC — Data Spotlight (April 2026: USD 2.1 billion reported losses to scams starting on social media in 2025, an eightfold increase since 2020) — https://www.ftc.gov/news-events/news/press-releases/2026/04/new-ftc-data-show-people-have-lost-billions-social-media-scams
5. Polizeiliche Kriminalprävention — Betrug (Enkeltrick, falsche Polizisten, Schockanrufe, Messenger-Betrug) — https://www.polizei-beratung.de/themen-und-tipps/betrug/
6. llama.cpp — https://github.com/ggml-org/llama.cpp
7. faster-whisper — https://github.com/SYSTRAN/faster-whisper
8. EU AI Act Article 50 (transparency obligations) — https://artificialintelligenceact.eu/article/50/