Investigative intelligence & OSINT analysis (entity resolution, link analysis)

Investigators, compliance officers and journalists must connect people, companies, accounts and events scattered over registers, leaks, news and case files. AI resolves "the same person under five spellings", builds the ownership and relationship graph, translates and summarises foreign sources and drafts a cited brief. It shortens the search-and-collate phase; it does not decide guilt or risk. For authorities the legal frame (GDPR/LED, AI Act Annex III, national police law) shapes the architecture as much as the technology.

Problem & value

• Today: analysts work across dozens of portals and spreadsheets, copy names between tools and redraw link charts by hand. Name variants, transliteration, shell-company chains and multilingual sources make it slow and error-prone; leaks of millions of documents are unmanageable without search and graph tooling.
• AI value: (1) probabilistic entity resolution merges records without shared IDs; (2) LLM-based extraction turns unstructured documents into entities and relations; (3) graph analytics highlight hidden control structures; (4) LLM drafting produces cited briefs and translations; (5) agents run repetitive lookups across sources.
• Evidence of scale: the ICIJ Offshore Leaks database combines five leaks (Offshore Leaks, Panama, Bahamas, Paradise, Pandora Papers) with more than 810,000 offshore entities [4]. The open linkage library Splink states it can link about a million records on a laptop in around a minute and scale to 100+ million records on Spark [2]. We found no independently audited, general productivity figure for AI-assisted investigations; measure your own baseline.

Variants

• Corporate / financial-crime compliance (KYC/KYB, due diligence): ownership graphs, sanctions and PEP matching, adverse media — closest to ANOM-002, ANOM-016, DOC-011.
• Investigative journalism and NGOs: leak analysis, company networks, geolocation and verification (Aleph/OCCRP; Bellingcat-style open-source methods [5][6]).
• Tax, customs and fraud authorities: cross-source case building (ANOM-018, ANOM-020, ANOM-021).
• Law-enforcement analysis units: fusion of police databases with public sources; the most tightly regulated variant (see Risks).
• Security-agency variant (intelligence, defence analysis): same core capability — entity resolution, link analysis, briefings — but with classified data, sovereign/air-gapped deployment and national oversight law. The AI Act excludes systems used exclusively for military, defence or national security purposes (Art. 2(3)) [9]; German constitutional limits (Hessendata ruling) still apply to police-type data analysis [7]. Documented here at policy and architecture level only.
• Corporate security / brand protection: executive-protection and threat monitoring from public sources (see SEC-021, SEC-008).
• Private use: genealogy, due diligence on a landlord or seller. Public registers only; personal profiling of private individuals raises GDPR issues.

How it works

1. sources (registers, leaks, news/web, case files, sanctions lists)
2. collection & provenance capture (hash, timestamp, source URL; legal basis recorded)
3. OCR / parsing / translation
4. entity & relation extraction (NER + LLM with schema)
5. entity resolution (blocking
6. probabilistic/embedding match
7. human review of uncertain pairs)
8. knowledge graph (FollowTheMoney-style schema; people, companies, accounts, addresses, events)
9. analytics (centrality, communities, shortest paths, ownership look-through, timelines)
10. LLM briefing / Q&A with citations to source records
11. analyst review, decision, audit log

• Entity resolution is the core: Fellegi-Sunter style probabilistic linkage (Splink [2]), commercial engines (Senzing, Quantexa) or embedding + LLM adjudication for borderline pairs. Every merge must be reversible and explainable.
• Graph layer: property graph or RDF store; data model such as FollowTheMoney, used by OpenSanctions and Aleph [3][4]. GraphRAG-style indexing (entity extraction, Leiden communities, community summaries) helps answer corpus-wide questions [8] — see KNOW-013.
• Human-in-the-loop: analysts confirm merges, label hypotheses as "confirmed / plausible / unverified", and sign off briefs. Output is a lead, never a verdict.

Data preparation

Data comes from many places: company and beneficial-ownership registers, sanctions and PEP lists, court and procurement records, news and web pages, leaked document sets, and internal case files, transaction extracts and CRM or ERP master data. In an organisation the internal part typically sits in case-management tools, mail archives, file shares and paper files.

Typical gaps: scanned PDFs without text, the same person in five spellings or transliterations, no shared IDs across sources, missing provenance (who collected what, when, on what legal basis), mixed languages, duplicates and unclear access rights per case.

Preparation steps: (1) record legal basis and purpose per source and per case; (2) capture provenance (hash, timestamp, URL) at collection; (3) OCR and parse scans with Docling or PaddleOCR (see PREP-006, PREP-007); (4) map everything to one schema such as FollowTheMoney; (5) normalise names, addresses and dates; (6) build a gold set of known matching and non-matching pairs (PREP-018, PREP-020) and test Splink on it; (7) pseudonymise exports used for testing (PREP-022); (8) define access compartments per case (PREP-023, PREP-024, PREP-025). See also DATA-003 for pipelines.

Rough share of effort: 50 to 60 percent of the project goes into collection, normalisation, entity resolution tuning and access design; the LLM layer is the smaller part. Check readiness first with GUIDE-003.

Implementation paths

Which path for whom

• Enterprise: C — case files about natural persons and investigation methods are highly sensitive, volumes are high, and a self-hosted graph and open-weight stack avoids per-seat and API costs.
• Public: C — authorities need purpose binding, need-to-know compartments and sovereign or air-gapped operation, which commercial US platforms rarely give.
• Developer: C — the open building blocks (Splink, FollowTheMoney, yente, Neo4j or Memgraph) are mature and let you build a product without vendor lock-in.

Path A — Buy (off-the-shelf)

• OSINT / link-analysis workbenches: Maltego (Graph, Search, Monitor, Evidence, Data with 120+ data partners, ISO 27001:2022 certified [1]), Fivecast ONYX (Australia; public-source collection for government and corporate security [10]), Babel Street.
• Entity-resolution / decision intelligence: Quantexa (UK; entity resolution and knowledge graphs for banks, public sector and law enforcement; Belgian federal finance service listed as customer [11]), Senzing (embeddable ER SDK).
• Platforms for authorities: Palantir Gotham and similar data-fusion platforms (US vendors; sovereignty concerns in Germany/EU), and European alternatives for police and agencies; evaluate exit options and legal basis before procurement.
• Open platforms for journalists/NGOs: Aleph (OCCRP; document search, network diagrams, cross-checks against hundreds of datasets, limited support policy [4]).
• Compliance screening data: OpenSanctions (sanctions, PEP and related entities; bulk data and self-hostable yente matching service [3]).
• Choose A when: a team needs results within weeks and the legal basis for the vendor's data sources is clear. Check data residency, subprocessor chain and whether collection tooling is lawful for your mandate.

Path B — Build on cloud APIs

• Frontier LLMs (Claude, Gemini, GPT, Mistral — EU region/EU provider such as Mistral AI where possible) for extraction, translation and cited summarisation; structured outputs for entity/relation JSON.
• Orchestration: agent framework with MCP connectors to registers, OpenSanctions API, search APIs; graph in a managed graph database; Splink on DuckDB for resolution.
• Do not send case files about natural persons to non-EU APIs without a verified legal basis and transfer mechanism. Suitable for open-source-only research (journalism, corporate due diligence); usually unsuitable for police or classified material.

Path C — Self-host (open source)

• Resolution: Splink (MIT, DuckDB/Spark) [2]; data model/tools: FollowTheMoney, Aleph, yente [3][4]; graph: Neo4j Community, Memgraph, Apache AGE (PostgreSQL) or Kuzu-class embedded stores.
• Document intelligence: Docling/PaddleOCR for scans; OpenSearch for hybrid search; open-weight LLMs (Qwen3 / Llama / Mistral family) served with vLLM for extraction and drafting; multilingual embeddings and a reranker; a translation model for foreign-language material.
• Hardware: HW3 (one 48–96 GB GPU) for 30–70B-class extraction and briefing; HW2 suffices for small models plus classical ER. Air-gapped operation is practical because nothing needs the cloud, except the public-source collection step, which can be separated by a one-way transfer.
• Ops effort: significant — data engineering, access control (need-to-know, per-case compartments), logging.

Path D — Custom training

Worth it for (a) a domain NER/relation model for local languages, document types or case jargon, (b) a learned match model for your own entity types (fine-tuned embedding or cross-encoder trained on analyst merge/split decisions). Analyst corrections are the training data; a few thousand labelled pairs are usually enough for a pairwise matcher. Not worth training a base LLM.

State of the art (2026)

• Agentic OSINT: LLM agents run multi-source lookups and draft dossiers; Bellingcat's tests of LLMs for geolocation report big improvements but conclude humans remain key, and that results vary strongly between models [6]. Treat agent output as hypothesis generation.
• Entity resolution + LLM: classical blocking and probabilistic scoring still do the bulk work at scale; LLMs are used for hard pairs and for explaining merges.
• GraphRAG / graph-aware retrieval supports corpus-level questions over leaks and case files [8].
• Shared data standards: FollowTheMoney and open datasets (OpenSanctions, Offshore Leaks) make results reproducible and exchangeable between tools [3][4].
• Limits and failure modes: false merges (two people with common names fused — severe for individuals), missed merges (transliteration), hallucinated relations in LLM extraction, stale registers, source poisoning and disinformation, confirmation bias amplified by tidy graphs, and "guilt by association" artefacts of centrality metrics. Co-occurrence is not a relationship.

Shared resources & combinations

• Same stack (knowledge-graph + llm + embeddings + ocr-docparse): KNOW-013 (graph construction, GraphRAG), KNOW-002 (RAG over documents), DOC-021 (eDiscovery/investigations), DOC-011 (due diligence).
• Domain feeds and consumers: ANOM-002 (AML monitoring), ANOM-016 (sanctions/PEP screening), ANOM-018, ANOM-020, ANOM-021 (tax, procurement, customs risk), ANOM-023 (crypto forensics), SEC-008 (threat intelligence), SEC-021 (digital risk protection), VIS-050 and VIS-051 (media forensics and geospatial analysis — separate, more restricted use cases).
• Research companions: KNOW-015 (desk research agent), LANG-017 (competitive intelligence).
• Hardware/data: one HW3 server with a vector index, search engine and graph database serves several of these; entity-resolution tables and the graph schema are the reusable asset.

Effort, cost & KPIs

• Path A: weeks to onboard; licence per seat plus data-partner fees; high for premium data.
• Path B: 2–4 months for a research assistant with registers and sanctions data.
• Path C: 4–9 months for a production platform incl. security accreditation; GPU server plus data engineers.
• Path D: adds 1–3 months for labelling and evaluation.
• KPIs: precision/recall of entity matches on a gold set; false-merge rate; time from tip to first structured case file; share of brief statements with verifiable citation; analyst override rate; number of leads confirmed by independent evidence.

People & change

Involve management and the department lead at the start to fix mandate, legal basis and prohibited uses. Bring the data protection officer and legal or compliance in before any tooling is chosen: the data concerns natural persons, often special categories or criminal-offence data, so a DPIA (Art. 35 GDPR) is very likely and every external processor needs an AVV (Art. 28 GDPR); authorities follow the LED and national police law. Involve IT security early for compartmentalised access, logging and prompt-injection tests on scraped content.

Works council: depends. Where analysts' queries and decisions are logged per user, the system is objectively suited to monitor behaviour or performance, so co-determination under §87(1) no. 6 BetrVG applies; §90(1) no. 3 requires timely information when work processes are planned with AI, and the council may call in an AI expert (§80(3)). In public bodies the staff council (Personalrat) is responsible. Agree what the audit log may be used for.

AI literacy: support analysts' AI literacy (Art. 4 AI Act) with training on entity-resolution errors, hallucinated links and unsupported claims (GUIDE-006). Start with a small analyst pilot on three closed past cases (GUIDE-009). Typical concerns: wrongful identification and deskilling. Address them by showing confidence and sources, keeping every merge reversible and requiring a human decision for each consequential step; the output is a lead, never a verdict. Further reading: GUIDE-004, GUIDE-005, GUIDE-008.

Risks, regulation & compliance

• EU AI Act: depends on the deployer. Corporate due diligence and journalism: usually minimal/limited risk. Law-enforcement uses fall under Annex III point 6 (e.g. profiling in the detection, investigation or prosecution of offences, evidence-reliability assessment) and migration/border uses under point 7 — high-risk, "in so far as permitted by Union or national law" [12]. High-risk obligations now apply from 2 Dec 2027 after the AI Omnibus (in force 27 July 2026) [13].
• Prohibited in typical form (Art. 5, applies since Feb 2025): assessing the risk that a person commits a crime solely based on profiling or personality traits (Art. 5(1)(d)); building face databases by untargeted scraping of the internet or CCTV (Art. 5(1)(e)); biometric categorisation inferring sensitive traits (Art. 5(1)(g)) [14]. A system that supports a human assessment already grounded in objective, verifiable facts linked to criminal activity is outside the ban. Consequently, "risk scores for people" from scraped social media or mass face matching cannot lawfully be built in the EU; only fact-based, case-bound link analysis is documented here.
• National security exclusion: Art. 2(3) excludes systems used exclusively for military, defence or national security purposes [9]; mixed-use systems are covered.
• GDPR / LED: processing by authorities is governed by the Law Enforcement Directive (EU 2016/680) and national police law; companies and journalists fall under GDPR (Art. 6, 9, 10 data, DPIA under Art. 35, purpose limitation, data minimisation; journalism exemptions per national law). "Publicly available" does not mean "free to process".
• German constitutional limit: the BVerfG held on 16 Feb 2023 (1 BvR 1547/19, 1 BvR 2634/20) that Hesse's § 25a HSOG and Hamburg's § 49 HmbPolDVG on automated data analysis violate informational self-determination for lack of a sufficient intervention threshold (at least a concretised danger for the covered purposes); Hamburg's rule was void, Hesse's applied only transitionally with strict conditions, excluding data from covert surveillance measures [7]. Lesson for design: purpose binding, restricted data sources, documented justification and logging per analysis.
• Other risks: wrongful identification and reputational harm, discrimination via proxy variables, doxxing/misuse by insiders, scraping terms-of-service and database rights, evidence admissibility (chain of custody).
• Mitigations: legal basis and purpose recorded per case; role-based, case-compartmented access; immutable audit log; reversible entity merges with confidence shown; mandatory source citation; human decision on every consequential step; DPIA and, for authorities, data-protection-officer review; red-team the pipeline for prompt injection from scraped content.

Getting started

1. Define mandate, legal basis and prohibited uses first (what data, which persons, which purpose); involve the DPO or legal counsel before any tooling.
2. Build a gold set of 200–500 known matching/non-matching record pairs from your own domain and test Splink [2] against a commercial ER engine.
3. Stand up a small graph over open data (OpenSanctions + a company register + Offshore Leaks [3][4]) and let analysts try three real past cases.
4. Add LLM extraction and cited briefs; require every sentence to link to a source record and measure unsupported claims.
5. Add audit logging, compartmentalised access and a review workflow before scaling to real case data.

Sources

1. Maltego — product overview (Graph, Search, Monitor, Evidence, Data; ISO 27001:2022) — https://www.maltego.com/
2. Splink (UK Ministry of Justice), probabilistic record linkage, MIT licence — https://github.com/moj-analytical-services/splink
3. OpenSanctions documentation (FollowTheMoney data, bulk data, yente on-premise matching) — https://www.opensanctions.org/docs/
4. ICIJ Offshore Leaks Database — https://offshoreleaks.icij.org/pages/about ; OCCRP Aleph documentation — https://docs.aleph.occrp.org/
5. Bellingcat resources — https://www.bellingcat.com/resources/
6. Bellingcat, articles on LLMs and geolocation (listed on the resources page above) — https://www.bellingcat.com/resources/
7. Bundesverfassungsgericht, Press release 18/2023 (Hesse/Hamburg automated data analysis) — https://www.bundesverfassungsgericht.de/SharedDocs/Pressemitteilungen/DE/2023/bvg23-018.html ; LTO coverage — https://www.lto.de/recht/nachrichten/n/bverfg-1BvR154719-1BvR263420-einsatz-software-hessendata-durch-polizei-verfassungswidrig
8. Microsoft GraphRAG documentation — https://microsoft.github.io/graphrag/
9. AI Act Article 2 (scope, national security/defence exclusion) — https://artificialintelligenceact.eu/article/2/
10. Fivecast ONYX — https://www.fivecast.com/
11. Quantexa — https://www.quantexa.com/
12. AI Act Annex III (points 6, 7, 8) — https://artificialintelligenceact.eu/annex/3/
13. European Commission, AI regulatory framework (application dates, AI Omnibus) — https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai
14. AI Act Article 5 (prohibited practices) — https://artificialintelligenceact.eu/article/5/