Topic modelling & trend detection in text corpora

Organisations sit on millions of free-text records: tickets, feedback comments, service reports, chatbot logs, news and papers. Nobody can read all of it, so new problems and opportunities are noticed late. Modern topic modelling combines sentence embeddings, clustering and LLM labelling. It turns an unlabelled corpus into a readable map of themes, keeps that map current as new text arrives and flags topics that are new or growing fast. Analysts stop reading samples and spend their time on the few themes that changed.

Problem & value

• Today: analysts read random samples, use keyword searches or Excel tagging, or run classic LDA, which returns word lists ("women, power, female") that still need interpreting. Predefined category schemes miss problems that nobody anticipated, and that is exactly what trend detection is for.
• AI value: themes come straight from the data, no labelled training set is needed, and labels and descriptions read like an analyst's own. Corpora in many languages map into one shared embedding space, and incremental updates show changes (new topics, growth, decline) rather than a static snapshot.
• Evidence: in Anthropic's Clio study, an LLM pipeline with privacy safeguards clustered 1 million conversations into a hierarchy. It surfaced the largest usage themes (e.g. software development at over 10%) along with thousands of small clusters, and agreed with existing trust-and-safety classifiers at r = 0.71 [3]. TopicGPT (NAACL 2024) reached a harmonic-mean purity of 0.74 against human-annotated topics, versus 0.64 for the strongest baseline [4]. Feedback-analytics vendor Caplena reports that IKEA analysed more than 3 million comments in 40 languages on its platform (vendor claim) [7].

Variants

• Voice of the customer / CX: emerging complaint themes in tickets, reviews and NPS verbatims, together with sentiment (LANG-013). The topic model finds what people talk about; the sentiment model says how they feel about it.
• Product quality & field service: clusters in warranty claims, service reports and complaint logs act as early warning of failure modes (manufacturing, automotive, medical-device post-market surveillance).
• LLM/chatbot log analytics ("Clio-style"): what users actually ask a chatbot or copilot, where it fails, and which intents are new. Feeds bot improvement (CONV family) and AI governance monitoring.
• Research & technology scouting: emerging topics in papers and patents (KNOW-008, KNOW-007), technology radars and horizon scanning (regulatory variant: LANG-020).
• Media & market: news and web topic trends. Brand-specific listening is LANG-018 and competitor tracking is LANG-017; both reuse this engine.
• Internal / HR: themes in employee surveys and ideas portals. Use aggregate reporting only; see risks.
• Public sector: citizen enquiries, petitions and complaint mailboxes. Formal consultation analysis with a duty to consider each submission is LANG-026. Code-frame qualitative research is LANG-019.
• Private / developer: exploring a personal note archive, a newsletter backlog or a dataset with BERTopic in a notebook.

How it works

1. ingest (DB, ticket system, CSV, crawler) + timestamps + metadata
2. clean, deduplicate, PII-redact, optional LLM "facet" extraction (summarise each doc to its core issue)
3. sentence embeddings (multilingual)
4. dimensionality reduction (UMAP) + clustering (HDBSCAN / k-means), or LLM-driven topic generation
5. topic representation: c-TF-IDF keywords + LLM label & description + representative docs
6. hierarchy (merge clusters into themes) + analyst review / rename / merge / seed
7. topic-over-time counts per segment
8. change-point / burst detection
9. alerts
10. dashboard (BI), drill-down to documents, export of assignments

• Embedding-cluster-label (BERTopic pattern [1][2]) is the workhorse: cheap and scalable, and the LLM only sees a few representative documents per cluster.
• LLM-first approaches: TopicGPT [4] and LLooM concept induction [5] have the LLM propose and refine topics, or concepts with explicit inclusion criteria, which are then assigned to documents. They give better interpretability at higher token cost, so they are best for corpora up to roughly 10⁴–10⁵ documents or for sampled subsets.
• Facet extraction first (Clio [3], Kura [9]): an LLM first rewrites each noisy conversation into a short, de-identified summary, which is then embedded and clustered. Clusters become cleaner and the analysis layer holds less personal data.
• Trend detection: a global model with per-time-bin topic representations (BERTopic topics_over_time [2]), online/incremental models for streams (Turftopic online models [8], BERTopic partial_fit/merge_models), and statistical change-point or anomaly detection on topic frequencies (shared with ANOM-030).
• Human-in-the-loop: analysts approve topic names, merge or split clusters, set minimum cluster sizes and decide which alerts matter. Topic maps are an exploratory instrument, not ground truth.

Data preparation

Data needed: a large collection of texts with date and source: tickets, feedback, chat logs, reports, news, papers. In an SMB these sit in the ticket system, CRM, survey tools, mailboxes and file shares; older material may be on paper or in PDF.

Typical gaps: no date or source field; many duplicates, signatures and mail quotes that distort topics; scanned PDFs; mixed languages; personal data in free text; no agreed vocabulary for the resulting themes; unclear access rights.

Preparation steps: 1. Inventory and export sources with date, channel and ID (PREP-005, GUIDE-003). 2. Remove duplicates, signatures, quoted replies and boilerplate; OCR scans if needed (PREP-006, PREP-007, PREP-011). 3. Mask names and contact data (PREP-022). 4. Collect abbreviations and company terms so topics are labelled sensibly (PREP-015). 5. Check GDPR purpose when old tickets are reused (PREP-025). 6. Have two experts judge sample topics as a quality test (PREP-018).

Tools: Python with pandas, Excel or Power Query, Paperless-ngx for scans, n8n for regular exports.

Effort share: roughly 30-40 % of the project; cleaning text noise has more impact on topic quality than the algorithm.

Implementation paths

Which path for whom

• SMB — A: feedback-analytics products with automatic theme discovery exist at SMB prices; one-off analyses need no own build.
• Enterprise — B: topic trends feed own BI and strategy processes; embeddings and LLM labelling via API integrate with the data platform.
• Developer — C: open tools such as BERTopic with local embeddings run on HW2, cost little per run and give full control over parameters.
• Public sector — C: texts from citizens or staff are sensitive; open-source topic models on sovereign infrastructure avoid external transfer.

Path A — Buy (recommended for most SMBs)

• Feedback & CX text analytics: Caplena (Zurich, CH, EU-oriented; customers include Lufthansa, DHL and Beiersdorf) [7], Thematic, Qualtrics XM Discover, Medallia Text Analytics. These platforms include topic discovery, trend alerts and dashboards.
• Ticket/helpdesk built-ins: Zendesk Intelligent Triage and Intercom/Freshdesk insights surface emerging ticket themes inside the helpdesk.
• Trend & technology scouting: ITONICS (German GmbH; AI layer "Prism" builds technology radars from curated signals) [10], plus listening suites (Brandwatch, Talkwalker → LANG-018).
• Data maps: Nomic Atlas for interactive embedding maps with automatic topic labels.
• Choose A when: the corpus is feedback or tickets, a dashboard is the goal, and there is no in-house data science team. Check EU hosting, DPA, PII redaction and the export of topic assignments.

Path B — Build on cloud APIs

• Embeddings: Mistral mistral-embed (EU provider) [11], Cohere Embed, OpenAI text-embedding-3, Google Gemini embeddings, Azure OpenAI in EU Data Zones, or AWS Bedrock (Frankfurt) Titan/Cohere embeddings.
• Labelling LLM: a small, cheap model (Mistral Small, GPT-mini/Gemini Flash/Claude Haiku class) labels clusters and extracts facets; use a stronger model only for hierarchy building and executive summaries. Use batch APIs, since the work is offline.
• Orchestration: BERTopic with LiteLLM or LangChain representation models [6] calling the API. Store vectors in pgvector or OpenSearch, schedule with Airflow, dbt or n8n, and push results to Power BI or Superset.
• Choose B when: you need custom slicing, your own taxonomy or integration into a data platform, and the data may leave the premises under a DPA.

Path C — Self-host (open source)

• Libraries: BERTopic (MIT; zero-shot, hierarchical, online and dynamic topics, LLM representations via llama.cpp/LiteLLM) [6], Turftopic (MIT; KeyNMF, S³, FASTopic, dynamic and online models) [8], Top2Vec, Kura (MIT, Clio reimplementation for chat logs) [9], DataMapPlot for visualisation. Microsoft GraphRAG community summaries offer a graph-based alternative, though the project is now in maintenance mode [12].
• Embeddings: Qwen3-Embedding (0.6B/4B/8B, Apache 2.0, 100+ languages; 8B was #1 on MTEB multilingual in June 2025) [13], multilingual-e5, BGE-M3, Granite embeddings. For labels, a quantised 7–30B instruct model (Mistral Small, Qwen3, Llama, Gemma) served with vLLM or Ollama.
• Hardware: HW2 is enough. A single 24–32 GB GPU embeds millions of short texts per day and labels hundreds of clusters. Small corpora run on HW1 (CPU, Model2Vec static embeddings [6]).
• Ops effort: low to medium; scheduled batch jobs plus a model registry for topic models.

Path D — Custom training

Rarely needed. Fine-tune the embedding model contrastively, for example with SetFit or sentence-transformers, when domain jargon (part numbers, clinical or legal terms) collapses distinct issues into one cluster. A few hundred to a few thousand labelled pairs or analyst merge/split decisions are enough. Once the taxonomy has stabilised, the better step is usually to convert discovered topics into a supervised classifier (LANG-012) for consistent, auditable routing and counting.

State of the art (2026)

• The standard pattern is embeddings, clustering and LLM labels. Classic LDA/NMF survive mainly as cheap baselines. Better multilingual embeddings (Qwen3-Embedding, Jina v4, Gemini/Cohere) mean one model covers EU languages without translation.
• LLM-native concept induction (TopicGPT, LLooM) produces natural-language topics with definitions that analysts can edit and constrain, beating the strongest baseline on purity in published evaluations [4][5].
• Privacy-preserving log analytics at scale (Clio and its open reimplementations) has become the reference design for understanding chatbot and agent usage [3][9].
• Agentic analysis: deep-research agents (KNOW-015) increasingly use topic maps as a tool: cluster first, then drill into a theme and write the report.
• Limits and failure modes: results depend on random seeds and hyper-parameters such as minimum cluster size and UMAP settings. A large "outlier" topic is common. LLM labels can sound more coherent than the cluster really is, and LLM-first methods may invent topics or miss rare ones. Coherence metrics correlate poorly with human judgement. Trend signals are distorted by changes in volume, channel or product mix, so normalise counts and check for data-pipeline breaks before raising alerts.

Shared resources & combinations

• Same building blocks (embedding-text + vector-db + llm-small/general): LANG-012 classification, LANG-013 sentiment, LANG-019 qualitative coding, LANG-026 consultation analysis, KNOW-001/KNOW-002 search and RAG. One embedding service and one vector store can serve all of them, and a single HW2 GPU hosts an embedding model plus a small labelling LLM.
• Engine for: LANG-017 market intelligence, LANG-018 social listening, LANG-020 horizon scanning, KNOW-007/KNOW-008 patent and literature landscapes, LANG-040 incident-report analysis.
• Upstream: DATA-004 web data extraction, SPEECH transcription of calls, DOC-014 parsing for PDFs.
• Downstream: ANOM-030 (anomaly/root-cause detection on topic time series), DATA-002 narrative reporting, DATA-001 conversational BI over topic tables, KNOW-013 knowledge graphs. PRED-034 is the image-based counterpart for fashion trends.

Effort, cost & KPIs

• Path A: days to a few weeks; SaaS priced per seat, per volume of verbatims or both.
• Path B: 3–8 weeks for a pipeline with dashboards. Embedding cost is negligible; LLM labelling cost scales with clusters and facet extraction, not with raw corpus size, unless every document is summarised.
• Path C: 1–3 months including the GPU server and scheduling.
• KPIs: analyst hours per insight report, time from first occurrence to detection of a new issue, share of corpus covered by named topics (outlier rate), analyst acceptance rate of topic labels, topic stability between runs, number of actioned alerts, and precision of trend alerts.

People & change

Who and when: management formulates the question (which trends matter); the department lead (CX, product, strategy) interprets topics and decides actions; data owners release the text sources; IT provides exports and compute; the DPO is involved before old tickets or chats are reused.

Works council: depends. Topic analysis over customer texts is normally uncritical. If internal chats, e-mails or agent notes are analysed, the system is objectively suited to monitor behaviour and §87(1) no. 6 BetrVG applies (BAG). Inform under §90(1) no. 3 and agree that results stay aggregated, with minimum group sizes.

Data protection: AVV with cloud providers (Art. 28 GDPR); DPIA when employee data or large-scale evaluation of individuals is involved (GUIDE-005).

AI literacy and communication: support literacy under Art. 4 AI Act, especially that topics are statistical suggestions, not facts (GUIDE-006). Run a first pilot on one corpus, e.g. last year's tickets, and present results to a small analyst group (GUIDE-009).

Typical concerns: "topics look arbitrary" - let experts rename and merge topics; "surveillance of staff" - restrict analysis to aggregated, anonymised data and document it (GUIDE-008).

Risks, regulation & compliance

• EU AI Act: minimal risk for typical corpus analytics, with no Annex III category and the Art. 4 AI literacy duty for staff. It changes when outputs evaluate individuals. Monitoring or evaluating employees' behaviour, or using topic outputs for HR decisions, can fall under Annex III no. 4 (high-risk). Inferring emotions of employees or students from biometric data (voice, face) is prohibited under Art. 5(1)(f); text-only topic analysis is not emotion recognition (Art. 3(39)). Analysing citizens' texts to assess eligibility for benefits is Annex III no. 5. Keep analysis aggregate and about themes, not people.
• GDPR: tickets, chats and survey answers contain personal data and often special categories (health, union membership). You need a legal basis (legitimate interest with a balancing test, or compatible further processing under Art. 6(4)), data minimisation (redact PII before embedding or summarising), minimum cluster sizes so that no small group is identifiable (as in Clio [3]), retention limits, and a DPA with any API provider. Remember that embeddings of personal data are still personal data.
• Works council (DE): employee surveys, internal chat or email logs → co-determination under §87(1) no. 6 BetrVG; conclude a works agreement before analysis.
• Sector rules: medical-device complaint trending feeds post-market surveillance and trend reporting (MDR Art. 83–88). Keep human-validated, documented trend decisions there. In banking or insurance, complaint analytics supports conduct-risk reporting; document the methodology.
• Mitigations: keep model versions and seeds for reproducibility, have analysts sign off topic names, show the source documents behind every topic, and normalise trend alerts against total volume.

Getting started

1. Pick one corpus with a clear owner and timestamps, for example 12 months of support tickets, and write down three questions it should answer.
2. Run BERTopic or Turftopic with a multilingual embedding model on a PII-redacted export in a notebook, or trial a Path A tool such as Caplena on the same export.
3. Review the clusters with domain experts: rename, merge and split them, set the minimum topic size and turn the result into a seed taxonomy.
4. Schedule weekly incremental runs with topic-over-time charts and a simple volume-normalised growth alert.
5. Promote stable, recurring topics into a supervised classifier (LANG-012) and keep the topic model for discovering new themes.

Sources

1. Grootendorst, M. — BERTopic: Neural topic modeling with a class-based TF-IDF procedure (arXiv 2203.05794) — https://arxiv.org/abs/2203.05794
2. BERTopic documentation — Dynamic topic modeling (topics over time) — https://maartengr.github.io/BERTopic/getting_started/topicsovertime/topicsovertime.html
3. Anthropic — Clio: privacy-preserving insights into real-world AI use — https://www.anthropic.com/research/clio
4. Pham et al. — TopicGPT: A Prompt-based Topic Modeling Framework (NAACL 2024) — https://arxiv.org/abs/2311.01449
5. Lam et al. — Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM (CHI 2024) — https://arxiv.org/abs/2404.12259
6. BERTopic releases (v0.16–v0.17: LiteLLM/llama.cpp representations, Model2Vec, online fixes) — https://github.com/MaartenGr/BERTopic/releases
7. Caplena — AI feedback analytics (customer references) — https://caplena.com/
8. Turftopic — topic models with sentence-transformers — https://github.com/x-tabdeveloping/turftopic
9. Kura — open-source reproduction of Clio for chat-log analysis — https://github.com/567-labs/kura
10. ITONICS — innovation, technology and trend scouting platform — https://www.itonics-innovation.com/
11. Mistral AI — Embeddings documentation — https://docs.mistral.ai/capabilities/embeddings/
12. Microsoft GraphRAG (maintenance-mode notice) — https://github.com/microsoft/graphrag
13. Qwen3-Embedding-8B model card — https://huggingface.co/Qwen/Qwen3-Embedding-8B
14. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj