Qualitative research coding (interviews, open survey answers)

Qualitative analysis is slow. A coder reads every transcript line by line, applies codes from a codebook, refines categories and then builds themes. A one-hour interview takes hours to code, and a survey with 10,000 open answers often gets sampled or skipped. LLMs can now apply a written codebook to every segment, propose inductive codes and subcodes, and draft themes with cited quotations. On well-defined codes they reach agreement levels close to human coders. Researchers stay in charge of interpretation: they define codes, review the suggestions, measure agreement and write the analysis. The result is more material covered, faster turnaround and a documented audit trail.

Problem & value

• Who: academic researchers (social sciences, health, education), market and opinion research agencies coding open-ends, UX and product research teams, HR teams analysing employee-survey comments, public bodies evaluating programmes, and students writing theses.
• Today: manual coding in QDA software (MAXQDA, ATLAS.ti, NVivo) or spreadsheets. Codeframes for survey open-ends are built by hand on a sample, then applied by trained coders. Inter-coder checks need a second person, so they are often skipped.
• Why AI helps: LLMs follow natural-language code definitions, so you need no training set. They can apply the same codebook to every answer in any language and explain each assignment.
• Evidence: in a study on paragraph-length socio-historical texts, GPT-4 with chain-of-thought prompting reached Cohen's κ ≥ 0.79 on 3 of 9 codes and κ ≥ 0.6 on 8 of 9 codes against a human gold standard. GPT-3.5 managed a mean κ of only 0.34 [1]. Earlier work found "fair to substantial" agreement for GPT-3 combined with expert codebooks [2]. RTI researchers showed LLMs can often match human agreement on deductive coding, and that agreement testing can reveal codes where the model is effectively guessing [3]. Vendors claim large speed-ups, for example Caplena's "95% faster at human-level accuracy" for open-ends [8]. These are marketing claims without published methodology.

Variants

• Deductive coding (codebook-driven): apply a fixed codeframe to segments. This is the most reliable mode and the standard for survey open-ends and tracking studies.
• Inductive / open coding: the model proposes initial codes and groups them into a hierarchy. This is useful as a "second reader" or starting point. It only partly reproduces the themes of human thematic analysis [4].
• Market research open-ends: high-volume coding of NPS, brand, ad-test and tracker verbatims against stable codeframes across waves and languages. Customer-feedback analytics with sentiment and aspect-based sentiment analysis lives in LANG-013.
• UX / product discovery: tagging interview highlights in a research repository and clustering them into insights. Running the interviews with an AI moderator is CONV-020.
• Employee surveys & HR: coding engagement-survey comments. This must stay at aggregate level, see Risks.
• Health & social-care research: patient interviews, often with special-category data, which needs on-prem or local AI.
• Students / private researchers: AI assistants in student licences of QDA tools.
• Boundaries: statutory consultations with a duty to consider every submission → LANG-026; unsupervised topic discovery in large corpora → LANG-016; simulated respondents → LANG-027; literature reviews → KNOW-008.

How it works

1. recordings
2. ASR + speaker diarization
3. transcript cleaning
4. pseudonymisation (names, places, employers) [human check]
5. segmentation (paragraph / speaker turn / answer)
6. codebook prompt: definitions, include/exclude rules, examples, research question
7. LLM coding per segment (multi-label, rationale, confidence, quoted span) ↳ optional: embeddings + clustering to suggest new codes / group a flat code list
8. human review of suggestions (accept / reject / recode) [core HITL]
9. agreement check on a double-coded sample (κ, Krippendorff's α, per code)
10. codebook refinement loop
11. full run
12. theme building, cross-tabs, memo/report drafting with quotation references

• Codebook as prompt: precise definitions, inclusion/exclusion criteria and 2–3 anchor examples per code matter more than model choice. Asking the model to justify each decision improves fidelity [1].
• Constrained output: JSON schema per segment (codes[], evidence_span, rationale, confidence) so that every code links back to the exact quotation.
• Inductive support: embeddings (e.g. multilingual-e5, BGE-M3) plus clustering (BERTopic-style) with LLM labels [9], or LLM batch theme generation followed by condensation and mapping, as in the open-source ThemeFinder pipeline [7].
• Human-in-the-loop: researchers own the codebook, review every suggestion (or a stratified sample for large open-end volumes), resolve disagreements and write the interpretation. Reflexive approaches treat AI output only as a prompt for reflection, not as analysis.

Data preparation

The core data are interview recordings, transcripts, focus-group notes and open survey answers. In an SMB they typically live as audio files on phones or in Teams recordings, as Word notes on file shares, and as Excel exports from survey tools.

Typical gaps: no transcripts yet, poor audio and dialects, speaker labels missing, answers in several languages, notes without consent records, direct identifiers (names, employers) in the text, and no agreed codebook.

Preparation steps: - Transcribe with Whisper or faster-whisper plus diarisation (SPEECH-001) and correct the text by hand. - Pseudonymise names, places and companies before any cloud processing (PREP-022); store consent forms separately. - Consolidate survey exports into tidy tables, one answer per row with respondent ID and metadata (PREP-011). - Define the codebook with definitions and examples (PREP-019) and have two humans code a gold sample for validation (PREP-018, DATA-023). - Convert legacy notes and scans to text (PREP-006, PREP-007).

Effort share: medium to high, about 30–40 percent, mostly transcription cleanup and codebook work. See GUIDE-003 and GUIDE-005.

Implementation paths

Which path for whom

• SMB — A: QDA tools with AI such as MAXQDA and ATLAS.ti (both German vendors) are mature, auditable and affordable, with EU data storage and a DPA; Local AI is available if needed.
• Enterprise — B: recurring trackers, custom codeframes and integration into survey platforms justify a cloud API pipeline with EU-region models.
• Developer — C: open-source tools (QualCoder, ThemeFinder) with local models give full control over prompts and data.
• Public — C: research institutes and authorities often face consent and ethics-board limits, so processing on own or sovereign infrastructure is safest.
• Private — C: local apps such as Ollama or LM Studio keep interview material on your own computer.

Path A — Buy (recommended)

• QDA software with built-in AI: MAXQDA AI Assist (VERBI, Berlin 🇩🇪). It offers AI coding with explanations, subcode and new-code suggestions, chat with documents and codes, summaries, AI reports, and "Local AI" in the Premium tier that runs an LLM on your own computer or server; data is stored encrypted in the EU [5]. ATLAS.ti (Berlin 🇩🇪) offers AI Coding and Intentional AI Coding, both using OpenAI models with a no-training agreement and explicit upload consent. An organisation-wide "AI Privacy Mode" switches the OpenAI integration off [6][11]. NVivo (Lumivero) also has an AI assistant.
• Survey open-end coding: Caplena (Zurich 🇨🇭) offers codebooks, cross-tabs with quant variables and 100+ languages [8]. Ascribe, Displayr and the text modules of Qualtrics/Medallia are alternatives.
• UX research repositories: Condens (EU or US hosting selectable, no training on customer data) [10], Dovetail, and similar tools with auto-tagging and clustering.
• Choose A when: standard interview/survey projects, researchers already use a QDA tool, and auditability matters more than custom logic. Check processing location, sub-processors, a DPA, and whether AI can be disabled per project.

Path B — Build on cloud APIs

• Models: Mistral (EU provider; Mistral Medium/Large), Claude or GPT via EU-region deployments (AWS Bedrock Frankfurt, Azure OpenAI EU Data Zone, Google Vertex AI EU). Use reasoning models for ambiguous, interpretive codes and smaller, cheaper models for high-volume deductive open-ends.
• Stack: Python pipeline with structured outputs; ThemeFinder (MIT, supports OpenAI, Azure, Gemini, Claude, Llama, Mistral) for theme generation and mapping [7]; batch APIs for cost; results exported back to QDA software (REFI-QDA project exchange) or survey tools (SPSS/Excel).
• Choose B when: you are an agency or research unit with recurring trackers, custom codeframes across many waves and languages, or integration into a survey platform.

Path C — Self-host (open source)

• Models: open-weight LLMs served with vLLM or Ollama, e.g. Qwen3 (incl. 30B-A3B MoE), Mistral Small 3.x, Gemma 3 or gpt-oss-20b/120b. Embeddings: multilingual-e5-large or BGE-M3. Transcription: Whisper large-v3 / faster-whisper with pyannote diarization.
• Tools: QualCoder (LGPL-3.0, open-source QDA, v4.0) with AI features that can use OpenAI or open models via the Helmholtz Blablador service and local embeddings [12]. Also ThemeFinder or BERTopic pointed at a local OpenAI-compatible endpoint.
• Hardware: HW2 (one 24–32 GB GPU) runs a quantised ~24–30B model and coding for a typical interview study. Use HW3 for 70–120B models when interpretive nuance or long transcripts matter.
• Choose C when: health data, vulnerable groups, ethics-board or consent restrictions forbid cloud processing. This is common in university and clinical research.

Path D — Custom training

Worth it only for high-volume, stable codeframes, such as a market-research agency coding millions of tracker open-ends a year or a statistical office coding occupations and "most important problem" answers. There, a fine-tuned encoder classifier (multilingual BERT/XLM-R class) or a small LoRA-tuned LLM is cheaper and more consistent than prompting. You need a few hundred to a few thousand human-coded answers per codeframe; LLM pre-labels plus human correction (DATA-023) cut the effort. Not sensible for one-off interview studies.

State of the art (2026)

• Deductive coding with frontier/reasoning LLMs reaches substantial-to-excellent agreement on many well-defined codes, but not on all of them. Performance varies strongly by code, so per-code agreement testing is mandatory [1][3]. Older/smaller models perform markedly worse [1].
• Inductive thematic analysis by LLMs gives plausible but shallow themes and only partly recovers human themes [4]. Reflexive, interpretive and phenomenological traditions largely reject delegating interpretation to AI.
• Product convergence: all major QDA tools now ship AI coding. Local/on-prem LLM options (MAXQDA Local AI) and privacy switches (ATLAS.ti AI Privacy Mode) answer GDPR concerns [5][6].
• Pipelines for scale: generate → condense → refine → map theme pipelines (ThemeFinder) are the open reference architecture for large open-text datasets [7].
• Failure modes: run-to-run variance (vendors themselves warn results differ between runs [6]), over-coding (assigning too many codes), drift toward generic or "positive" themes, missed irony, dialect and implicit meaning, fabricated or paraphrased "quotations", and blindness to silences and what was not said. Mitigate with fixed model versions, temperature 0, verbatim span checks and documented prompts.

Shared resources & combinations

• Upstream: SPEECH-001 (transcription + diarization) produces the transcripts. CONV-020 (AI-moderated interviews) produces interviews at scale that need this coding step. DOC-018 (anonymisation) handles pseudonymisation before analysis.
• Same text-analytics stack (llm-general + embedding-text + vector-db): LANG-013 feedback & sentiment, LANG-016 topic modelling, LANG-026 consultation analysis, LANG-015 entity extraction and LANG-008 summarisation. One self-hosted LLM server (HW2/HW3) plus one embedding service can serve all of them.
• Validation partner: LANG-027 synthetic respondents must be validated against real coded human data from this use case. KNOW-008 and KNOW-015 add literature and desk research to the findings.
• Labelling loop: DATA-023 model-in-the-loop annotation tools double as coding-review UIs, and the coded data feeds Path D classifiers.

Effort, cost & KPIs

• Path A: days. Cost is the QDA licence plus the AI add-on (MAXQDA offers a free tier with limited prompts and a paid Premium tier [5]). The main effort is writing a good codebook.
• Path B: 2–6 weeks for a reusable pipeline with agreement reporting. API cost per answer is small compared to coder time, and batch processing lowers it further.
• Path C: 3–8 weeks plus GPU server and operations. This is attractive where several research groups share one server.
• KPIs: per-code κ / Krippendorff's α versus human coders (e.g. a target of ≥ 0.6–0.8 per code before automated use), share of suggestions accepted unchanged, coding hours per transcript or per 1,000 answers, turnaround from fieldwork end to topline, share of verbatim-verified quotations, and coverage (share of material actually coded).

People & change

Involve early: the research or project lead owns the codebook and interpretation; management sets scope and budget. End users (researchers, HR, UX) pilot on one finished study and compare AI coding with human coding. IT checks hosting and accounts. Legal and the DPO come in before the first upload.

Works council: depends. For customer or market research, usually no. For employee interviews or engagement surveys it matters: coded statements about colleagues or teams can be used to evaluate behaviour, so co-determination under §87(1) no. 6 BetrVG is likely and anonymity must be agreed in a works agreement. Inform the works council under §90(1) no. 3 BetrVG and involve HR (GUIDE-004).

Data protection: interviews are personal data and may contain special categories (health, beliefs). Involve the DPO, sign an AVV (Art. 28 GDPR), document consent and the legal basis, and expect a DPIA (Art. 35 GDPR) for sensitive topics (GUIDE-005).

AI literacy: support researchers with training on prompt design, validation and bias (GUIDE-006).

Concerns: "AI flattens nuance." Keep humans as interpreters: AI proposes codes with quotes, researchers decide and document. Report AI use transparently in the methods section (GUIDE-009).

Risks, regulation & compliance

• EU AI Act: typically minimal risk: research analysis tooling is not an Annex III category, and the Art. 4 AI-literacy duty applies. AI systems developed and put into service solely for scientific research are excluded from the Act's scope (Art. 2(6)). Coding becomes high-risk if the output is used to evaluate individual employees (Annex III No. 4) or learners (No. 3), e.g. coding an individual's 360° comments to make decisions (see LANG-036). Keep employee-survey coding aggregated and anonymous. Inferring emotions of employees from their texts is not Art. 5(1)(f) emotion recognition, which covers biometric data. Even so, avoid individual-level "mood" scoring at work.
• GDPR: transcripts and open answers contain personal and often special-category data (health, political views, religion). You need a legal basis (consent per the participant information sheet, or Art. 6(1)(e)/(f)), Art. 89 safeguards for research (pseudonymisation, data minimisation) and a DPA with every AI vendor. Check third-country transfers, since US model providers sit behind several QDA tools [6], and run a DPIA for sensitive studies. Consent forms and ethics approvals written before 2023 rarely cover cloud AI processing, so check them before uploading.
• Research integrity: report AI use in methods sections, archive prompts, model versions and agreement statistics, and never present AI themes as findings without human analysis. Some journals and funders restrict uploading unpublished data to AI services.
• Bias: LLMs can under-recognise minority dialects or non-native phrasing and impose dominant framings. Stratify agreement checks by group.

Getting started

1. Write or tighten the codebook: definition, include/exclude rules and 2–3 anchor quotes per code.
2. Check consent forms, ethics approval and data classification, and pick the processing mode (cloud EU, vendor AI with DPA, or local AI) accordingly. Pseudonymise transcripts first.
3. Pilot in your QDA tool (MAXQDA or ATLAS.ti AI coding) on 2–3 transcripts or 300 open answers that you also code manually. Compute κ per code.
4. Drop or redefine codes below your agreement threshold, then fix the model version and prompts.
5. Scale up with human review of all, or a stratified sample of, AI suggestions. Document the procedure in your methods section or project report.

Sources

1. Dunivin, Z. O. — Scalable Qualitative Coding with LLMs: Chain-of-Thought Reasoning Matches Human Performance in Some Hermeneutic Tasks (arXiv:2401.15170) — https://arxiv.org/abs/2401.15170
2. Xiao, Z. et al. — Supporting Qualitative Analysis with LLMs: Combining Codebook with GPT-3 for Deductive Coding (IUI '23 Companion) — https://arxiv.org/abs/2304.10548
3. Chew, R. et al. (RTI) — LLM-Assisted Content Analysis: Using LLMs to Support Deductive Coding — https://arxiv.org/abs/2306.14924
4. De Paoli, S. — Can LLMs emulate an inductive Thematic Analysis of semi-structured interviews? — https://arxiv.org/abs/2305.13014
5. MAXQDA — AI Assist product page (features, Local AI, EU storage, free/premium tiers) — https://www.maxqda.com/products/ai-assist
6. ATLAS.ti — AI Coding powered by OpenAI (consent, no-training agreement, run-to-run variance) — https://atlasti.com/ai-coding-powered-by-openai
7. UK i.AI — ThemeFinder (open-source LLM theme pipeline, MIT) — https://github.com/i-dot-ai/themefinder
8. Caplena — AI open-ended feedback coding (vendor claims) — https://caplena.com
9. BERTopic documentation — https://maartengr.github.io/BERTopic/index.html
10. Condens — AI features and EU/US hosting — https://condens.io/ai/
11. ATLAS.ti — Intentional AI Coding & AI Privacy Mode — https://atlasti.com/intentional-ai-coding
12. QualCoder — open-source QDA software with AI features — https://github.com/ccbogel/QualCoder
13. Dai, S.-C., Xiong, A., Ku, L.-W. — LLM-in-the-loop: Leveraging LLMs for Thematic Analysis (EMNLP 2023 Findings) — https://arxiv.org/abs/2310.15100
14. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
15. Regulation (EU) 2016/679 (GDPR), incl. Art. 89 — https://eur-lex.europa.eu/eli/reg/2016/679/oj