RAG assistant over company documents ("chat with your data")

Company knowledge sits in file shares, SharePoint, wikis and PDFs that nobody can search effectively. A retrieval-augmented generation (RAG) assistant indexes these documents, retrieves the passages relevant to a question, and lets an LLM write an answer that cites its sources. Done well it replaces long searches and "who knows this?" emails; done badly it produces confident wrong answers or leaks documents across permission boundaries.

Problem & value

• Today: employees search by file name or keyword, ask colleagues, or re-do work that exists already. Classic intranet search misses content when the query wording differs from the document wording.
• AI value: natural-language questions answered from the company's own, current documents, with citations that make answers verifiable. Knowledge stays in the documents (updatable without retraining), which is the core argument for RAG over fine-tuning facts into a model [1].
• Measured effects: retrieval quality is the main lever. Anthropic reported on its own benchmark that adding LLM-generated context to chunks cut top-20 retrieval failures by 35% (embeddings alone), 49% (with BM25) and 67% (with reranking), from a 5.7% baseline [2]. This is a vendor benchmark, not a guarantee for your corpus; measure on your own questions.
• Relation to neighbours: KNOW-001 is finding documents (result lists); this case is generating answers. Domain-specific variants (HR, service manuals, standards, clinical) are KNOW-003/004/009/010.

Variants

• Team/SMB assistant: one shared drive or SharePoint site, 10–500 users, hosted product, no custom code.
• Enterprise knowledge platform: many sources, permission-aware retrieval (ACL trimming), multi-tenant, audit logging.
• Developer/engineering docs: code, runbooks, tickets, API docs (see KNOW-004 for field technicians).
• Public sector: internal regulations, circulars and procedures; typically on-prem or sovereign cloud.
• Private use: local "chat with my PDFs/notes" with Ollama + Open WebUI; see KNOW-018 (personal knowledge).
• Agentic RAG: the LLM plans sub-queries, searches several sources and iterates (e.g. Azure AI Search agentic retrieval) [3].

How it works

1. sources (SharePoint, file share, wiki, DMS, e-mail)
2. connectors + incremental sync (incl. ACLs / group membership)
3. parsing (layout-aware: tables, headings, OCR for scans)
4. chunking (+ optional context prefix per chunk)
5. embeddings
6. index: vector + keyword (BM25) + metadata/ACL fields query: question
7. (rewrite / decompose)
8. hybrid retrieval with ACL filter
9. reranker
10. top-k passages
11. LLM answer with citations
12. guardrails (grounding check, PII, injection filters)
13. user feedback + logs

• Hybrid retrieval (BM25 + dense vectors, fused, then reranked) beats vectors alone, especially for part numbers, names and abbreviations. BGE-M3 provides dense, sparse and multi-vector output in one model [4].
• Permission trimming is essential: the filter must be applied at query time in the retrieval layer, not by asking the LLM to hide things.
• Human-in-the-loop: thumbs up/down with reasons, "report wrong answer" routed to the document owner, a curated list of golden questions that is re-run after every change.

Data preparation

The quality of a RAG assistant is the quality of its documents. Typical sources are policies, handbooks, quotations, project files, wiki pages and mail on file shares, SharePoint or the DMS. Typical gaps are scans without text, several contradicting versions of the same document, outdated content, knowledge that exists only in people's heads or mail, and documents that not every employee may see.

Preparation steps: - Choose a bounded, authoritative corpus for the start, not "everything" (PREP-013). - Mark one valid version per document, remove or archive duplicates and old versions (PREP-009, PREP-010). - Run OCR on scans and convert legacy formats (PREP-006, PREP-007, DOC-014). - Rewrite unclear key documents so they can be retrieved in chunks (PREP-014) and list company terms and abbreviations (PREP-015). - Check permissions and exclude confidential material (PREP-023, PREP-024). - Collect 30 to 50 real questions with correct answers as test set (PREP-018). - Plan a review cycle for the content (PREP-017).

Expect 40 to 60 percent of the effort in this preparation; the technology is the smaller part. See GUIDE-003.

Implementation paths

Which path for whom

• SMB: Path A, because ready-made knowledge-assistant products with EU hosting exist at SMB prices and the use case (questions over own documents) is standard.
• Enterprise: Path B, because assistants over core knowledge need own permission handling, evaluation and integration into existing portals, and volume makes seat licences expensive.
• Developer: Path C, because open-source RAG stacks (Open WebUI, LlamaIndex or Haystack with Ollama or vLLM) are well proven.
• Public sector: Path C on sovereign EU or German infrastructure, because internal documents are sensitive and procurement rules favour controlled hosting.

Path A — Buy (recommended for most SMBs)

• Suite assistants: Microsoft 365 Copilot (grounded in SharePoint/OneDrive via Microsoft Graph, honours existing permissions; EU Data Boundary), Google Gemini with Workspace, Atlassian Rovo, Notion AI, Glean (enterprise search + assistant).
• EU/German vendors: Langdock (Berlin; model-agnostic chat, agents, integrations; EU hosting, managed/own-cloud/on-prem options) [5], Mistral Le Chat Enterprise (France), Aleph Alpha PhariaAI (Germany, on-prem/sovereign focus), IONOS and STACKIT AI model hubs, Dust (France), Nuclia / Progress Agentic RAG (France-origin RAG-as-a-service), DocuWare and d.velop with AI add-ons for DMS content.
• Choose A when: the content already lives in one suite, users < a few thousand, and no special data-sovereignty constraint. Check: ACL handling, EU data residency, citation quality, admin audit logs, subprocessor list, and whether prompts are used for training.

Path B — Build on cloud APIs

• Managed retrieval: Azure AI Search (hybrid + semantic ranking, agentic retrieval with knowledge bases and sources, Entra ID permission inheritance, private endpoints) with Azure OpenAI/Foundry models in EU regions [3]; AWS Bedrock Knowledge Bases (Frankfurt/Paris etc.); Google Vertex AI Search/RAG Engine (EU locations).
• Models: Claude, GPT, Gemini (via EU-region endpoints) or Mistral (EU provider) for generation; Cohere Rerank, Voyage or Mistral embeddings, or Azure/Bedrock-hosted equivalents. Verify the current model list and EU region availability in each vendor's console before committing.
• Frameworks: LlamaIndex, LangChain/LangGraph, Haystack (deepset, Berlin); low-code: n8n, Dify (open-source, self-hostable) [6].
• Vector stores: pgvector (reuse PostgreSQL), Qdrant (Berlin-founded), Weaviate, Elasticsearch/OpenSearch hybrid.
• Choose B when: you need custom sources, UI integration into an existing app, or specific access logic.

Path C — Self-host (open source)

• Parsing: Docling (MIT, layout/table/OCR, runs air-gapped) [7], plus PaddleOCR or MinerU for scans.
• Embeddings/rerankers: BGE-M3 (MIT, 100+ languages, 8k tokens) [4], Qwen3-Embedding 0.6B/4B/8B and Qwen3-Reranker (Apache-2.0) [8]. German-language checks on your own corpus are mandatory.
• LLMs: open-weight instruction models served with vLLM, Ollama or llama.cpp: e.g. Mistral Small / Mistral Medium-class open models, Qwen3 family, Llama, gpt-oss, Gemma. Choose by German quality and license; verify current releases.
• Stack: Open WebUI (RAG with BM25 + vector hybrid, reranking, Docling/Tika extraction; note its branding-preserving license) [9], or Dify, RAGFlow, AnythingLLM, LibreChat; store in Qdrant/pgvector/OpenSearch.
• Hardware: HW2 (one 24–32 GB GPU) serves a ~14–30B quantised model for tens of concurrent users; embeddings and reranker can share it. HW3 for larger models, many users or long contexts. CPU-only works for a pilot with small models and a hosted-free embedding model.
• Ops effort: patching, re-indexing on embedding-model change, backup of index, monitoring of connectors.

Path D — Custom training

Rarely required for the knowledge itself. Worth considering: fine-tuning an embedding model or reranker on domain query/passage pairs (a few thousand pairs, often generated synthetically from your documents and filtered by humans) when jargon-heavy corpora score poorly; fine-tuning a small LLM for tone, citation format or strict refusal behaviour. Do not fine-tune to "teach" facts that change.

State of the art (2026)

• Standard recipe: layout-aware parsing → contextualised chunks → hybrid retrieval → reranker → cited answer. Reranking and context-enriched chunks give the largest measured gains [2].
• Agentic retrieval: LLM-based query planning, parallel sub-queries across several knowledge sources and structured citations are now productised (Azure AI Search / Foundry IQ marks agentic retrieval as the recommended approach for new builds, with some parts still in preview) [3].
• Long context vs RAG: million-token windows do not replace retrieval for large corpora (cost, latency, ACLs, freshness), but allow sending whole documents once the right ones are found.
• Evaluation: Ragas-style metrics (faithfulness, context precision/recall, answer relevance) and golden question sets are standard [10]; LLM judges need spot checks by humans.
• Limits and failure modes: wrong chunk boundaries and broken tables; questions needing aggregation across many documents (counting, comparing all contracts) fail with top-k retrieval (use SQL/structured extraction, or KNOW-013 GraphRAG); outdated or contradictory document versions; hallucinated citations; multilingual mismatch; prompt injection through indexed documents and e-mails (OWASP lists prompt injection and vector/embedding weaknesses among its LLM Top 10) [11].

Shared resources & combinations

• Shared stack: the retrieval backbone (embedding model, reranker, vector/hybrid index, parser, LLM server) is the same as for KNOW-001 enterprise search, KNOW-003 HR assistant, KNOW-004 technical documentation, KNOW-009 standards Q&A and KNOW-010 clinical guidelines. Build it once as an internal platform and run domain assistants as separate collections with their own ACLs.
• Ingestion: DOC-014 (PDF-to-structured parsing) and KNOW-011 (knowledge base / FAQ maintenance); KNOW-025 (records classification, ROT clean-up) improves corpus quality before indexing.
• Extensions: KNOW-013 GraphRAG for relationship-heavy questions; AGENT-002 workflow automation and agents that act on answers (tickets, drafts).
• Hardware: one HW2/HW3 server for LLM + embeddings + reranker also carries DOC-001-style extraction and LANG summarisation workloads.

Effort, cost & KPIs

• Path A: days to 2 weeks (connect sources, set permissions, pilot group). Per-seat licences typically dominate cost.
• Path B: 1–3 months to production incl. connectors, ACL handling and evaluation set; cost drivers are tokens per query (retrieved context), reranker calls, index storage and engineering.
• Path C: 2–4 months; GPU server plus operations time; no per-token cost.
• KPIs: answer-correctness and faithfulness on the golden set, citation accuracy, "no answer" rate for out-of-scope questions, share of questions answered without escalation, weekly active users, user-rated helpfulness, time saved per query (sampled), permission-leak incidents (target zero), cost per query.

People & change

Management sets scope and budget; the department lead names the pilot corpus and a content owner. Subject-matter experts write the test questions and review answers, so involve them from the start. IT handles accounts, hosting and connectors; the IT security officer checks prompt-injection and leakage risks (GUIDE-013). The data protection officer (DPO) is involved when documents contain personal data; a DPIA is advisable if mail or HR files are included, and an AVV is needed with every hosting provider (GUIDE-005).

Works council: depends. If a works council exists, an assistant that logs user questions is suited to monitor behaviour, so §87(1) no. 6 BetrVG co-determination is likely, and §90(1) no. 3 requires early information. Agree on no individual evaluation of questions (GUIDE-004).

Pilot with one team of 10 to 20 people, collect feedback weekly and name AI champions. Typical concerns: "will it replace my expertise" (answer: it finds documents, people decide) and "can I trust it" (answer: every answer cites sources, there is a feedback button). Humans stay in control: the assistant never decides, and wrong answers are reported to the content owner. Support AI literacy with a short training on prompting and verification (GUIDE-006, GUIDE-008, GUIDE-009).

Risks, regulation & compliance

• EU AI Act: typically limited risk: an internal chatbot must make clear that users interact with AI (Art. 50(1), applicable from 2 Aug 2026 per Art. 113 unless obvious from context) [12]; AI literacy duty (Art. 4) for staff. It becomes high-risk if used for Annex III purposes (e.g. as a decision aid in recruitment, worker evaluation or access to essential services); keep the scope documented. Re-check timelines in case of EU "digital omnibus" amendments.
• GDPR: indexed documents often contain personal data (HR files, contracts, e-mails). Needs a legal basis, data-processing agreement with providers, EU region/transfer assessment, retention and deletion that propagates to the index and logs, and a DPIA for broad mailbox or HR indexing. Works-council co-determination (§87 BetrVG) applies in Germany to tools that can monitor employees, including usage logs.
• Security: enforce source permissions at retrieval; separate collections for confidential areas (HR, board, legal); treat all retrieved text as untrusted (prompt injection, exfiltration through links/images); no write actions without confirmation [11].
• Quality/liability: display citations, show document date/version, state "not found" explicitly; do not use unverified answers for legal, medical or safety decisions.
• Mitigations: red-team the assistant with adversarial documents, log retrievals, review top failed questions weekly, owner per collection.

Getting started

1. Pick one narrow, valuable corpus (e.g. IT/process documentation, ~1,000 documents) and a pilot group of 10–30 users; clean outdated versions first.
2. Write 50–100 real questions with reference answers and source passages (the golden set).
3. Run the golden set against 2–3 options (Path A product, Path B managed search, Path C open stack) and compare faithfulness, citations and German-language quality.
4. Verify permissions: test with users who must not see certain documents; involve data protection officer and works council early.
5. Launch with feedback buttons and owners per collection; re-run the golden set after every model, chunking or corpus change.

Sources

1. Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (NeurIPS 2020) — https://arxiv.org/abs/2005.11401
2. Anthropic — Introducing Contextual Retrieval — https://www.anthropic.com/news/contextual-retrieval
3. Microsoft Learn — RAG and generative AI in Azure AI Search (agentic retrieval vs classic RAG) — https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview
4. BAAI — BGE-M3 model card — https://huggingface.co/BAAI/bge-m3
5. Langdock — product and hosting overview — https://www.langdock.com/
6. Dify — open-source LLM app platform — https://github.com/langgenius/dify
7. Docling project — https://github.com/docling-project/docling
8. Qwen — Qwen3-Embedding-8B model card — https://huggingface.co/Qwen/Qwen3-Embedding-8B
9. Open WebUI — https://github.com/open-webui/open-webui
10. Ragas — LLM application evaluation toolkit — https://github.com/explodinggradients/ragas
11. OWASP — Top 10 for LLM applications — https://genai.owasp.org/llm-top-10/
12. EU AI Act Article 50 (transparency obligations) — https://artificialintelligenceact.eu/article/50/