Enterprise semantic search across intranet, drives & tools

Company knowledge sits in file shares, SharePoint, wikis, ticket systems and chat, each with its own weak search. Enterprise search connects these sources to one index, matches by meaning as well as keywords, and shows each person only what they are already allowed to see. The result is a single search box (often with a cited AI answer on top) that cuts the time employees lose looking for information, and it is the retrieval foundation for every internal assistant.

Problem & value

• Today: employees search in five to fifteen tools, each with its own keyword search and its own permission model. Knowledge is re-created because the old document cannot be found; new staff ask colleagues instead; the IT helpdesk answers the same questions repeatedly.
• AI value: embeddings find documents by meaning ("home office rules" finds "Regelung zum mobilen Arbeiten"), rerankers push the best hits to the top, OCR makes scans searchable, and an LLM can summarise the top results with citations. Cross-lingual retrieval lets a German query find English documents.
• Benefit figures: vendors publish time-saving claims, but they are mostly self-reported and rarely independently measured. Treat them as hypotheses and measure your own baseline (search success rate, time to find) before and after a pilot. No generic figure is quoted here.
• Hard part: not the model but connectors, freshness and, above all, permission-correct results.

Variants

• SMB: search across Microsoft 365 or Google Workspace plus a file server; usually the built-in search of the suite is enough.
• Enterprise: cross-silo search over 20+ systems with identity-provider sync, federated and indexed connectors, analytics, and agent access (via MCP) to the same index.
• Public sector: intranet and document-management (DMS/E-Akte) search with strict data residency, often on-prem.
• Developer / IT: code, runbook and incident search (see KNOW-004 for technician manuals).
• Private: local semantic search over your own files and mail (see KNOW-018).
• Out of scope: conversational Q&A as the main product (KNOW-002), HR self-service (KNOW-003), product search for shops (KNOW-005), image/video archives (KNOW-016).

How it works

1. connectors (crawl / webhook / federated MCP) + ACL & group sync from IdP (Entra ID, LDAP, Okta)
2. text extraction + OCR + layout-aware chunking (DOC-014)
3. metadata enrichment (source, author, date, type, language, ACL)
4. hybrid index: BM25 keyword + dense (and optional sparse) vectors
5. query: rewrite / expand
6. hybrid retrieval with ACL filter
7. fusion (RRF)
8. cross-encoder rerank
9. results (snippets, facets) [+ optional LLM answer with citations]
10. feedback & analytics (clicks, zero-result queries)
11. tuning

• Hybrid retrieval is the production default: keyword search handles exact terms (part numbers, contract IDs, names), vectors handle paraphrase and cross-language matches, and reciprocal rank fusion merges both without score calibration [5].
• Permission trimming must happen inside the query (filter on ACL fields), not after the LLM has seen the text. Index the ACL with each chunk and sync group changes regularly [1][2].
• Federated connectors (live query via MCP, nothing indexed) suit sensitive or fast-changing sources; synced connectors give better relevance and speed [2].
• Human-in-the-loop: search administrators curate synonyms, promoted results and source scope; users give feedback. The system only reads, so approval steps matter only when agents act on results.

Data preparation

Search needs no new data, but it exposes every flaw in the existing data. Sources are typically the file server and SharePoint (Office, PDF, scans), the DMS, mail archives, wiki or intranet, the ticket system and the ERP attachments. Typical gaps are scanned PDFs without text, duplicate and outdated versions, folder structures nobody understands, missing metadata and, above all, grown access rights: if a folder is open to everyone, the search will show it to everyone.

Preparation steps: - Inventory the sources and name an owner for each (PREP-001, PREP-004). - Run OCR on scanned archives with tools such as Paperless-ngx, OCRmyPDF or Azure Document Intelligence (PREP-006, DOC-014). - Remove obvious duplicates and archive dead folders; add basic metadata such as type, date and owner (PREP-009, PREP-010). - Clean up permissions before indexing and test with a normal user account (PREP-023). - Build a small set of real test questions (PREP-018).

Expect roughly half of the project effort in data and permission clean-up, much of it done by the data owners, not by IT. A staged start with one department's sources keeps this manageable. See also GUIDE-003.

Implementation paths

Which path for whom

• SMB: Path A, because mature EU-hostable search products and Microsoft 365 or Atlassian search features exist at SMB prices and connectors for standard sources are ready-made.
• Enterprise: Path B, because many heterogeneous sources, own permission models and high query volume justify a tailored retrieval layer on cloud APIs.
• Developer: Path C, because open-source stacks (OpenSearch or Qdrant with open embedding models) give full control over connectors and ranking.
• Public sector: Path C on sovereign EU or German infrastructure, because document stores are sensitive and procurement rules favour controlled hosting.

Path A — Buy (recommended for most SMBs)

• Suite-native: Microsoft 365 Copilot Search and Copilot connectors (over 100 prebuilt connectors, synced or federated via MCP, source permissions respected) [2]; Google Workspace / Gemini Enterprise search; Atlassian Rovo for Jira/Confluence.
• Dedicated platforms: Glean, Coveo (CA), Elastic Enterprise Search components, Sinequa (FR, EU), IntraFind iFinder (DE, EU), Mistral Le Chat Enterprise with connectors (FR, EU), Onyx managed cloud. Check: connectors you actually use, ACL fidelity (including nested groups), EU hosting or on-prem option, answer citations, admin analytics, per-user pricing.
• Choose A when: most content already lives in one suite or a handful of mainstream SaaS tools.

Path B — Build on cloud APIs

• Managed retrieval: Azure AI Search (hybrid and semantic ranking, EU regions) with document-level access control: security filters (generally available) or native ACL, SharePoint-ACL and Purview label enforcement (preview as of the doc version consulted, so check status before production) [1]; Elastic Cloud or OpenSearch Service (EU regions) with hybrid queries [5][6].
• Models: embedding and rerank APIs from Cohere, Voyage, Mistral (EU) or OpenAI/Google; an LLM for the optional answer layer (Claude, Gemini, GPT, Mistral). Use EU regions or EU providers for GDPR.
• Orchestration: custom ingestion services or n8n; connector SDKs from the sources themselves (Microsoft Graph, Confluence, Google Drive APIs).
• Choose B when: you need custom sources, custom ranking or to embed search in your own product.

Path C — Self-host (open source)

• Platforms: Onyx (MIT community edition, 50+ connectors, air-gappable deployment) [4]; OpenSearch or Elasticsearch, Apache Solr, Vespa, Qdrant, Weaviate, pgvector for the index.
• Models: BGE-M3 (MIT, dense + sparse + multi-vector, 100+ languages, 8192 tokens) [3]; Qwen3-Embedding-8B and Qwen3-Reranker-8B (Apache 2.0, 100+ languages) [7]; multilingual-e5 as a lighter option. Extraction via Docling, Apache Tika, Unstructured.
• Hardware: HW2 suffices for embedding and reranking a few million chunks and a 7-30B answer model; a smaller index runs on CPU (HW1) with a small embedding model. Add HW3 only for a strong local answer LLM. Plan RAM for the index, not only the GPU.
• Ops effort: connectors and ACL sync are the permanent workload, not the model.

Path D — Custom training

Usually not needed. Fine-tune an embedding model or reranker only if domain jargon (chemistry nomenclature, internal codes, legal German) is retrieved poorly by general models. Needs a few thousand query-document pairs, which can be mined from click logs and from LLM-generated synthetic queries, validated on a held-out set. Always measure against a tuned hybrid baseline first.

State of the art (2026)

• Hybrid plus reranking beats pure vector search in practice; open multilingual embedders such as Qwen3-Embedding-8B (listed first on the multilingual MTEB leaderboard at its June 2025 release) and BGE-M3 are strong self-hostable baselines [3][7][8]. Leaderboard scores are general and do not replace testing on your own German corporate documents.
• Agentic retrieval: assistants split a question into sub-queries, search several sources and merge results; Azure AI Search ships this as agentic retrieval and a managed knowledge layer [1]. Search indexes are increasingly exposed to agents through MCP, including federated live connectors [2].
• Permission-aware indexes are now a built-in feature of major platforms, but sync lag (permissions reflected only after the next sync) remains a known gap [1].
• Limits and failure modes: stale or duplicated documents; ACL leaks through over-broad group mapping or sync delay; OCR errors on scans; tables and slides losing structure in chunking; AI answers that sound right but misread a policy version; prompt injection planted in indexed documents; poor results for short, ambiguous queries without click feedback.

Shared resources & combinations

• Same stack (embedding-text + reranker + vector-db + search-engine): KNOW-002 RAG assistants, KNOW-003 HR/policy assistant, KNOW-004 technical documentation, KNOW-009 standards Q&A, KNOW-013 GraphRAG. One shared hybrid index and ACL layer can serve all of them; build it once.
• Ingestion: DOC-014 (document parsing for RAG) and DOC-008 (mailroom classification) feed the same extraction pipeline; KNOW-025 (records classification, ROT clean-up) removes obsolete content and supplies retention labels used as filters.
• Neighbours: KNOW-016 multimodal archives (add CLIP/vision embeddings to the same index), KNOW-012 expert finding (people as an indexed entity type).
• Hardware: one HW2 GPU server can host the embedding model, the reranker and a small LLM together with the use cases above.

Effort, cost & KPIs

• Path A: days to weeks for suite-native search; dedicated platforms weeks, priced per user.
• Path B: 1-3 months for a production index with 3-5 connectors and ACL sync.
• Path C: 2-4 months plus ongoing connector maintenance; one server and operations time.
• Cost drivers: number and quality of connectors, ACL complexity, re-indexing volume, answer-layer token costs, licences per user.
• KPIs: search success rate (click or "found it" feedback), zero-result rate, time to find, nDCG or recall@10 on a gold set of 100-300 real queries, index freshness lag, permission-test pass rate, adoption (weekly active searchers), ticket volume for repetitive questions.

People & change

Start with management and the department leads: they decide which sources go in and who owns them. Involve the data owners and IT early, because connectors, accounts and permissions are the critical path. The IT security officer and the data protection officer (DPO) review access control and logging before the pilot; a DPIA is advisable because personal data in documents and mail becomes searchable (GUIDE-005).

Works council: depends. If a works council exists, search logs and query analytics are technically suited to monitor behaviour, so §87(1) no. 6 BetrVG co-determination is likely; §90(1) no. 3 requires timely information anyway. Settle a works agreement that bars individual performance evaluation from search logs (GUIDE-004). In the public sector the staff council (Personalrat) takes this role.

Pick a pilot group of one or two departments with a visible search pain. Typical concerns are "everyone can see my files" (answer: permissions are enforced and tested; oversharing is cleaned up first) and "the AI answers wrongly" (answer: results always link to the source, the answer is a convenience). Measures supporting AI literacy under Art. 4 AI Act can be short onboarding sessions on query writing and checking sources (GUIDE-006, GUIDE-008).

Risks, regulation & compliance

• EU AI Act: minimal risk in the typical case; search and retrieval is not an Annex III use. AI literacy duty (Art. 4) applies [9]. If the answer layer is a chatbot, tell users they interact with AI (Art. 50 transparency). Becomes higher-risk only if the index is used for HR decisions, employee monitoring or similar Annex III purposes (e.g. searching employee communications to evaluate staff).
• GDPR: the index contains personal data (mail, chat, HR files); lawful basis, DPA with vendors, EU region or on-prem, deletion propagating from source to index, DPIA recommended. Works-council co-determination (BetrVG sec. 87) applies in Germany when searching employee communications.
• Security: least-privilege crawler accounts, ACL enforcement at query time, exclusion of sensitive sources (HR, legal hold) unless separately governed, audit logs, prompt-injection defence for the answer layer.
• Mitigations: permission tests with real accounts before roll-out, source whitelist, sensitivity labels as filters, citations for every generated answer.

Getting started

1. List the 5-10 systems that hold most knowledge; rank by use and by sensitivity.
2. Check what the suite you already pay for offers (Copilot Search, Google, Atlassian) before buying.
3. Collect 100 real search queries from helpdesk tickets and interviews with expected results; this is your evaluation set.
4. Pilot one product or stack with 2-3 sources and test permissions with accounts of different roles.
5. Add the answer layer only after retrieval quality is good; track the KPIs monthly.

Sources

1. Microsoft Learn — Document-level access control in Azure AI Search — https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview
2. Microsoft Learn — Microsoft 365 Copilot connectors overview — https://learn.microsoft.com/en-us/microsoft-365/copilot/connectors/overview
3. BAAI — BGE-M3 model card — https://huggingface.co/BAAI/bge-m3
4. Onyx — open-source repository — https://github.com/onyx-dot-app/onyx
5. Elastic — Hybrid search and RRF — https://www.elastic.co/search-labs/blog/hybrid-search-elasticsearch
6. OpenSearch documentation — vector and hybrid search — https://docs.opensearch.org/latest/vector-search/
7. Qwen — Qwen3-Embedding-8B model card — https://huggingface.co/Qwen/Qwen3-Embedding-8B
8. MTEB / MMTEB embedding benchmark — https://github.com/embeddings-benchmark/mteb
9. AI Act Art. 4 (AI literacy) — https://artificialintelligenceact.eu/article/4/