Multimodal search in image, video & audio archives

Archives of photos, footage and recordings are usually searchable only through hand-entered titles, keywords and folder names, so most of the content is effectively invisible. Multimodal embeddings, speech recognition and vision-language models index the content itself: a user types "CEO shaking hands at the trade fair, 2019" or uploads a reference image, and gets the matching images, video moments and spoken passages with timecodes. Typical benefit: far less time spent hunting for assets, and re-use of material that would otherwise be re-shot or re-bought.

Problem & value

• Today: media houses, marketing departments, museums, broadcasters, municipalities, universities and manufacturers keep terabytes of images, footage and recordings in DAM/MAM systems or on file shares. Findability depends on manual cataloguing (inconsistent, expensive, incomplete) and on people remembering where something is. Audio is practically unsearchable without transcripts.
• AI value: content-based retrieval replaces manual tagging as the primary access path: text-to-image, text-to-video-moment, image-to-image, audio/speech search and combined filters. Output is a moment (file + timecode), not just a file.
• Evidence: vendors report retrieval gains on their own data, e.g. Google's launch post for Gemini Embedding 2 quotes a partner text-to-video Recall@1 of 85.3 % — a partner-reported figure on that partner's data, not an independent benchmark [1]. Always measure on your archive with a labelled query set (see Getting started).

Variants

• Media & broadcast / agencies: rights-cleared footage and photo search, b-roll finding, news archive, highlights; speech + visual + on-screen text combined.
• Corporate marketing & DAM: brand image libraries, product shots, event photos, trade-fair video.
• Public sector & cultural heritage: municipal photo archives, parliament recordings, oral-history audio; overlaps with KNOW-019 (collection discovery, metadata standards).
• Education & research: lecture-video search, interview archives (qualitative research).
• Industry: inspection footage, training videos, maintenance clips ("show the filter change on model X"); related VIS-025/VIS-031 for analytics and camera feeds.
• Private use: personal photo library search ("dog on the beach") — e.g. Immich, Google/Apple Photos.
• Out of scope here: surveillance-camera analytics (VIS-031/VIS-010), face-based identification of people (VIS-016, see risks), forensic seized-media triage (VIS-050), moderation (VIS-027).

How it works

1. ingest (files, DAM/MAM, object storage)
2. pre-processing (shot/scene detection, frame sampling, audio extraction, de-duplication)
3. per-modality indexing: images/frames
4. multimodal embedding (CLIP/SigLIP-class) + VLM caption + OCR of on-screen text audio/speech
5. ASR with word timestamps (+ diarization)
6. text embedding + BM25 video
7. clip-level video embedding and/or frame embeddings + transcript + captions
8. store (vector DB + keyword index + metadata/rights in relational store)
9. query (text | example image/clip | filters)
10. hybrid retrieval (dense + sparse)
11. rerank
12. UI with thumbnails/timecodes (optional: VLM/LLM answer with cited moments)

• Joint embedding space: image-text models such as SigLIP 2 (Apache-2.0) map pictures and text into one vector space for zero-shot retrieval [3]. Newer natively multimodal embedders (Gemini Embedding 2: text, image, video clips up to 120 s, audio without transcription, PDFs) put several modalities in one space [1]; video-native models such as Twelve Labs Marengo index visual, audio and text signals of a video [5][6].
• Speech layer: Whisper (MIT) and its optimised derivatives provide transcripts; WhisperX adds word-level alignment and speaker labels [2][8][9]. Transcript search is still the most precise way to find spoken content, so production systems combine it with embeddings (hybrid).
• Document-like visuals (slides, scanned pages, charts inside video): late-interaction models such as ColPali/ColQwen embed page images directly, without OCR [4].
• Human-in-the-loop: editors confirm results before publication; rights/consent metadata is a hard filter, not an AI guess; low-confidence auto-tags are shown as suggestions.

Data preparation

Archives live on NAS and file shares, in a DAM or media server, on external drives, in cloud storage and as tapes or paper prints. Typical gaps are inconsistent file names, missing or wrong metadata (date, place, rights holder), duplicates and different versions of the same asset, analogue material, mixed formats and codecs, and unclear usage rights and consent of people shown.

Preparation steps: (1) inventory the archive and decide which part is searchable first (PREP-001); (2) digitise analogue film, tapes and prints (PREP-006); (3) transcode old formats into standard codecs and proxies (PREP-007); (4) remove duplicates and agree on a folder and naming convention (PREP-009); (5) transcribe audio and speech with Whisper (DATA-008); (6) add essential metadata - rights, source, date - and a tag scheme (PREP-010, PREP-019); (7) build a test set of typical search queries with the editors (PREP-018); (8) mark assets with personal data or restricted rights (PREP-024).

Tools: ffmpeg, Whisper, Immich or a DAM for indexing, ExifTool for metadata. Effort share: medium, about 30-40 percent, mostly digitisation and rights clean-up rather than the AI search itself.

Implementation paths

Which path for whom

• SMB: Path A - digital asset management products with AI search are mature and priced for small teams, and the archive process is standard.
• Enterprise: Path B - large archives need integration with the existing asset and rights systems and cost control through cloud APIs and own indexing.
• Developer: Path C - open models (CLIP-type embeddings, Whisper) and vector databases are well documented and run on a single GPU.
• Public sector: Path C - archives with personal images and records call for self-hosting or a sovereign EU cloud.
• Private: Path C - local apps such as Immich keep family photos and recordings on the own device.

Path A — Buy (recommended for most SMBs and media teams)

• DAM/MAM with AI search built in: e.g. Adobe Experience Manager Assets, Bynder, Canto, Brandfolder, Cloudinary, Frame.io; media-asset systems such as Dalet, EVS, Avid; photo tools such as Google Photos and Apple Photos (private). EU/German options: Picturepark (CH), Celum (AT), Censhare (DE), Pimcore DAM (AT/DE), Sedna/ Dalet (FR). Check which AI features run in the EU region.
• Specialised video-search platforms: Twelve Labs (API and Amazon Bedrock models) [5][6], Google Vertex AI Search / Video Intelligence, Microsoft Azure AI Video Indexer, Veritone, Moments Lab (FR), Valossa (FI).
• Choose A when: < ~1 million assets, a DAM already exists, no special data-residency demands. Test with your own hard queries, and verify hosting region, retention and training-on-customer-data terms.

Path B — Build on cloud APIs

• Embeddings: Gemini Embedding 2 (public preview as of its March 2026 launch via Gemini API and Vertex AI; check current GA/EU-region status) [1]; Twelve Labs Marengo; text embeddings from EU providers (Mistral, Aleph Alpha-class offerings) for transcripts.
• Captioning / question answering: frontier VLMs (Claude, Gemini, GPT, Mistral) over sampled frames with structured output; Gemini additionally accepts video natively.
• ASR: managed speech APIs in EU regions (Azure, Google, AWS Transcribe in Frankfurt, Speechmatics, Mistral Voxtral) or Whisper hosted at an EU provider.
• Index & orchestration: Qdrant, Weaviate, pgvector, OpenSearch/Elasticsearch for hybrid search [10]; n8n or a custom service for ingest. Cost driver is one-off indexing of the archive, then cheap queries.
• Choose B when: custom UI or deep integration is needed, archive is large, and cloud processing is legally acceptable (see GDPR below).

Path C — Self-host (open source)

• Models: SigLIP 2 / CLIP-class (Apache-2.0) [3], open multimodal embedders (e.g. recent Qwen3-VL- or Nemotron-based omni-modal embedding models — check licence on each model card), ColPali/ColQwen for page-like visuals (note: ColPali weights carry the Gemma licence, ColQwen variants Apache-2.0) [4], Whisper / faster-whisper (MIT) and WhisperX (BSD-2) for speech [2][8][9], open VLMs (Qwen-VL family) for captions, vLLM for serving.
• Stack: Qdrant (dense, sparse, multi-vector, hybrid) [10] or OpenSearch + pgvector, ffmpeg + PySceneDetect for shots, MinIO/S3 for storage. Immich (AGPL-3.0) is a mature turnkey option for photo/video libraries with CLIP search [7]; check AGPL implications for commercial embedding.
• Hardware: HW2 (one 24–32 GB GPU) is sufficient for indexing hundreds of thousands of images and thousands of hours of audio over days to weeks, and for interactive queries; faster-whisper reports roughly 4x speed over reference Whisper [8], WhisperX up to 70x realtime with batched large-v2 [9] (vendor-stated, hardware-dependent). Use HW3 for VLM captioning of whole video corpora or larger omni models. Indexing is a batch workload: schedule at night, scale GPU time to archive size.
• Choose C when: footage is confidential (unpublished material, investigations, patient or personal recordings), on-prem is required, or the archive is so large that API indexing costs dominate.

Path D — Custom training

Rarely needed first. Worth it for domain-specific vocabulary (e.g. fine-tuning a CLIP/SigLIP encoder on product or part imagery, or a domain ASR on dialect/technical speech), for in-house labelled concepts (own logos, products, locations), or to distil a cheap tagger from a VLM. Needs a few thousand query–result pairs or labelled images; click logs and editor corrections from path A–C are the natural training data.

State of the art (2026)

• Natively multimodal embeddings arrived: Gemini Embedding 2 (March 2026) embeds text, images, short video, audio and PDFs into one space, with Matryoshka dimensions (default 3072) [1]. Open-weight omni-modal embedders followed; verify maturity and licences before committing.
• Video-native retrieval models (Twelve Labs Marengo/Pegasus, also on Amazon Bedrock) offer "search a moment by description" and video Q&A as a service [5][6].
• Late-interaction (multi-vector) retrieval for visually rich documents: ColPali-family models score around 89–91 on the ViDoRe leaderboard (as listed in the repository) and are supported by vector DBs with multivector storage [4][10].
• Hybrid wins: dense multimodal retrieval alone misses exact names, numbers and rare terms; transcripts + OCR + BM25 + reranker remain the robust recipe.
• Limits and failure modes: video embeddings handle short clips (Gemini: 120 s per request), so long footage must be segmented; fine-grained queries (counting, left/right, small text, "who said what" without diarization) fail often; CLIP-class models are weak on non-English queries unless multilingual variants are used; ASR degrades with dialect, overlapping speakers, music and noise; VLM captions can hallucinate details; retrieval models carry demographic and cultural bias; benchmarks on public data overstate performance on niche archives. Reported partner figures are not independent [1].

Shared resources & combinations

• Same building blocks: vlm + embeddings + vector-db + asr are shared with VIS-022 (visual search), VIS-025 (video tagging/highlights), VIS-031 (video agents over camera feeds), SPEECH-001/SPEECH-002 (transcription pipelines — the same Whisper/WhisperX server), DOC-014 (document parsing for RAG, ColPali for scanned pages) and KNOW-001/KNOW-002 (hybrid retrieval, reranker, ACL-aware indexing).
• Natural combinations: KNOW-019 (cultural-heritage metadata and catalogue standards on top of the index), VIS-030 (anonymise faces/plates before publication or sharing), VIS-027 (moderate before release), SPEECH-010 (speaker identification — note biometric law), LANG-018 (media monitoring).
• Hardware: one HW2/HW3 GPU server running vLLM (VLM captions), an embedding service and faster-whisper can serve all of the above; the vector DB and object storage are shared infrastructure.

Effort, cost & KPIs

• Path A: days to weeks; licence per user/storage, AI search often a paid tier.
• Path B: 1–3 months to production incl. ingest, UI and rights filter; main cost is one-off indexing (per minute of video/audio and per image) plus re-indexing when the embedding model changes — budget for that, because switching embedding models means re-embedding the archive.
• Path C: 2–4 months; GPU server, storage, operations; index rebuild on model upgrades.
• KPIs: Recall@k / MRR on a labelled query set of real user searches; time-to-find an asset; share of searches ending without a click; share of assets reused; editor correction rate on auto-tags; indexing throughput (hours of media per GPU-hour); cost per indexed hour.

People & change

Involve management for scope and budget, the department lead (editorial, marketing, archive) as owner, and the archivists and editors as end users who define what a good hit is. Data owners confirm which collections and rights may be indexed; IT handles storage, GPU or cloud capacity; the DPO and legal/compliance clarify rights, consent of depicted people and biometric aspects.

Works council: depends. Searching a product or media archive does not by itself monitor staff. It does when employees or customers appear in the material (CCTV, shop-floor video, training recordings) and face or voice search could identify them, or when search logs show who looks for what: then §87(1) no. 6 BetrVG applies, and §90(1) no. 3 requires early information. Exclude person-identifying search in the pilot; public bodies involve the Personalrat. GDPR: photos and voices are personal data, face recognition touches special-category biometric data; an AVV with every processor and a DPIA for large archives with identifiable people are likely (GUIDE-005).

AI literacy: support the team with training on how semantic search fails (missed hits, wrong captions) and on AI-generated tags (GUIDE-006). Communicate that AI proposes hits and tags, while humans decide on publication and rights. Pilot with one collection and 3-5 editors, measure time to find an asset, and collect corrections (GUIDE-009). Typical concern: loss of control over rights - address with permission-aware search and logged access.

Risks, regulation & compliance

• EU AI Act (Reg. 2024/1689): the plain search function is minimal risk (Art. 4 AI literacy applies) [11]. Classification becomes depends: face-based search or identification of persons is biometric identification (Annex III; real-time remote biometric identification in public spaces for law enforcement is restricted under Art. 5), and building face databases by untargeted scraping of images from the internet or CCTV is prohibited (Art. 5(1)(e)). This use case therefore documents content search without identity matching; person look-ups should use consented, closed lists and are handled in VIS-016. If search results or auto-captions are published as AI-generated content, Art. 50 transparency applies to synthetic media, not to retrieval itself.
• GDPR: images, video and voice of identifiable people are personal data (voice and face may become special-category data when used for identification). Needs a legal basis (legitimate interest with a balancing test for archive use; consent or journalistic/archival exemptions as applicable), data-protection impact assessment for large people-centred archives, processor agreements and EU region/processing for cloud paths, retention rules, and deletion that also removes embeddings and transcripts. Embeddings are derived personal data.
• Copyright & rights: index search does not grant usage rights; keep licence, model-release and expiry metadata as hard filters. Check text-and-data-mining terms when archives contain third-party material.
• Security: enforce source ACLs at query time (as in KNOW-001); transcripts leak sensitive speech content; prompt-injection through on-screen text or spoken content is possible when a VLM/LLM answers over results.
• Mitigations: hybrid retrieval with citations (timecode), human review before publication, audit log, region-pinned processing, opt-out for sensitive collections, evaluation on your own bias-relevant queries.

Getting started

1. Pick one archive slice (e.g. 5,000 images, 200 h of video/audio) and collect 50–100 real user queries with known correct results as a gold set.
2. Quick benchmark: your DAM's AI search vs. a Path-B prototype (frame embeddings + Whisper transcripts + hybrid search in Qdrant/pgvector); compare Recall@10 and failure cases.
3. Clarify legal ground early: persons visible/audible, rights metadata, hosting region, DPIA need.
4. Decide on segmentation (shots/scenes, 10–60 s clips) and the metadata schema (timecode, speaker, rights).
5. Roll out with an editor feedback button; log corrections and plan re-indexing when switching models.

Sources

1. Google — Gemini Embedding 2: first natively multimodal embedding model (accessed 2026-10) — https://blog.google/innovation-and-ai/technology/developers-tools/gemini-embedding-2/
2. OpenAI — Whisper repository (models, MIT licence, turbo) — https://github.com/openai/whisper
3. Google — SigLIP 2 SO400M model card on Hugging Face (Apache-2.0) — https://huggingface.co/google/siglip2-so400m-patch14-384
4. ColPali / ColQwen engine and ViDoRe leaderboard table — https://github.com/illuin-tech/colpali
5. Twelve Labs documentation — Marengo and Pegasus, search/analysis/embeddings — https://docs.twelvelabs.io/docs/get-started/introduction
6. AWS — TwelveLabs models on Amazon Bedrock — https://aws.amazon.com/bedrock/twelvelabs/
7. Immich — self-hosted photo and video management with CLIP search (AGPL-3.0) — https://github.com/immich-app/immich
8. SYSTRAN — faster-whisper (benchmarks, MIT) — https://github.com/SYSTRAN/faster-whisper
9. WhisperX — word-level timestamps and diarization (BSD-2) — https://github.com/m-bain/whisperX
10. Qdrant documentation — multivector, hybrid search, self-hosting — https://qdrant.tech/documentation/
11. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj