Video is the least searchable asset in most organisations: archives are tagged by hand, if at all, and editors scrub through hours of footage to find a goal, a quote or a product shot. Multimodal models now read frames, speech and on-screen text together, produce time-coded descriptions and tags, build a semantic index and propose highlight clips. Humans stay in the loop for editorial choice and rights.
Problem & value
- Today: broadcasters, publishers, sports clubs, corporate communications and e-learning teams hold thousands of hours of footage with sparse metadata. Logging, clip cutting for social media and compliance checks are manual and slow; most material is never re-used because it cannot be found.
- AI value: searchable archives (find "CEO on stage with a red banner"), automatic chaptering and summaries, faster cutdowns for social and short-form formats, consistent metadata for recommendation and ad placement, and accessibility inputs such as scene descriptions.
- Benefit figures: vendors report large time savings on logging and clip creation, but independent, comparable studies are scarce; measure the baseline in your own workflow before promising numbers.
Variants
- Media archive and MAM enrichment: tagging of legacy and daily ingest (public broadcasters, publishers).
- Sports and events highlights: event-driven clip generation from live feeds, see VIS-015, VIS-055.
- Corporate video, training and webinars: chapters, summaries, searchable town halls, see SPEECH-001.
- Marketing and social: vertical reframing and short-form cutdowns from long recordings.
- Ads and brand safety: contextual tagging for ad placement, brand detection, see VIS-035.
- Private use: searching family videos and auto-generated memory reels in photo apps.
How it works
- ingest (file / live stream)
- shot and scene boundary detection, frame sampling (1-2 fps + keyframes)
- parallel analysis: VLM captions & Q&A | ASR + diarisation | OCR on screen text | object/person/logo detection + tracking | audio events (applause, whistle, crowd)
- fusion: time-coded segments with description, tags, entities, saliency/excitement score
- embeddings (video-text)
- vector index + keyword index
- highlight ranking (rules + model score + data feeds)
- candidate clips
- editor review (accept / trim / reject)
- export to MAM, CMS, social tools
- Video-language models take sampled frames or short clips plus audio transcripts and answer questions or emit structured JSON (segments, tags, summaries). Long videos are handled by chunking and hierarchical summaries.
- Multimodal embeddings place clips and text queries in one vector space for semantic search.
- Highlight detection combines signals: audio energy, crowd noise, scoreboard OCR, event feeds, motion and replay patterns, and model-judged "interestingness". For narrative content, an LLM ranks segments against an editorial brief.
- Human-in-the-loop: editors approve clips before publication; tags with low confidence go to a review queue; rights and personality checks stay manual.
Data preparation
Video archives rarely come as clean data. Footage typically sits on file shares, NAS systems, MAM/DAM tools or external drives, often in mixed formats (MP4, MXF, legacy tape digitisations), with thin or inconsistent metadata and subtitles only for part of the material. Typical gaps are missing or wrong titles, duplicate copies in several resolutions, unknown rights and consent status, no shared tag vocabulary and unclear access rights for unpublished material.
Preparation steps: take an inventory of storage locations and formats (PREP-001), transcode to one proxy format (for example with ffmpeg) so models see small, uniform files, and de-duplicate by checksum. Agree on a tag vocabulary before the first run (PREP-019) and map existing metadata fields from the MAM (PREP-010). Collect 50 to 100 manually tagged clips as a test set (PREP-018). Check rights and personal-data status of faces and voices (PREP-025). Old tape or film material first needs digitising (PREP-006 covers the paper analogue). Speech tracks benefit from existing transcripts (DOC-014 for document-style handling).
Rough share of effort: 25 to 35 percent of the project, mostly inventory, proxy creation and vocabulary work. Models and indexing are the smaller part.
Implementation paths
Which path for whom
- smb (B): Small media teams and marketing departments get tagging fast via cloud video APIs wired into their existing DAM or CMS; no single product fits their metadata fields.
- enterprise (C): Large archives mean high per-minute API costs and rights-sensitive footage, so a shared internal video-AI platform on own GPUs (HW3) pays off.
- developer (B): Cloud multimodal APIs with EU regions give the fastest route to a working pipeline; self-hosting can follow once volumes justify it.
- public (C): Public broadcasters and archives need sovereignty and procurement-proof hosting, so open-weight models on German or EU cloud or own hardware fit best.
Path A — Buy (off-the-shelf)
- Video AI platforms: Twelve Labs (video search and understanding API and models), Google Cloud Video Intelligence, Azure AI Video Indexer, Amazon Rekognition Video / Nova video understanding, Veritone, Clarifai.
- Media-industry (EU/Europe-based): Moments Lab (France, archive indexing and generative metadata), Limecraft and Mediagenix (Belgium), Dalet (France, MAM with AI tagging), Sony Ci. Verify EU hosting and sub-processors for each.
- Sports: Pixellot and Veo (Denmark) auto-highlights for clubs, WSC Sports for broadcasters.
- Choose A when: you need results in weeks, a MAM connector exists and volume is moderate.
Path B — Build on cloud APIs (recommended for most SMBs)
- Models: Gemini (native video input with timestamps, Vertex AI EU regions), Amazon Nova and Bedrock in EU regions, Azure OpenAI/Foundry vision models with frame sampling, Twelve Labs Marengo (embeddings) and Pegasus (generation), Mistral for text-side summarisation and tagging (EU provider).
- Speech: Whisper-class ASR via EU-hosted APIs or self-hosted; diarisation with pyannote.
- Orchestration: FFmpeg and PySceneDetect for cutting, a queue (Temporal, n8n, Airflow), vector DB (pgvector, Qdrant, OpenSearch) and write-back to the MAM via API.
- Choose B when: custom taxonomy, own UI or tight integration is needed, without on-premises constraints.
Path C — Self-host (open source)
- Models: Qwen-VL family (Qwen2.5-VL / Qwen3-VL, with video and temporal grounding), InternVL, LLaVA-Video, SigLIP-based and open video-text embeddings (e.g. InternVideo2, X-CLIP), Whisper / faster-whisper, YOLO or RT-DETR with ByteTrack for objects and people, TransNet V2 for shot boundaries.
- Serving: vLLM or SGLang for VLMs, NVIDIA DeepStream / Metropolis blueprints for stream pipelines, Qdrant or Milvus for vectors, FFmpeg for decoding (NVDEC).
- Hardware: HW3 (48 GB class GPU) for a 30B-class VLM across a growing archive; HW2 works with 7-8B models and aggressive frame sampling. Archive back-catalogues are batch jobs; live highlights need low-latency GPU capacity.
Path D — Custom training
Worth it for sport-specific events (e.g. detecting a particular play type), unusual domains (industrial footage, lecture halls) or a house taxonomy. Needs a few hundred to several thousand labelled clips per class; editors' accept/reject decisions on proposed highlights are the natural training signal. Fine-tune a small VLM or a video classifier on top of frozen embeddings.
State of the art (2026)
- Frontier multimodal models accept hours of video natively, with timestamped answers; open-weight models (Qwen3-VL class) support long video and temporal grounding, narrowing the gap to proprietary APIs.
- Video-text embeddings enable "search by description" across archives without hand-made tags.
- Limits and failure modes: fine-grained temporal localisation (exact frame of an event) is weaker than coarse scene description; models miss brief events between sampled frames; counting and fast action are error-prone; hallucinated details in summaries; bias in "interestingness" towards typical, visually busy moments; cost scales with video length and frame rate; speech in dialects and noisy environments lowers tag quality. Evaluate on your own footage with editors as judges.
Shared resources & combinations
- Same stack (vlm + object-detection + tracking): VIS-015 sports tracking, VIS-031 camera-feed video agents, VIS-055 automated AI camera production; share a GPU video-inference cluster (HW3).
- Audio/text side: SPEECH-001 and SPEECH-004 (ASR, captions), SPEECH-014 (audio description), SPEECH-007 (dubbing and localisation) reuse transcripts and segments.
- Search and governance: KNOW-016 multimodal archive search, VIS-022 visual search, VIS-030 anonymisation before sharing, VIS-027 moderation before publication, VIS-035 logo detection.
Effort, cost & KPIs
- Path A: days to weeks; pricing typically per hour of video indexed or per API minute.
- Path B: 1-3 months incl. MAM integration and review UI; cost scales with minutes processed and frame rate, so index once and cache embeddings.
- Path C: 2-4 months plus GPU operations.
- KPIs: tag precision/recall against editor-labelled samples, search success rate, share of proposed highlights accepted by editors, time from event to published clip, minutes of archive re-used, cost per processed hour.
People & change
Start with management and the department lead (editorial, marketing) to fix scope and the rule that AI proposes tags and clips while editors decide. Involve the data owners of the archive and IT early for storage access, proxy generation and GPU or API budget. Legal and compliance must check image and personality rights, music and licence terms. The DPO is needed because faces and voices are personal data; a DPIA is likely when people are identified or indexed at scale, and every cloud provider needs an AVV (Art. 28 GDPR).
Works council: depends. Archive tagging of published content alone is usually not employee monitoring, but footage showing staff (events, internal video, plant tours) or logging of editors' use of the tool can trigger co-determination under §87(1) no. 6 BetrVG; §90(1) no. 3 requires timely information when work processes change through AI. Where a works council exists, inform it before the pilot.
AI literacy: under AI Act Art. 4 the company must take measures to support staff AI literacy, for example a short session on model errors, hallucinated descriptions and rights checks. Run a pilot with two or three editors on one archive segment, collect their corrections, and show that the system saves search time rather than replacing editorial judgement. Typical concerns (job loss, wrong tags, rights risk) are addressed by keeping a human approval step on every highlight and published tag. See GUIDE-008, GUIDE-005 and GUIDE-009.
Risks, regulation & compliance
- EU AI Act: typically limited/minimal risk. Transparency duties (Art. 50) apply if clips are synthetically generated or manipulated (deepfake labelling); AI literacy (Art. 4) for staff. Keep the system away from Art. 5 prohibited uses: emotion recognition of employees or students from video, and untargeted scraping of facial images. Face identification of persons is a separate, tightly regulated biometric use case (see VIS-016) and should not be enabled by default.
- GDPR: footage of identifiable people is personal data; faces, voices and tags such as inferred sensitive traits (health, religion) raise Art. 9 issues. Define purposes and retention, do not infer sensitive attributes, use EU hosting and a data processing agreement; blur where not needed (VIS-030).
- Copyright and rights: the archive's licence, personality and music rights must be tracked; AI tags do not clear rights. Check provider terms on training with your content.
- Quality risks: wrongly described scenes in news or archive contexts; mitigate with confidence scores, editor sign-off and an audit log of AI-generated metadata.
Getting started
- Pick one workflow (e.g. archive search or weekly highlight reel) and measure the manual baseline.
- Collect 50-100 representative clips with editor-made tags or desired highlights as a test set.
- Run 2-3 options (one Path A product, one Gemini/Twelve Labs API, one open VLM) on the set and compare.
- Add a review queue in the editor's tool; log accept/reject decisions.
- Scale to the full archive in batch and monitor cost per hour and tag accuracy monthly.
Sources
- Twelve Labs documentation (Marengo and Pegasus models) — https://docs.twelvelabs.io
- Google Cloud — Gemini video understanding — https://ai.google.dev/gemini-api/docs/video-understanding
- Microsoft — Azure AI Video Indexer — https://learn.microsoft.com/azure/azure-video-indexer/
- Amazon Rekognition Video — https://docs.aws.amazon.com/rekognition/latest/dg/video.html
- Qwen3-VL project — https://github.com/QwenLM/Qwen3-VL
- PySceneDetect — https://github.com/Breakthrough/PySceneDetect
- NVIDIA Metropolis / DeepStream — https://developer.nvidia.com/deepstream-sdk
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj