Publishers, broadcasters and streaming platforms hold far more content than any user can browse. Recommender systems learn from behaviour and from the content itself which article, video, track or podcast to show next. Done well they raise watch time, subscriptions and return visits; done badly they create filter bubbles, favour clickbait and fall foul of EU transparency and profiling rules. In Europe the design question is as much "what are we optimising for, and can users opt out" as "which model".
Problem & value
- Today: home pages are curated by editors plus simple rules (most-read, latest, same category). Users face large catalogues, churn when they do not find something quickly, and publishers monetise attention poorly for the long tail.
- AI value: personalised ranking that mixes collaborative signals (what similar users consumed) with content understanding (text, audio and video embeddings), handles cold start for new items (news is perishable) and balances engagement with diversity, freshness and editorial priorities.
- Evidence: Netflix has publicly described its recommender as central to retention and discovery, and Meta/Spotify/YouTube report personalised feeds as core products; published figures come from vendors and are self-reported, so measure your own uplift with A/B tests rather than relying on them [1][5].
- Public service media: European broadcasters use recommenders for reach while committing to diversity and public value; see EBU guidance [6].
Variants
- News publisher: short item lifetime, strong recency, editorial front-page control, paywall propensity, newsletter personalisation.
- Video/audio streaming (SVOD, AVOD, FAST): session-based "up next", row/carousel composition, artwork personalisation, kids profiles.
- Public service broadcaster: diversity-aware ranking, "public value" objectives, opt-in personalisation, no third-party tracking.
- SMB/local media: managed service using content similarity and little behavioural data.
How it works
- event tracking (consented) + content ingestion
- content understanding (text/audio/video embeddings, topics, entities, language)
- candidate generation (two-tower / sequence model + ANN vector search + editorial pools)
- ranking (multi-objective model: relevance, completion, satisfaction, diversity, freshness)
- re-ranking & business rules (licences, age rating, geo-blocking, sponsor/editorial slots, dedup)
- serving (feed, carousel, newsletter, push)
- feedback loop (A/B tests, bandit exploration, offline evaluation)
- Candidate generation: matrix factorisation or two-tower retrieval returns a few hundred candidates from millions via approximate nearest-neighbour search; for new items, content embeddings from multilingual text encoders or audio/video models solve cold start.
- Ranking: gradient-boosted trees or deep ranking models (DLRM-style, transformers over user history) predict clicks, completion and longer-term satisfaction signals; modern systems optimise several objectives, not click-through alone.
- LLMs: tags, summaries, conversational discovery and explanations; they rarely replace the core ranker (latency, cost).
- Human-in-the-loop: editors pin, boost or block items and set guardrails (e.g. sensitive topics, election coverage); data scientists review diversity and fairness dashboards; users control profile, history and a non-personalised mode.
Data preparation
Data needed: content items (text, audio, video metadata, topics, licence and age-rating information) and interaction events (reads, plays, completions, skips, impressions) with a consent flag. In a media SMB these live in the CMS, a video platform, web analytics, the newsletter or subscription system and often in archives with inconsistent tagging.
Typical gaps: missing or free-form tags, duplicate articles, no unified user ID across web, app and newsletter, impressions not logged (only clicks), events collected without consent flag, and an unstructured archive with legacy formats.
Preparation steps: - Define a metadata schema and tag the archive consistently, using LLM-assisted tagging with editorial review (PREP-010, PREP-019). - Convert legacy formats and old archives (PREP-007). - Build the event pipeline with impressions, consent flag and pseudonymised IDs (PREP-005, PREP-022, PREP-025). - Build evaluation sets with editors (PREP-018).
Tools: CMS APIs, Kafka or a managed event stream, dbt, embedding models for content vectors, a consent management platform.
Effort share: about 40-50 percent of the project, because logging quality limits everything downstream. See also GUIDE-003.
Implementation paths
Which path for whom
- SMB: A. Small publishers and local media can use managed recommenders (e.g. Recombee, Piano) at SMB prices instead of building ML operations.
- Enterprise: B. Large catalogues and traffic, editorial rules and own data make a custom pipeline on managed EU-region cloud services the best fit, with lower fees than per-request products.
- Developer: C. Open-source libraries, Qdrant or pgvector and Feast give full control and avoid lock-in.
- Public: C. Public broadcasters and public-sector media need sovereignty, transparent ranking and public-service values (diversity, no engagement-only optimisation), best met on self-hosted or sovereign EU/German cloud.
Path A: Buy (off-the-shelf)
- Managed recsys platforms: Amazon Personalize, Google Cloud Vertex AI Search for media (media recommendations), Microsoft Azure-based options and Algolia Recommend; these include user-item-interaction ingestion and ranking recipes [2][3].
- Media-specialised vendors: Piano (news personalisation, subscription propensity), Taboola/Outbrain (content discovery, ad-supported), Recombee (EU, Czech recommender API), Bloomreach (EU region available). Verify EU hosting, DPA and cookie-free operation.
- Choose A when: small team, catalogue under a few million items, no wish to run ML operations, and standard objectives are good enough.
Path B: Build on cloud APIs (recommended default for enterprises)
- Retrieval and ranking: managed vector search (OpenSearch/Elasticsearch, Vertex AI Vector Search, Azure AI Search, pgvector) plus a managed training service (Vertex AI, SageMaker, Azure ML) in EU regions; features from your warehouse (BigQuery, Snowflake, Databricks).
- Embeddings and LLM layer: multilingual embedding models (e.g. Mistral embed, Cohere Embed, OpenAI/Gemini embeddings, EU-hosted) for content vectors, and an LLM API for tagging, summaries and explanations.
- Integration: event pipeline (Kafka/Kinesis/Pub/Sub), feature store, serving API with sub-100 ms latency budget, consent management platform feeding a consent flag into every event.
- Choose B when: you need custom business rules, editorial controls and your own data, without operating GPUs.
Path C: Self-host (open source)
- Libraries: NVIDIA Merlin (HugeCTR, NVTabular, Transformers4Rec), TorchRec, RecBole, Microsoft Recommenders, implicit/LightFM for baselines, LightGBM for rankers, Meta's open-sourced generative-recommender code (HSTU) for research reproduction [4].
- Retrieval: Qdrant (EU, Berlin), Milvus, Weaviate, FAISS; embeddings from open models (BGE-M3, multilingual-e5, Qwen3-Embedding).
- Serving and ops: Kubernetes, Feast feature store, MLflow, Triton or vLLM for LLM parts.
- Hardware: HW1 for CPU-based retrieval at small scale; HW3 (one pro GPU) for training and serving mid-size deep rankers; large catalogues need HW4.
Path D: Custom training
For large audiences where a few percent uplift pays for the team: train sequence rankers on months of event logs incl. impressions and negative feedback; debias for position and popularity; keep a non-personalised holdout.
State of the art (2026)
- Sequence and generative recommenders: transformer-based models over user histories (e.g. Meta's HSTU, "Actions Speak Louder than Words") scale with compute like language models and are replacing classic feature-heavy DLRM stacks at the largest platforms [4]. Semantic IDs plus LLM-style generation of the next item are an active research area.
- Multi-objective and long-term optimisation: moving from clicks to satisfaction surveys, return rate and "regret" signals; bandits and reinforcement learning for exploration.
- Diversity and pluralism: diversity-aware re-ranking, exposure metrics and user-controlled feeds, driven by regulation (Digital Services Act) and public-service remits [6].
- Limits and failure modes: popularity and position bias, filter bubbles, feedback loops that amplify sensational content, cold start for new users and items, offline metrics that do not match online outcomes, and gaming by content producers. News recommenders are sensitive because pluralism and trust matter beyond engagement.
Shared resources & combinations
- Shared stack: recommender, embeddings, vector DB, event pipeline and consent layer with REC-001, REC-003, REC-006; segments from REC-004.
- Content understanding: KNOW-016, VIS-025, SPEECH-013, LANG-028 produce metadata/embeddings; LANG-014 and VIS-027 filter candidates.
Effort, cost & KPIs
- Path A: weeks; priced per request or per active user.
- Path B: 3 to 6 months for a first production system with A/B testing; cost drivers are event volume, feature store, training runs and low-latency serving.
- Path C/D: 6 to 12 months and a small ML platform team; GPU cost dominates at scale.
- KPIs: click-through and completion rate, watch/read time per session, return frequency, subscription conversion and churn, catalogue coverage and long-tail share, diversity and exposure metrics, share of users in non-personalised mode, latency, and online A/B uplift against a non-personalised control.
People & change
Involve: management and the editor-in-chief or product lead (process owner) from the start, since ranking shapes editorial identity; IT for event pipeline and CMS integration; the data protection officer for tracking, profiling and consent (GUIDE-005); legal for licences, age rating and, for public bodies, mandate requirements; readers or users through a visible control to switch to non-personalised mode.
Works council: depends. The recommender evaluates readers, not staff, so §87(1) no. 6 BetrVG is usually not triggered. But editorial dashboards that show performance per journalist are suited to monitor behaviour and need a works agreement; also inform the council under §90(1) no. 3 BetrVG when editorial workflows change (GUIDE-004). Public bodies involve the Personalrat.
Data protection: AVV with processors; a DPIA is likely for large-scale profiling.
AI literacy: under AI Act Art. 4, take measures that support staff AI literacy; train editors on how ranking works, what it optimises and how to pin, boost or block items (GUIDE-006).
Pilot and communication: pilot on one section or newsletter with a non-personalised control. Typical concerns are editorial independence and filter bubbles; address them with editor override, diversity metrics and transparency to readers. See GUIDE-008 and GUIDE-009.
Risks, regulation & compliance
- EU AI Act: recommender systems are generally minimal risk; no Annex III category applies to media recommendation. Art. 4 AI literacy applies. Art. 5 prohibits manipulative or exploitative techniques that materially distort behaviour and cause significant harm, so dark patterns, exploiting minors' vulnerabilities or subliminal techniques in engagement design are a risk. Generated summaries or synthetic content bring Art. 50 transparency duties [7].
- Digital Services Act: online platforms must explain the main parameters of recommender systems in their terms (Art. 27); very large online platforms (VLOPs) must offer at least one recommender option not based on profiling (Art. 38), assess systemic risks (Art. 34) and, for minors, apply high privacy, safety and security protections (Art. 28); profiling-based ads to minors are banned [8].
- GDPR and ePrivacy: behavioural profiling needs a legal basis, usually consent for tracking via cookies or device identifiers (ePrivacy Art. 5(3)); legitimate interest can cover first-party on-site personalisation but needs a documented balancing test, transparency and an easy opt-out; special-category inferences (religion, politics, sexuality from reading habits) require care and generally explicit consent. Run a DPIA for large-scale profiling [9].
- Media law: the European Media Freedom Act and public-service remits stress pluralism and editorial independence.
- Mitigations: consent-aware event pipeline, non-personalised fallback, editorial guardrails, diversity metrics reviewed regularly, audit logs of ranking changes, user controls to reset history, documentation of recommender parameters, red-teaming for harmful-content amplification.
Getting started
- Define objectives beyond clicks (satisfaction, return, diversity) and a non-personalised baseline to beat.
- Instrument consented event tracking incl. impressions; fix item IDs and metadata quality first.
- Start with content-based similarity (embeddings plus vector search) and popularity/recency blends; add collaborative signals once data volume allows.
- Run an A/B test of one placement (e.g. "related articles") with guardrail metrics and editors involved.
- Publish a plain-language explanation of recommender parameters and add a non-personalised option; complete a DPIA.
Sources
- Gomez-Uribe & Hunt — The Netflix Recommender System: Algorithms, Business Value, and Innovation — https://dl.acm.org/doi/10.1145/2843948
- AWS — Amazon Personalize documentation — https://docs.aws.amazon.com/personalize/latest/dg/what-is-personalize.html
- Google Cloud — Vertex AI Search for media — https://cloud.google.com/solutions/media-recommendations
- Zhai et al. — Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations (HSTU) — https://arxiv.org/abs/2402.17152
- NVIDIA Merlin — https://github.com/NVIDIA-Merlin/Merlin
- EBU — Public service media and recommendation systems (EBU Technology & Innovation) — https://tech.ebu.ch/
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Regulation (EU) 2022/2065 (Digital Services Act) — https://eur-lex.europa.eu/eli/reg/2022/2065/oj
- Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj