Music information retrieval (tagging, stem separation, transcription)

Music libraries, labels, sync agencies, producers and music teachers handle large amounts of audio that is poorly described and hard to edit. Music information retrieval (MIR) models listen to a track and return tags, embeddings, separated stems and note or chord transcriptions. This turns manual cataloguing and ear-transcription into a reviewable, automated step. Many tools run locally on a laptop or one GPU.

Problem & value

• Today: catalogue metadata is entered by hand or by freelance taggers; production libraries and sync teams search by keywords that do not match what they hear; arrangers transcribe by ear; remixers, karaoke producers and sample-based artists lack clean stems.
• AI value: consistent tagging at catalogue scale, search by example ("find tracks that sound like this"), stems from stereo mixes, and draft scores or chord sheets for a human to correct.
• Evidence level: vendor and research claims on tagging accuracy and separation quality are benchmark-specific (e.g. MUSDB18, MTG-Jamendo). Measure on your own repertoire before trusting any figure.

Variants

• Catalogue tagging & similarity search (labels, production music, streaming, broadcasters, archives): genre/mood/instrument/BPM/key tags and embedding search; see KNOW-016 for multimodal archive search.
• Stem separation (producers, DJs, remix/karaoke, film post-production, restoration of old recordings): also used to isolate vocals before clean-up (SPEECH-008).
• Automatic music transcription (arrangers, publishers, music educators): audio → MIDI/MusicXML, chord and beat tracking.
• Real-time performance feedback / music learning apps (private, tagged private; Yousician, Simply Piano, Flowkey): the same pitch, onset and chord models run in real time against a score, on the phone, to tell learners whether notes and timing were right. Related to EDU-004 (AI tutoring).
• Rights and royalty support: content identification and AI-generated-music detection; this overlaps with GEN-007 (music generation) and is a policy-heavy variant.

How it works

1. audio ingest (file / stream / microphone)
2. decode, resample, normalise
3. feature front end (mel-spectrogram, CQT) or waveform encoder
4. task models: tagging | embedding | beat/tempo/key | source separation | note/chord transcription
5. post-processing (thresholds, note segmentation, quantisation, ensembling)
6. outputs: tags, vectors in index, stems, MIDI/MusicXML
7. human review (tag QA, score editing in a notation editor)
8. catalogue / DAW / app

• Tagging/embeddings: audio encoders (CNN or transformer, self-supervised models such as MERT or contrastive audio-text models like CLAP) produce vectors used for tags, zero-shot text search and similarity; vectors live in a vector database.
• Separation: neural source separators (Demucs family, hybrid waveform/spectrogram transformers, band-split RNN/RoFormer variants) estimate vocals, drums, bass and other.
• Transcription: lightweight pitch/onset models (e.g. Basic Pitch) for monophonic or light polyphonic audio; larger transformer models for piano and multi-instrument audio.
• Human-in-the-loop: editors correct tags, and arrangers correct transcriptions; automatic output is a draft. Stems used commercially need a rights check first.

Data preparation

The data is audio plus catalogue metadata (ISRC, genre, BPM, rights). In a small label, studio or music school it sits on file shares, external drives, DAW project folders and in a spreadsheet or catalogue tool; larger catalogues live in a media asset management system or DMS. Typical gaps are mixed formats and sample rates, duplicate versions (demo, master, radio edit), inconsistent or missing tags, unclear rights per track, and no ground truth for your own genre vocabulary.

Steps: 1. Inventory files and rights status (see PREP-001). 2. Decode and normalise to one format (ffmpeg), deduplicate by audio fingerprint. 3. Fix a tag scheme before tagging (PREP-019). 4. Harvest existing labels from catalogue records as a starting set (PREP-020). 5. Build a small test set with a musician or A&R editor (PREP-018).

Pre-trained models work without training data, so preparation is a small share of the effort, roughly 10 to 20 percent, mostly rights clarification and tag-scheme decisions.

Implementation paths

Which path for whom

• smb: Path C, because Demucs, Essentia and Basic Pitch run on HW2 and unreleased masters stay in-house.
• enterprise: Path C, because large catalogues make per-minute API or seat costs high and one internal audio-embedding platform serves tagging, search and recommendation.
• developer: Path C, because the open-weight models are the building blocks of the product.
• private: Path A, because stem and practice apps are mature, cheap and need no setup.

Path A — Buy (off-the-shelf)

• Catalogue tagging/search: Cyanite (EU, Germany), Musiio (now part of SoundCloud), AIMS API, Musicstax-style metadata services; DAM/MAM suites for broadcasters.
• Stem separation and transcription tools: LALAL.AI, Moises, AudioShake (B2B), iZotope RX, Steinberg SpectraLayers, Logic Pro and other DAWs with built-in stem split; AnthemScore, Klangio (EU, Germany) for transcription.
• Music learning apps: Yousician (EU, Finland), Simply Piano (Simply), Flowkey (EU, Germany).
• Choose A when: a small team wants results quickly with few files, or needs a polished UI.

Path B — Build on cloud APIs

• Vendor APIs for tagging and separation (Cyanite, AudioShake, LALAL.AI); general multimodal APIs (Gemini, GPT-class audio models) can describe music in free text but are less reliable for precise note-level work.
• Orchestrate with queue + object storage in an EU region; store tags in your DAM or search index.
• Choose B when: you need MIR inside an existing product without GPU operations.

Path C — Self-host (open source)

• Separation: Demucs / HT Demucs (Meta, repository now community-maintained), Open-Unmix, Spleeter (older), Band-Split RoFormer-class models via the Music Source Separation Training toolkit.
• Transcription: Basic Pitch (Spotify, runs on CPU and in the browser), MT3 (Google Magenta), Omnizart, Onsets and Frames for piano.
• Tagging/embeddings: Essentia and its pretrained models (MTG, Universitat Pompeu Fabra, EU), MERT, CLAP, MusiCNN; librosa and madmom for beats, tempo and chords.
• Stack: Python, PyTorch or ONNX, a vector database such as pgvector/Qdrant, batch queue.
• Hardware: HW1 for tagging and Basic Pitch; HW2 (one 16-24 GB GPU) for batch separation of a large catalogue and for real-time use on a workstation.

Path D — Custom training

Worthwhile for a house taxonomy (own mood vocabulary), non-Western or niche repertoire (folk, early music, Indian or Arabic modal music), or an instrument-specific transcriber. Needs a few thousand tracks with consistent labels; label quality matters more than quantity. Licensed training data is mandatory; see risks.

State of the art (2026)

• Separation from stereo mixes is good for vocals and drums, weaker for dense "other" content and instruments with overlapping spectra; audible artefacts (bleed, phasing) remain, especially on live or lo-fi recordings.
• Self-supervised and audio-text models give strong embeddings for zero-shot tagging and text search over catalogues; vocabulary and cultural bias towards Western pop remains a known limit.
• Transcription is mature for solo piano and guitar; multi-instrument, dense polyphony, drums and singing voice remain error-prone. Outputs need an editor.
• Generative music (GEN-007) made detection of AI-generated tracks a new MIR task; detectors trade off false positives against evasion and should not be the sole basis for penalising artists.
• Real-time pitch and onset detection runs on-device on phones, which makes practice apps feasible without sending audio to a server.

Shared resources & combinations

• Same audio stack (audio-classification, embeddings, edge-runtime) as SPEECH-009 (audio event detection) and SPEECH-015 (bioacoustics); separation overlaps SPEECH-008 (enhancement, podcast editing).
• Search: embeddings and tags feed KNOW-016; vector index shared with text RAG.
• Generation: GEN-007 produces material that MIR tags; SPEECH-007 (dubbing) needs vocal/background separation.
• Learning: EDU-004 for tutoring around the feedback loop.
• One HW2 GPU server can host separation, tagging and transcription jobs in a queue.

Effort, cost & KPIs

• Path A: days; per-track or subscription pricing.
• Path B: 1-2 months for integration with DAM and search.
• Path C: 1-3 months; one GPU, light operations; batch processing is easy to schedule.
• Path D: 3+ months incl. data licensing and annotation.
• KPIs: tag precision/recall against an expert-labelled sample, search success rate and time to find a track, separation quality (SDR or listening tests), note-level F-score for transcription, editing time saved per track, practice-app retention for learner variants.

People & change

Involve management (budget, rights policy) and the department lead (catalogue, studio or teaching) at the start. Musicians, producers and teachers are the end users: let a small pilot group test the tools on a few tracks and decide what counts as good enough. IT sets up accounts and hardware; legal or compliance clarifies who may process which recordings, since stems and transcriptions touch copyright and performer rights.

Works council: usually no. The system analyses music, not employee behaviour, so §87(1) no. 6 BetrVG is normally not triggered; still inform the council early under §90(1) no. 3 BetrVG if it changes work processes, and check usage logs. A DPO is only needed if recordings contain identifiable speech or personal voice data; then check the legal basis and an AVV with any cloud provider.

AI literacy: support staff with a short introduction on limits (tag errors, transcription mistakes, copyright of generated stems), in line with AI Act Art. 4. Communicate that output is a draft: editors correct tags and scores, and humans decide before anything is published. Typical concern is replacement of creative work; address it by positioning the tools as assistants for routine tagging.

Guides: GUIDE-008, GUIDE-006, GUIDE-009.

Risks, regulation & compliance

• EU AI Act: minimal risk for tagging, separation, transcription and learning feedback. Emotion recognition in the workplace or education is prohibited under Art. 5(1)(f): do not use voice or music "mood" analysis to infer learners' or employees' emotions. Synthetic audio output is subject to Art. 50 transparency; this concerns GEN-007 more than this use case. AI literacy (Art. 4) applies.
• Copyright (key risk): separating or transcribing protected recordings creates derivative material needing licences for commercial use. The text-and-data-mining exception in Art. 4 of Directive (EU) 2019/790 allows rightsholders to opt out of training; the AI Act (Art. 53) requires general-purpose model providers to respect it. Check the licence terms of pretrained models and training datasets (some are non-commercial).
• GDPR: mostly no personal data, but singing voices, user recordings and learner accounts in practice apps can be personal data; voice recordings must not be used for biometric identification without a legal basis (see SPEECH-010). Use data-processing agreements with API vendors.
• Mitigations: licence audit of models and data, human review of tags and scores, logging of source and rights status per stem, no emotion inference on people.

Getting started

1. Pick 100-200 representative tracks and have an expert label them (tags, a few transcriptions).
2. Run Essentia/CLAP for tags and Demucs plus Basic Pitch locally; compare against your labels.
3. Compare 1-2 commercial tools (Cyanite, LALAL.AI, Moises) on the same sample.
4. Put embeddings into a vector index and test search by example with real users.
5. Clarify licence status of recordings and models before any commercial use of stems.

Sources

1. Demucs (source separation) — https://github.com/adefossez/demucs
2. Basic Pitch (Spotify, audio-to-MIDI) — https://github.com/spotify/basic-pitch
3. Essentia (MTG, Universitat Pompeu Fabra) — https://essentia.upf.edu/
4. MERT music understanding model — https://huggingface.co/m-a-p/MERT-v1-330M
5. LAION CLAP audio-text model — https://github.com/LAION-AI/CLAP
6. MT3 multi-instrument transcription (Magenta) — https://github.com/magenta/mt3
7. SigSep, MUSDB18 and separation evaluation — https://sigsep.github.io/
8. Directive (EU) 2019/790 on copyright in the Digital Single Market — https://eur-lex.europa.eu/eli/dir/2019/790/oj
9. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
10. Cyanite (music tagging, Germany) — https://cyanite.ai/