Music & sound-effect generation

Small teams need music and sound for videos, ads, podcasts, apps, games and shops, but stock libraries are generic and commissions are slow and costly. Text-to-music and text-to-sound models now produce usable tracks, loops and effects in seconds. The hard part is no longer generation but rights: training-data provenance, licence terms, collecting-society (GEMA/SACEM) questions and AI-content labelling.

Problem & value

• Today: marketing and content teams license stock music (per-track or subscription), hire composers/sound designers, or search sound-effect libraries. Results are generic, licences are restrictive, and every new format (15-second cut-down, loop, stinger) costs time.
• AI value: many on-brief variations in minutes (length-matched cut-downs, mood variants, loops, stems), cheap sound-effect and ambience generation, prototype scoring before a composer is hired, adaptive game and app audio.
• Realistic scope: best for background music, jingles, hold music, ambience and Foley-style effects. Hero compositions, brand anthems and film scores still benefit from human composers, with AI used for sketches. No reliable public figures on time or cost savings are cited here; measure them in a pilot.

Variants

• Marketing & social video: background music matched to clip length and mood (see GEN-002, GEN-012).
• Podcasts, e-learning, corporate video: intro/outro, beds, sonic logo; combined with TTS (SPEECH-006).
• Games & apps: loops, adaptive stems, UI sounds, ambience (GEN-013).
• Film/TV/ad production: temp tracks and sound design for previs (GEN-021).
• Retail & hospitality: in-store or hold music with fully cleared licence.
• Private use: songs for gifts, family videos, hobby music; consumer tools, personal-use terms apply.

How it works

1. brief (mood, genre, BPM, length, brand rules; optional video/reference)
2. prompt building (LLM expands the brief into model-specific prompts)
3. generation (text-to-music / text-to-audio model, 2–4 candidates)
4. post-processing (loudness normalisation, loop points, stems, trim to picture, mastering)
5. human review (listening, similarity check, brand fit, legal/licence check)
6. labelling & metadata (AI-generated flag, provenance/watermark, licence record)
7. delivery to editor / DAW / CMS / game engine

• Models: latent diffusion or autoregressive transformers over audio codec tokens produce stereo audio at 44.1/48 kHz; modern systems add vocals with lyrics, section structure, stem separation and editing (extend, inpaint, remix).
• Control: text, lyrics, reference audio (style or melody conditioning), duration and sometimes tempo/key; video-to-audio models align effects to on-screen events.
• Human-in-the-loop: a person listens to every asset before publication, checks for recognisable melodies or voices, and records which tool, prompt and licence tier produced it.

Data preparation

Music and sound-effect generation needs little classic data; the key inputs are a clear audio brief (genre, mood, tempo, duration), brand sound guidelines, reference tracks and timing information from the video. In an SMB these are rarely documented: reference tracks sit in chat messages, brand jingles on an agency drive, video projects on file shares, licences in e-mail. Typical gaps are missing sound guidelines, no overview of which tracks are cleared for which use, unlabeled audio libraries and unclear rights in existing material.

Preparation steps: (1) write a short sonic brand brief with marketing; (2) build an inventory of existing audio assets with licence, source and permitted use (PREP-010 for metadata and tagging); (3) collect good reference examples (PREP-016); (4) define a usage policy covering which tools may generate commercial audio and how outputs are documented, including prompt and tool version; (5) check rights, since training data provenance differs between vendors. Tools: a simple spreadsheet or DAM, audio editors such as Audacity, file naming convention. Share of effort: about 10 to 15 percent. Also see GEN-005 for decks that use the audio.

Implementation paths

Which path for whom

• SMB: Path A, because licensed music generators with clear commercial terms exist at subscription prices and the need is occasional.
• Enterprise: Path B, because brand audio at scale needs API integration into production pipelines, usage tracking and contractual licence indemnities.
• Developer: Path C, because open music and audio models (MusicGen/Stable Audio Open class) run on HW2 and allow integration in apps without per-track fees.
• Private: Path A, because hosted apps or local tools cover hobby projects without setup.

Path A — Buy

• Prompt-to-song platforms: Suno and Udio (US; some major labels settled in late 2025 and moved the services towards licensed-model plans, while other label claims remain open, so check current terms; on 31 July 2026 the Munich Regional Court I largely ruled for GEMA against Suno, case 42 O 763/25, not final), ElevenLabs Music (cleared-licence positioning), Stability AI Stable Audio (commercial plans), Google Lyria in YouTube/Gemini tools.
• Library-style generators for creators: Soundraw, Mubert, Beatoven.ai, AIVA (Luxembourg, EU), Loudly. Often simpler licensing: royalty-free for commercial use on paid plans.
• Sound effects: ElevenLabs sound-effects generator, Stable Audio, plus classic libraries now adding AI search (Epidemic Sound, Artlist; check AI-generation features).
• Choose A when: you need quick background audio and can accept vendor licence terms. Verify: commercial-use grant on your plan, indemnity, training-data statement, content-ID claims on YouTube, export formats, EU data processing.

Path B — Build on cloud APIs

• APIs: ElevenLabs Music and Sound Effects APIs, Stability AI Stable Audio API, Google Lyria via Vertex AI (EU region available for many Google services; verify per model), Replicate/fal-hosted open models.
• Orchestration: an LLM turns briefs into prompts; n8n/Make/custom service calls the API, normalises loudness (ffmpeg, EBU R128), writes results and licence metadata to a DAM/CMS.
• Choose B when: you need batch generation (hundreds of cut-downs, per-market variants), integration into a video pipeline or app, or audit logs.

Path C — Self-host (open source)

• Open-weight models: Stable Audio Open (short clips and effects, community licence), Meta AudioCraft (MusicGen, AudioGen; code MIT, model weights non-commercial, so check), ACE-Step (open music foundation model), YuE (lyrics-to-song), plus Tencent/other open text-to-audio releases. Licences differ widely, and several weights are non-commercial.
• Stack: PyTorch/Diffusers, ComfyUI audio nodes, Demucs for stem separation, ffmpeg.
• Hardware: HW2 (one 16–24 GB GPU) runs most open models near real-time or faster; HW1 is slow but workable for short clips.
• Choose C when: data must stay on-prem, volumes are high, or you want a controllable royalty-free pipeline; accept lower quality than leading closed products.

Path D — Custom training

Fine-tuning (LoRA) on your own catalogue is viable for labels, game studios and agencies that own their recordings: it yields a house style and a clean rights position. Needs hundreds to thousands of hours of licensed, well-tagged audio and written consent from composers and performers. Never train on third-party catalogues without a licence.

State of the art (2026)

• Quality: leading systems produce full songs with vocals, structure and stereo mastering that many listeners cannot reliably tell from human production in short clips. Strengths: genre pastiche, background beds, loops. Weaknesses: long-range structure, fine editing, exact timing to picture, precise instrument control, lyrics intelligibility in some genres.
• Controllability: stems, section editing, extend/inpaint, reference-audio conditioning and video-to-audio sound effects are becoming standard features.
• Licensed models: after litigation by major labels and collecting societies against prompt-to-song services, the market moved to licensing deals and "cleared" training claims. Several lawsuits remain open or are on appeal (for example GEMA against Suno and OpenAI-related music cases in Germany, where first-instance courts have largely favoured GEMA); outcomes will shape the EU text-and-data-mining (TDM) rules in practice.
• Detection and provenance: C2PA content credentials, inaudible watermarks (for example SynthID in Google audio models) and streaming-platform AI-tagging policies are spreading, but detection is unreliable once audio is re-encoded.
• Failure modes: accidental resemblance to existing songs, imitation of artist voices or styles, artefacts and loop clicks, and inconsistent loudness. Do not trust vendor "royalty-free" claims without reading the licence.

Shared resources & combinations

• Same stack (music-gen, GPU HW2): GEN-013 game content, GEN-021 production temp tracks; audio post-processing shared with SPEECH-008.
• Combines with: GEN-002 video generation and GEN-012 automated editing (soundtrack per cut), GEN-003 avatar videos, SPEECH-006 narration and SPEECH-007 dubbing (mix music beneath voice), KNOW-016 multimodal archive search (find existing sounds first).
• Workflow engine and DAM shared with GEN-001/GEN-002 asset pipelines.

Effort, cost & KPIs

• Path A: days; subscription per seat or per credit. Cost driver: licence tier (commercial, broadcast, indemnity).
• Path B: 2–6 weeks for a batch pipeline with metadata and review; API cost per generated minute.
• Path C: 1–2 months; one GPU workstation plus engineering time; licence review per model.
• Path D: quarters; data licensing and consent management dominate.
• KPIs: accept rate of generated candidates, time from brief to approved track, licence incidents / content-ID claims (target zero), share of assets with complete provenance record, audience engagement of AI vs. stock audio in A/B tests, composer hours redeployed.

People & change

Involve management for the brand and budget decision, the marketing or video lead as process owner, and the creators who use the audio daily. Legal or compliance must clear licence terms, indemnities and collecting society (GEMA) questions; IT handles accounts and storage. Musicians or agencies who worked for you before should hear early how their role changes.

Works council: no in most cases, as the tool does not monitor employee behaviour; §87(1) no. 6 BetrVG would apply only with individual usage logging in a company-wide platform. §90(1) no. 3 BetrVG requires information when work processes change through AI, relevant if in-house audio staff exist. Data protection: usually no personal data; involve the DPO if voices of identifiable people are cloned or used as reference, since voice is personal data and consent is required, and sign an AVV (Art. 28 GDPR) with the vendor.

AI literacy: support staff competence under Art. 4 AI Act with a short session on prompting, licence terms and Art. 50 AI Act transparency duties for synthetic audio (GUIDE-006, GUIDE-014). Run a pilot with one campaign or video series and compare cost and quality with stock music. Communicate that AI covers background music and effects, while the final creative choice stays with a person who listens to and signs off each track. Typical concerns are copyright claims, loss of composer work and uniform sound; answer with documented licences, human selection and hybrid use. See GUIDE-008.

Risks, regulation & compliance

• EU AI Act: typically limited risk. Generating synthetic audio is subject to Art. 50: providers must mark outputs machine-readably, and deployers must disclose deepfakes. Pure background music is lower risk, but audio imitating a real person's voice is a deepfake. Art. 5 prohibited practices (e.g. manipulative or subliminal techniques) must not be used in sound design. General-purpose model providers owe copyright-policy and training-data summary obligations (Art. 53), which feed vendor due diligence. Art. 4 AI-literacy applies.
• Copyright: EU TDM exception (DSM Directive Art. 4) with opt-out; purely AI-generated output generally lacks copyright protection in the EU, so exclusivity cannot be assumed. Ask vendors for training-data statements and indemnities; avoid prompts naming artists or songs.
• Collecting societies: GEMA and others claim remuneration for AI training and outputs; check whether generated works must be registered or excluded from repertoire claims.
• GDPR: mostly no personal data, but voice samples, reference recordings of identifiable performers and user accounts are personal data; voice imitation requires consent (also personality rights). Use DPAs and EU processing where possible.
• Mitigations: written AI-audio policy, vendor licence archive, similarity checks before publication, label AI-generated content, no artist-name prompts, human approval.

Getting started

1. Define the use (ads, podcast, game) and classify risk: any real-person voice or style imitation is out of scope.
2. Compare 2–3 Path A tools on the same 10 briefs; read commercial-use, indemnity and training-data terms.
3. Set a review gate: listening check, similarity check, loudness (EBU R128) and licence record per asset.
4. Add labelling (AI-generated in credits, C2PA/watermark where available) for published content.
5. Scale via API (Path B) once the accept rate is stable; keep composers for hero pieces.

Sources

1. Regulation (EU) 2024/1689 (AI Act), Art. 5, 50, 53 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
2. Directive (EU) 2019/790 (DSM), Art. 4 text-and-data mining — https://eur-lex.europa.eu/eli/dir/2019/790/oj
3. European Commission — Code of practice on marking and labelling AI-generated content — https://digital-strategy.ec.europa.eu/en/policies/code-practice-ai-generated-content
4. ElevenLabs — Music and sound effects API documentation — https://elevenlabs.io/docs/overview/capabilities/music
5. Stability AI — Stable Audio — https://stability.ai/stable-audio
6. Stable Audio Open (open weights) — https://huggingface.co/stabilityai/stable-audio-open-1.0
7. Meta AudioCraft (MusicGen, AudioGen) — https://github.com/facebookresearch/audiocraft
8. ACE-Step open music foundation model — https://github.com/ace-step/ACE-Step
9. Google DeepMind — Lyria music generation — https://deepmind.google/models/lyria/
10. GEMA — AI and copyright (position and legal actions) — https://www.gema.de/en/
11. C2PA — Content credentials specification — https://c2pa.org/
12. Landgericht München I — press release on GEMA v. Suno, 42 O 763/25 (31 Jul 2026) — https://www.justiz.bayern.de/gerichte-und-behoerden/landgericht/muenchen-1/presse/2026/16.php