AI-assisted data labelling & annotation (model-in-the-loop)

Every custom vision, speech or language model needs labelled data, and manual labelling is slow, costly and inconsistent. In model-in-the-loop labelling, foundation models (SAM-class segmenters, vision-language models, LLMs) propose labels, active learning selects the items where a human adds the most value, and QA metrics keep quality measurable. The result is training and evaluation datasets produced in a fraction of the manual effort, with humans doing review instead of drawing every polygon.

Problem & value

• Today: teams draw boxes and polygons, tag text spans or transcribe audio by hand, or outsource it to annotation vendors. Guidelines drift, annotators disagree and rework is high. Labelling is often the largest single cost and the main schedule risk of a custom-model project.
• AI value: pre-labels turn "draw from scratch" into "accept or correct"; click-prompt segmentation replaces tracing outlines; LLMs label text classes from guidelines; uncertainty sampling prioritises what to label next; consensus and gold-set checks quantify quality.
• Benefit figures: the Segment Anything papers report interactive, promptable segmentation as the basis for such tools; vendor claims about labelling speed-ups vary by data type and should be verified on your own pilot batch rather than assumed [1][2].
• Second job: the same pipeline builds the evaluation (golden) sets that LLM and agent applications need for testing (see CODE-018).

Variants

• Computer vision: boxes, masks, keypoints, video object tracking with propagated masks (SAM 2 style) [1].
• Text/NLP: classification, NER, relation labels, preference/ranking data for LLM fine-tuning, LLM-as-annotator with human audit.
• Audio/speech: ASR pre-transcription (Whisper-class) corrected by humans, speaker turns, sound events.
• Document AI: layout and field labels on scans for extraction models (see DOC-014).
• Medical/industrial: expert-in-the-loop with DICOM or defect images; fewer but more expensive labels (see VIS-001).
• Managed workforce: vendor-provided annotators on top of the tooling.
• Private/hobby: small projects (wildlife camera, home-lab) with free CVAT or Label Studio.

How it works

1. raw data (images/video/text/audio)
2. ingestion + de-duplication + PII screening
3. guideline + label schema definition (ontology)
4. pre-labelling (SAM-class segmenter / open-vocabulary detector / VLM / LLM / ASR)
5. active learning: uncertainty / diversity sampling picks items for humans
6. human review & correction (annotation UI)
7. QA: gold-set checks, inter-annotator agreement, reviewer sampling
8. export versioned dataset
9. train / evaluate model
10. new predictions feed back to pre-labelling

• Pre-labelling: promptable segmentation (SAM 2) [1], open-vocabulary detectors (Grounding DINO, OWL-ViT-class), VLMs for captions and attributes, LLMs with guideline prompts for text classes.
• Active learning: the current task model scores unlabelled data; low-confidence or novel (embedding-distant) items are queued first, so each human minute improves the model most.
• Quality: agreement metrics (Cohen/Krippendorff), hidden gold items, review queues, per-annotator dashboards. LLM labels are validated against a human-labelled sample before being trusted in bulk.
• Human-in-the-loop points: guideline authoring, correction of low-confidence items, adjudication of disagreements, final dataset sign-off.

Data preparation

Labelling needs raw material (images, video, text, audio, documents), a label schema with annotation guidelines and a small seed set. In an SMB the material sits on machine and production PCs, camera SD cards, file shares, mail and the DMS; the labels often hide in existing records such as quality reports, tickets or ERP booking codes. Typical gaps: duplicates and near-duplicates, blurred or unrepresentative samples, no agreed categories, faces or personal data in images, and nobody who owns the guidelines.

Preparation steps: (1) inventory and collect the data (PREP-001, PREP-002); (2) de-duplicate, strip metadata, blur faces or pseudonymise (PREP-022); (3) define the category scheme with domain experts before pre-labelling (PREP-019); (4) harvest labels from existing business records (PREP-020); (5) build a small gold set with the experts (PREP-018, PREP-016); (6) feed corrections back (PREP-021). Tools: fdupes or imagehash, FFmpeg, Whisper for audio, Label Studio or CVAT.

Effort share is medium, roughly 30 percent: defining the schema and gold set takes longer than the tooling. See DATA-003, DATA-008 and DOC-014.

Implementation paths

Which path for whom

• Enterprise (B): pre-labelling via cloud APIs (VLMs, LLMs) plugs into existing ML pipelines and scales without GPU operations; use platform products (A) when many annotators and audit trails are needed.
• Developer (C): Label Studio, CVAT, SAM 2 and open-weight VLMs run on HW2 and give full control over datasets and formats.
• SMB (C): raw images, audio or documents are often production or customer data; Label Studio or CVAT with SAM 2 on a single GPU workstation is simple, cheap and keeps the data in-house, and the same setup serves several vision and document projects.
• Public (C): sovereignty and procurement rules favour self-hosted tools or an EU/German cloud with an AVV over US platforms.

Path A — Buy (off-the-shelf)

• Platforms: Labelbox, Encord, V7 (Darwin), SuperAnnotate, Scale AI, Roboflow (Annotate), Kili Technology (EU, France), Dataloop, Humans in the Loop, and cloud services such as Amazon SageMaker Ground Truth [3]. Prodigy (Explosion, Germany) suits NLP teams.
• Check: EU hosting or on-prem option, SSO, audit trail, SAM/foundation-model assistants, active-learning support, export formats, whether your data is used to train the vendor's models, and where outsourced annotators sit.
• Choose A when: several annotators, regulated data needs access control, and you want QA tooling without engineering effort.

Path B — Build on cloud APIs

• Pre-labelling by API: multimodal LLMs (Claude, Gemini, GPT, Mistral Pixtral/Mistral models as EU provider) with structured outputs for text and image attributes; cloud ASR for audio.
• Orchestration: Label Studio (open source or Enterprise) with an ML backend calling an API, or the vendor tool's model hooks; batch APIs reduce cost for bulk labelling.
• Choose B when: no sensitive imagery or personal data leaves the EU region, volumes are moderate and speed matters.

Path C — Self-host (open source)

• Tools: Label Studio (HumanSignal) [3], CVAT (originally Intel, now independent, supports SAM and auto-annotation), Doccano, Argilla (text/LLM feedback data), FiftyOne (dataset curation and error analysis), Cleanlab (label error detection).
• Models: SAM 2 / SAM 3 family (Meta, open weights) [1], Grounding DINO, YOLO-family detectors for bootstrap labels, Whisper or faster-whisper for audio, open-weight VLMs (Qwen-VL family) and LLMs (Llama, Mistral, Qwen) served via vLLM.
• Hardware: HW2 (one 24 GB GPU) is enough for interactive SAM-assisted annotation and a 7-8B VLM; HW3 for video propagation at scale and larger VLMs.
• Ops: Postgres, object storage, SSO and backup; roughly part-time DevOps.

Path D — Custom training

Train the task model itself on the labelled data (the purpose of the pipeline) and optionally a lightweight domain-specific pre-labeller from the first few thousand reviewed labels. Fine-tune only when generic foundation models mislabel a specialised domain (medical, industrial defects); a few hundred expert-reviewed examples often suffice to start.

State of the art (2026)

• Promptable segmentation is a commodity: SAM 2 added video support with memory-based mask propagation [1]; later releases (SAM 3) add text and concept prompts, and nearly every labelling platform integrates a SAM variant.
• LLM/VLM annotators: LLM labelling of text rivals crowd workers on simple tasks, but quality varies with task subjectivity; best practice is calibration against a human-labelled gold sample, plus periodic audits [4].
• Label-error detection (confident learning, as in Cleanlab) finds mislabelled items in existing datasets, often giving larger gains than adding more data [5].
• Preference and evaluation data for LLMs (rankings, rubric grades, red-team cases) is now a major labelling workload, with expert annotators.
• Limits: pre-labels bias reviewers (automation bias: humans accept wrong suggestions); models labelling data they later train on reinforce their own errors (confirmation loop); SAM-class masks degrade on thin, transparent, low-contrast or domain-specific objects (medical, microscopy, X-ray); LLM labels can be systematically wrong yet consistent, so agreement alone does not prove correctness.

Shared resources & combinations

• Same stack: VLM + segmentation + vector index as in VIS-001 (inspection); ASR as in SPEECH-001; OCR/layout as in DOC-014.
• Upstream: DATA-003 (cleaning and de-duplication before labelling), DATA-005 (synthetic data reduces the labelling volume needed; synthetic data still needs validation).
• Downstream: model training (path D of many use cases), DATA-008 (data quality monitoring and drift trigger re-labelling), CODE-018 (evaluation sets for LLM apps).
• Shared GPU server for SAM, VLM and ASR serves interactive annotation plus overnight batch pre-labelling.

Effort, cost & KPIs

• Path A: days to weeks to onboard; pricing typically per seat, data volume or labelled unit; managed workforce adds per-label fees.
• Path B: 2-6 weeks for a pre-labelling integration; API cost per item usually small compared with human minutes.
• Path C: 1-3 months incl. SSO, storage and model serving; GPU server plus operations time.
• KPIs: human seconds per accepted label, pre-label acceptance rate, inter-annotator agreement, gold-set accuracy, label error rate found in audits, model performance gain per 1,000 labels, cost per labelled item, turnaround time.

People & change

Involve the department lead (quality, R&D, data) who owns the label schema, and the domain experts as end users: their corrections are the quality. IT provides the GPU workstation or server, accounts and storage; data owners release the material. If images, audio or text contain people, the DPO checks legal basis and purpose of reuse, a DPIA is likely for video of employees or special categories, and every external platform needs an AVV (Art. 28 GDPR).

Works council: depends. Annotation tools usually log per-annotator speed and agreement, which makes them suited to monitor performance (§87(1) no. 6 BetrVG, objective suitability suffices). Use anonymous or team-level dashboards and agree this in a works agreement; data from camera or audio recordings of staff is sensitive in itself. Inform under §90(1) no. 3; in public bodies involve the staff council.

AI literacy: train annotators on the guidelines and on automation bias, because accepting pre-labels blindly lowers quality (GUIDE-006). Pilot with one label class and two reviewers, measure agreement, then expand. Concern "we label ourselves out of a job": show that expert review stays central. See GUIDE-008, GUIDE-004 and GUIDE-005.

Risks, regulation & compliance

• EU AI Act: the labelling tool is minimal risk. The datasets may feed high-risk systems, where Art. 10 requires documented data governance, relevance, representativeness and bias examination for training, validation and test data [6]. Keep dataset lineage, guidelines and QA records. Labelling for prohibited practices (Art. 5, e.g. emotion recognition at work or school, untargeted facial-image scraping) must not be supported.
• GDPR: images, voice and text often contain personal data; define lawful basis and purpose, minimise (blur faces or plates where not needed), DPIA for sensitive data (health, biometrics), processor agreements and transfer safeguards with annotation vendors, especially for offshore workforces [7].
• Workforce ethics: fair pay and psychological protection for annotators handling disturbing content; check vendor labour practices.
• Copyright: scraped data requires text-and-data-mining opt-out checks (DSM Directive Art. 4).
• Mitigations: blind review of a share of items (no pre-label shown), gold sets, versioned datasets, access control, retention limits, bias audits of label distribution.

Getting started

1. Define the task, label schema and written guidelines with examples; have two people label 100 items and measure agreement.
2. Set up Label Studio or CVAT (or trial a Path A platform) with a SAM or LLM-based pre-labeller.
3. Pilot 500-1,000 items: measure human time with and without pre-labels, and audit a blind sample.
4. Train a first model, then enable active learning to choose the next batch.
5. Version every dataset, document lineage and QA results, and schedule re-labelling when drift is detected.

Sources

1. Ravi et al., "SAM 2: Segment Anything in Images and Videos" (Meta) — https://arxiv.org/abs/2408.00714 — and https://github.com/facebookresearch/sam2
2. Kirillov et al., "Segment Anything" — https://arxiv.org/abs/2304.02643
3. Label Studio documentation (ML backend, pre-annotations, active learning) — https://labelstud.io/guide/ml
4. Gilardi et al., "ChatGPT outperforms crowd workers for text-annotation tasks" (PNAS 2023) — https://arxiv.org/abs/2303.15056
5. Northcutt et al., "Confident Learning: Estimating Uncertainty in Dataset Labels" — https://arxiv.org/abs/1911.00068
6. Regulation (EU) 2024/1689 (AI Act), Art. 5 and Art. 10 — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
7. Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj
8. CVAT documentation — https://docs.cvat.ai/
9. Amazon SageMaker Ground Truth — https://docs.aws.amazon.com/sagemaker/latest/dg/sms.html