Technical drawing & engineering document understanding (P&ID, CAD drawings)

Decades of engineering knowledge sit in 2D drawings: manufacturing drawings with tolerances, and P&IDs that describe how a plant is plumbed. Most exist only as PDFs or scans, so quoting, reuse, maintenance planning and plant changes need people to read them by eye. Vision models, detectors and graph methods now extract title blocks, dimensions, GD&T and P&ID connectivity into structured data. Accuracy is good enough to accelerate review, not yet to replace engineering sign-off.

Problem & value

• Today: estimators re-key dimensions from customer drawings into quotes; plant owners hold thousands of legacy P&IDs as raster PDFs whose tag lists do not match the asset register; engineers search drawings by file name. Digitising a P&ID manually (redrawing in an intelligent tool) is slow and expensive, which is why many brownfield plants never do it.
• AI value: faster quoting and incoming inspection planning, searchable drawing archives, tag/asset reconciliation, input for digital twins and maintenance systems, fewer misreadings.
• Measured evidence (academic, not field ROI): a YOLOv11 + Donut pipeline on manufacturing drawings reported 94.77 % precision on GD&T and a 5.23 % hallucination rate [1]; a Relationformer-based method for P&IDs improved edge (connection) detection accuracy by more than 25 % over a modular baseline [2]. Vendor claims on time saved are rarely independently published; measure on your own drawings.

Variants

• Manufacturing drawings (SMB job shops, OEMs): title block, material, quantity, dimensions, tolerances, surface finish, GD&T for quoting (see PRED-032, AGENT-027), CAM prep (SCI-023) and incoming-inspection plans.
• Process industry P&IDs: symbol + line detection, tag OCR, graph of equipment/instruments/valves, export towards DEXPI, the neutral P&ID exchange format [3]; used for brownfield digital twins, HAZOP preparation, MoC (management of change).
• Electrical/instrumentation schematics, isometrics, single-line diagrams: same pipeline with other symbol libraries.
• Construction plans: floor plans and quantity takeoff are covered by DOC-030; rule checks of plans by DOC-016; 3D scan-to-BIM by VIS-040.
• Engineering documents (text-heavy): datasheets, specifications, vendor documents, RFQ drawings packages — question answering over them combines this use case with KNOW-013 / DOC-014.
• Private use: marginal (e.g. reading a house plan or appliance wiring diagram with a chat VLM).

How it works

1. intake (PDF/TIFF/DWG export; multi-sheet sets)
2. pre-processing (de-skew, tiling of large sheets, vectorised-PDF text layer if present)
3. region detection (title block, views, dimension callouts, symbols, tables, notes)
4. recognition (OCR of tags/dimensions, GD&T frames, symbol classes, line tracing)
5. structuring (JSON schema | graph of nodes & edges for P&IDs)
6. validation (plausibility: tolerance ranges, tag syntax, unit consistency, cross-check against ERP/asset register, revision comparison)
7. human review (side-by-side with highlighted source region, corrections logged)
8. export (ERP/PLM/CMMS, DEXPI/graph DB, search index)

• Detectors + parsers: oriented-box object detectors (YOLO family) localise callouts and symbols; a document transformer or VLM turns each crop into structured text [1]. Crops keep resolution high, which matters because whole-sheet VLM inputs are downscaled.
• Graph transformers for P&IDs extract symbols and connections jointly rather than in separate steps [2].
• VLMs / LLMs answer questions, normalise results and map them to schemas; they hallucinate numbers when text is tiny, so numeric outputs must be verified by OCR or vector-PDF text.
• Vector PDFs (exported from CAD) contain exact text and geometry — parse these first, use vision only for scans.
• Human-in-the-loop: engineering review of safety-relevant data is mandatory; low-confidence regions are flagged.

Data preparation

Drawings typically live in PLM/PDM, on file shares in folder-per-order structures, in ERP attachments and, for older parts, as paper in plan chests. Typical gaps: low-resolution or skewed scans, several revisions of the same drawing side by side, inconsistent title blocks, missing tag lists, symbols that deviate from the standard, and CAD originals that nobody can open any more. Access rights are often coarse: everyone sees everything, including customer-owned drawings under NDA.

Preparation steps: (1) inventory drawings and pick 50-200 representative sheets per drawing type; (2) scan paper at 300-400 dpi and clean it (deskew, denoise), see PREP-006; (3) deduplicate revisions and mark the valid one; (4) export vector PDF/DXF where available, because it beats raster; (5) build a small gold set with engineers who mark title block, dimensions and tags, see PREP-018 and PREP-016; (6) align tag lists with the asset register, see PREP-011. Tools: Paperless-ngx or the DMS for storage, OCR and layout parsing as in DOC-014. Expect 40-60 % of the total effort in data preparation, mostly expert time for the gold set.

Implementation paths

Which path for whom

• SMB: Path A — a specialised drawing-extraction product avoids building a vision pipeline; check that drawings (IP) may be hosted in the EU under an AVV.
• Enterprise: Path C — drawings are core know-how and volumes are high, so a self-hosted VLM/detector stack on HW3 under own control pays off.
• Developer: Path C — open detectors and VLMs allow tuning to symbol libraries and DEXPI export, which products rarely expose.

Path A — Buy (recommended start)

• Drawing extraction specialists: Werk24 (Germany, API for extracting title-block, dimension, GD&T and material data from manufacturing drawings); quoting platforms with built-in drawing reading (e.g. Xometry, Hubs-type marketplaces, and ERP/CPQ add-ons for job shops).
• Industrial data platforms: Cognite Data Fusion (Norway) offers diagram parsing to link P&ID tags to assets; AVEVA, Hexagon (Sweden), Siemens, Bentley and Autodesk ship P&ID/diagram conversion and data-extraction features in their engineering suites. Specialist P&ID digitisation services are also offered by many engineering contractors.
• Check: EU hosting or on-prem option, export to DEXPI/CAD/PLM, handling of your sheet sizes, confidentiality terms for customer drawings (NDAs), evaluation on your own drawings.
• Choose A when: drawings are fairly standard and the integration target (ERP/CPQ/CMMS) is supported.

Path B — Build on cloud APIs

• Document AI: Azure Document Intelligence (layout/read models and custom extraction models; EU regions) [4], Google Document AI, AWS Textract for OCR/layout; Mistral OCR (EU provider).
• Frontier multimodal models (Claude, Gemini, GPT, Mistral) with structured outputs for tile-by-tile reading and question answering; combine with your own detector for symbol localisation.
• Orchestration: tiling and crop service, JSON-schema validation, rule checks, review UI, integration via n8n/Power Automate or custom services.
• Choose B when: you need custom schemas/validation (e.g. company-specific tag standards) and data may go to an EU cloud region under a DPA/NDA.

Path C — Self-host (open source)

• Parsing/VLMs: Docling (MIT, includes the 258M-parameter Granite-Docling VLM) for text/table documents such as specifications and datasheets [5]; Qwen3-VL (Apache-2.0; 2B–235B sizes, OCR in 32 languages) for drawing question answering [6]; PaddleOCR for tag/dimension OCR.
• Detection: YOLO-family or DETR-type detectors fine-tuned on your symbol set; Relationformer-style graph models and the public P&ID benchmark dataset from [2] as a starting point.
• Stack: vLLM/Ollama, a graph DB (Neo4j, Memgraph) for P&ID graphs, OpenSearch/pgvector for search.
• Hardware: HW3 (48 GB+ GPU) for a 32B-class VLM at reasonable sheet-level quality; HW2 suffices for detector + OCR + an 8B VLM on crops. Confidential customer drawings and plant designs are a strong reason for on-prem.

Path D — Custom training

Usually required for the symbol/line part: company- or standard-specific P&ID symbols (ISO 10628, ISA 5.1, company legends) and drawing conventions (DIN ISO 128 / ISO 1101 / ASME Y14.5) differ from public data. Needs a few hundred annotated sheets (boxes, classes, connections) per symbol family; reviewer corrections provide ongoing labels. Fine-tuning a document transformer on cropped regions is cheap compared with training whole-sheet models [1].

State of the art (2026)

• Hybrid pipelines beat end-to-end VLMs on drawings: detection of regions followed by crop-level parsing gives higher precision than prompting a general model with a whole sheet [1].
• P&ID digitisation moved from separate symbol/line steps to joint graph prediction with transformers, and the first public P&ID benchmark with graph ground truth exists [2]; standardised outputs (DEXPI 2.x) make results exchangeable between tools [3].
• General VLMs improved OCR and layout reading (multilingual, rotated and low-quality text) [6], but remain unreliable at counting, tracing long lines across a sheet, and reading dense tolerance stacks.
• Failure modes: tiny fonts, overlapping lines, hand annotations, revision clouds, multi-sheet references (off-page connectors), unusual symbol libraries, hallucinated numbers. Treat outputs as proposals with confidence scores.
• Not solved: full automatic 3D reconstruction from 2D views and automatic safety verification of P&IDs; keep these under expert control.

Shared resources & combinations

• Same stack (ocr-docparse + vlm + llm-general): DOC-014 (PDF parsing for RAG), DOC-016 (rule checks of plans), DOC-019 (certificate verification), DOC-030 (construction takeoff) — one VLM/OCR server on HW3 can serve all.
• Detector + GPU: shares object-detection tooling with VIS-040 (scan-to-BIM) and visual inspection.
• Downstream: PRED-032 (should-cost/instant quoting), AGENT-027 (RFQ-to-quote), SCI-023 (CAM prep), SCI-024 (drawing automation), KNOW-020 (similar-part search; extracted features improve ranking), KNOW-013 (knowledge graph of plant assets), PRED-006 (predictive maintenance linked to asset tags).
• Data pipeline: tag/part master data from ERP/PLM/CMMS is the validation reference for several use cases.

Effort, cost & KPIs

• Path A: weeks for a pilot; typically priced per drawing/page or per project (check vendors).
• Path B: 2–4 months for a validated pipeline with review UI; API cost per sheet is low relative to engineer time, but tiling raises token use.
• Path C: 3–6 months incl. GPU server, detector training and operations.
• Path D: adds annotation effort (weeks of engineer time) and MLOps.
• KPIs: field-level precision/recall by category (dimensions, GD&T, tags), connection accuracy for P&IDs, share of drawings processed without correction, review minutes per sheet, quote turnaround time, tag-register match rate.

People & change

Involve early: management (budget, IP policy), the engineering department lead as process owner, and data owners of PLM/file shares; IT and IT security for hosting and drawing confidentiality (customer NDAs, export control). End users - design engineers, estimators, maintenance planners - should shape the gold set and review output from the first pilot.

Works council: usually no co-determination under §87(1) no. 6 BetrVG, because the system analyses documents, not employee behaviour; still inform it under §90(1) no. 3 if work processes change, and check whether usage logs name individual users - then it becomes relevant. Data protection: drawings rarely hold personal data (author names in title blocks aside); an AVV with any cloud provider is still needed.

AI literacy: support staff with short training on what extraction can and cannot do, see GUIDE-006. Pilot: one drawing type with 2-3 engineers, measure error rate against the gold set, then widen. Concerns: "will it replace drafters?" - position it as read-assist; engineers sign off every extraction that feeds quoting or plant changes. Keep source-region highlighting visible so errors are easy to spot. See GUIDE-008 and GUIDE-009.

Risks, regulation & compliance

• EU AI Act: typically minimal risk — extraction for engineering productivity is not an Annex III use. It becomes relevant if outputs feed a safety component of critical infrastructure (Annex III point 2, e.g. water, gas, heating, electricity supply) or a product safety function under Annex I law (e.g. Machinery Regulation) [7]; keep AI as a proposal tool with documented engineering sign-off. AI literacy (Art. 4) applies.
• GDPR: drawings rarely hold personal data; title blocks and revision tables may name designers or customer contacts. Depends on content.
• Confidentiality / IP: drawings are trade secrets and may be export-controlled (dual-use items); prefer EU regions or on-prem, NDAs with vendors, no training on your data.
• Safety/quality: wrong tolerances or missing lines cause scrap or hazards; require verification on safety-critical data (HAZOP, MoC), versioned audit logs and reference to the source sheet.
• Standards: DIN/ISO drawing standards, ISA 5.1, IEC 81346 designation; archive per company document control (ISO 9001).

Getting started

1. Collect 50–100 representative drawings (clean, scanned, hand-marked) and define the target schema.
2. Identify which are vector PDFs (parse text directly) and which are scans.
3. Benchmark one Path A vendor and one VLM approach on this set; score fields against a manual ground truth.
4. Add validation against ERP/asset register and a side-by-side review UI.
5. Start with a low-risk process (quote pre-filling, tag reconciliation) before safety-relevant use.

Sources

1. Khan et al. — Automated Parsing of Engineering Drawings for Structured Information Extraction Using a Fine-tuned Document Understanding Transformer (arXiv:2505.01530) — https://arxiv.org/abs/2505.01530
2. Stürmer, Graumann, Koch — From Engineering Diagrams to Graphs: Digitizing P&IDs with Transformers (arXiv:2411.13929) — https://arxiv.org/abs/2411.13929
3. DEXPI e.V. — Data Exchange in the Process Industry (P&ID exchange standard) — https://www.dexpi.org/
4. Microsoft — Azure Document Intelligence overview — https://learn.microsoft.com/en-us/azure/ai-services/document-intelligence/overview
5. Docling project (MIT license, Granite-Docling support) — https://github.com/docling-project/docling
6. Qwen3-VL repository (Apache-2.0, model sizes, OCR) — https://github.com/QwenLM/Qwen3-VL
7. EU AI Act Annex III — https://artificialintelligenceact.eu/annex/3/
8. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj