Construction is one of the least digitised industries: progress is still judged by walking the site, and repetitive on-site tasks such as drilling ceilings or marking layouts are done by hand. Two AI-enabled families address this: (1) reality capture — people, drones or legged robots record the site, and computer vision compares it to the BIM model and schedule; (2) task robots that drill, mark, lay bricks or tie rebar. The first is deployable today with modest risk; the second is early-adopter and tightly bounded by site safety.
Problem & value
- Today: site managers photograph, walk and write reports; deviations from the plan are found late, when rework is expensive. Percent-complete drives payments and schedule forecasts but is often subjective. Skilled labour is scarce and overhead drilling is physically demanding.
- AI value: objective, frequent (weekly or daily) progress data per element, earlier detection of misplaced or missing work, fewer disputes about payment claims, documented as-built state for handover, and offloading ergonomically harmful tasks to robots.
- Evidence: vendors report productivity and rework savings, but figures come from vendor case studies and vary strongly by project; treat them as hypotheses to verify in a pilot rather than as benchmarks [1][2][3].
Variants
- Progress monitoring (capture + AI): helmet-mounted or handheld 360 cameras, fixed time-lapse cameras, drone photogrammetry, terrestrial laser scanners.
- Autonomous site walks: legged or wheeled robots following a repeatable route and capturing data (see ROBO-015 for the plant-inspection analogue).
- Layout and marking robots: drive on the slab, position against a total station and print layout lines from the BIM model.
- Drilling and installation robots: ceiling-drilling, anchor-setting and facade-element robots.
- Masonry, rebar and 3D printing: brick-laying systems, rebar-tying gantries/drones, additive concrete printing.
- Heavy-equipment autonomy: semi-autonomous excavators, compactors and dozers (adjacent to ROBO-022).
- Private use: none relevant; homeowner scans fall under VIS-040.
How it works
- BIM/IFC + schedule (4D)
- capture (360 video, drone, LiDAR, robot walk) with localisation to site coordinates
- reconstruction (SLAM / photogrammetry / Gaussian splatting
- point cloud or mesh)
- registration to BIM (alignment, element-level matching)
- AI recognition (segmentation/detection of walls, ducts, rebar, fixtures; installed vs. planned)
- progress & deviation engine (percent complete, tolerance checks, schedule impact)
- dashboard / issue tracker (human review, assignment, RFI)
- [optional] robot task planning from BIM
- execution with safety-rated controllers
- Perception: segmentation and open-vocabulary detectors identify building elements; depth and point-cloud processing give geometry; a VLM can summarise findings or answer "what is missing on level 3?".
- BIM comparison: the hard part is registration and element matching under occlusion, clutter and changing scenes, not detection alone.
- Robots: the AI handles localisation, perception and path planning; the motion and safety layer remains deterministic and certified. Drilling robots use the BIM to derive hole positions and verify with onboard sensors.
- Human-in-the-loop: site managers confirm progress and deviations before they affect payment or schedule; robot work zones are cleared and supervised, and operators hold an emergency stop.
Data preparation
Core data are the BIM/IFC model, the construction schedule (4D), 360-degree walks, LiDAR scans, drone images and site logs. In an SMB contractor these typically live in the planner's CAD/BIM tool, an MS Project or Excel schedule, file shares, e-mail and WhatsApp photos, and paper daily reports.
Typical gaps: BIM models that are outdated or lack consistent element IDs, schedules not linked to model elements, unsorted photos without location or date, scans without a common coordinate system (no control points), and unclear access rights for subcontractor data.
Preparation steps: (1) freeze one authoritative model version and link schedule tasks to element IDs; (2) define control points and a fixed capture route and rhythm; (3) store captures in one structured project space with date, zone and floor metadata (see PREP-009, PREP-010); (4) digitise paper daily reports (PREP-006, PREP-008); (5) blur faces and licence plates before sharing images (PREP-022). Use IFC viewers, a common data environment and the capture vendor's registration tools.
Rough share of effort: about 25-35 percent of the pilot, mostly model clean-up and capture routine. Check readiness with GUIDE-003.
Implementation paths
Which path for whom
- SMB: Path A — site-capture and progress-monitoring products (360-degree capture, drone services) are mature and rentable per project; task robots are best hired as a service.
- Enterprise: Path B — large contractors with their own BIM/4D data and many sites integrate vision APIs and capture feeds into their own platform to keep control of project data.
- Developer: Path C — open-source SLAM, point-cloud and detection stacks (e.g. Open3D, ROS 2) on own GPUs give full control for building tools on top of BIM/IFC data.
Path A — Buy (recommended for most SMBs)
- Progress monitoring: Buildots (helmet cameras + BIM comparison), Doxel (autonomous capture + progress tracking), OpenSpace (360 capture), Reconstruct, Disperse, Cupix, Avvir; EU/European-origin options include Scaled Robotics (Barcelona) and NavVis (Munich, SLAM scanners).
- Platforms with progress features: Autodesk Construction Cloud, Procore, Trimble Connect, Bentley iTwin, Dalux (Denmark), PlanRadar (Austria), Fieldwire.
- Robots: Hilti Jaibot (drilling) and Hilti Jobsite Robotics (Liechtenstein/Switzerland), Boston Dynamics Spot for site capture, Dusty Robotics and HP SitePrint (layout marking), Husqvarna and Okibo (finishing), Fastbrick/FBR (brick laying), Built Robotics (autonomous earthmoving), COBOD (Denmark, 3D concrete printing), PERI (Germany, 3D printing).
- Choose A when: a general contractor wants results on 1–3 pilot projects without building a data science team. Check: BIM/IFC import, EU data hosting, face/person blurring, API and export.
Path B — Build on cloud APIs
- Use drone/360 imagery with a cloud VLM (Claude, Gemini, GPT, Mistral) for daily-report drafting, photo captioning and safety observations; combine with point-cloud tools from Trimble, Esri or cloud photogrammetry (Pix4D, Switzerland; DJI Terra; RealityCapture/Epic).
- Compare to IFC using open libraries (IfcOpenShell) and orchestrate with a workflow engine. Choose B for custom reporting across many sites. Cloud VLMs do not replace geometric comparison against BIM.
Path C — Self-host (open source)
- Reconstruction: COLMAP, OpenSfM/OpenDroneMap, Nerfstudio and 3D Gaussian Splatting implementations, Open3D and PCL for point clouds; SLAM with ORB-SLAM3 or RTAB-Map.
- Perception: SAM 2/SAM 3-class segmenters, YOLO or RT-DETR detectors, open-vocabulary detectors (Grounding DINO); open VLMs such as Qwen-VL for reports.
- BIM: IfcOpenShell, xBIM, BlenderBIM/Bonsai; ROS 2 with Nav2 for mobile robots.
- Hardware: HW2 for reconstruction and inference; Jetson-class edge on the robot (HW1). Ops effort is high; fine for research groups and software-capable contractors.
Path D — Custom training
Worthwhile for element classes specific to your trade (MEP installations, formwork, rebar) or for unusual building types. Needs hundreds to thousands of labelled scans or images paired with BIM; weekly reviewer corrections are the training data. Simulation (ROBO-010) helps for robot skills but real sites are far more variable than lab settings.
State of the art (2026)
- Reality capture is mature; automated BIM comparison is early-adopter. Cheap 360 cameras, LiDAR phones and drones produce data quickly; commercial systems can track installation of many element types (drywall, ducts, sockets) but accuracy varies by element class and trade.
- Gaussian splatting and neural reconstruction speed up photorealistic site models (GEN-025); scan-to-BIM remains semi-manual (VIS-040).
- Legged and wheeled robots do autonomous repeat site walks; stairs, mud, dust and changing layouts remain failure sources.
- Task robots are commercial for narrow tasks (layout marking, overhead drilling, bricklaying); general-purpose humanoids are pilots only (ROBO-004). Vision-language-action models and demonstration learning (ROBO-003) are research for construction tasks.
- Limits: occlusion, reflective or dark surfaces, GPS-denied indoor localisation, weather, constantly changing scenes, and inconsistent BIM quality. A model that is "98% accurate on elements" can still be wrong on the ones that drive payment — always sample-verify.
Shared resources & combinations
- 3D vision stack: VIS-040 scan-to-BIM, GEN-025 photo-to-3D, VIS-003 drone infrastructure inspection use the same capture and reconstruction pipeline; one GPU server (HW2/HW3) can serve all.
- Planning: OPT-027 schedule optimisation consumes the progress data; DOC-030 quantity takeoff supplies planned quantities.
- Robotics: ROBO-006 drones, ROBO-015 inspection robots, ROBO-016 AI-based safety functions, ROBO-010 simulation.
Effort, cost & KPIs
- Path A: weeks to onboard a project; subscription per project or per area, plus capture hardware; robots are capex or rental (price depends on vendor and quote).
- Path B: 2–4 months for a reporting pipeline.
- Path C/D: 6+ months, specialised team.
- KPIs: progress accuracy against manual survey (sampled), time to detect a deviation, rework cost and share, schedule variance, capture coverage per week, robot task throughput and positional accuracy, near-misses, share of findings accepted by site managers.
People & change
Involve management for budget and the choice of pilot site, the department lead (site manager, BIM coordinator) as process owner, IT for data space and connectivity, and the DPO early, because images of workers count as personal data. Suppliers and subcontractors must agree to capture on their areas.
Works council: depends. Where a works council exists, regular capture of site areas where employees work can be suited to monitor behaviour or performance, so §87(1) no. 6 BetrVG applies on the objective-suitability test; §90(1) no. 3 requires timely information about AI-supported work processes. Agree on a works agreement: capture for progress only, no individual performance evaluation, blurring of persons.
DPIA is likely if people are identifiable; sign an AVV with the capture vendor. Support AI literacy (Art. 4 AI Act) with short training for site managers on reading deviation reports and their limits. Communicate openly why robots and cameras are used, start with one pilot site and a small team, and address fears of surveillance and job loss. Humans confirm every deviation before it triggers a claim or a schedule change.
Guides: GUIDE-004, GUIDE-005, GUIDE-008.
Risks, regulation & compliance
- EU AI Act: depends. Progress analytics is generally minimal risk. Robots are machinery: if AI acts as a safety component in machinery requiring third-party conformity assessment, the product is high-risk under Art. 6(1)/Annex I (obligations from 2 Aug 2028 per the Digital Omnibus, Reg. (EU) 2026/1744; Machinery Regulation (EU) 2023/1230, applying from January 2027, replacing the Machinery Directive 2006/42/EC). Art. 5 caution: do not use cameras for emotion inference of workers, and do not use progress data for worker scoring or biometric categorisation. Worker productivity ranking via AI in employment can fall under Annex III (employment/worker management) as high-risk.
- GDPR: 360 and drone footage captures workers, neighbours and passers-by. Needs a legal basis, DPIA for continuous monitoring, face/plate blurring at ingest, short retention, and works-council involvement (Germany: BetrVG §87(1) no. 6). Prefer capture of spaces without people.
- Safety: robots on site require risk assessment, CE marking, exclusion zones, and operator training; drones fall under EU UAS rules (Regulation (EU) 2019/947). Sensor-based person detection (ROBO-016) is a safety function, not a perception add-on.
- Contractual: AI progress figures used for payment (VOB/FIDIC) must be auditable and agreed in advance.
- Mitigations: human sign-off, audit logs, data minimisation, supplier due diligence on hosting, cyber security for site networks.
Getting started
- Pick one active project with a usable BIM model and define the measurable question (e.g. weekly percent complete for interior fit-out).
- Run a 6–8-week pilot with one Path A vendor and a manual baseline survey on the same areas.
- Agree privacy measures first: blurring, signage, retention, works-council agreement.
- Pilot one task robot (layout or drilling) in a cleared work zone with vendor safety assessment.
- Decide per KPI whether to scale, switch vendor, or build on open tools.
Sources
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Regulation (EU) 2023/1230 (Machinery Regulation) — https://eur-lex.europa.eu/eli/reg/2023/1230/oj
- Hilti Jobsite Robotics (Jaibot) — https://www.hilti.com/content/hilti/W1/US/en/business/business/trends/jaibot.html
- Buildots — https://www.buildots.com/
- Doxel — https://www.doxel.ai/
- Boston Dynamics Spot — https://bostondynamics.com/products/spot/
- IfcOpenShell — https://ifcopenshell.org/
- COLMAP — https://github.com/colmap/colmap
- Regulation (EU) 2019/947 (UAS operations) — https://eur-lex.europa.eu/eli/reg_impl/2019/947/oj