Municipalities maintain thousands of kilometres of roads, signs, lights and drains, but surveys are infrequent, expensive and subjective. Cameras on service vehicles, refuse trucks or buses plus detection models produce a continuous, geo-referenced condition picture: where the potholes and cracks are, how severe, and how fast they worsen. The result is data-driven maintenance prioritisation instead of reactive repair after complaints.
Problem & value
- Today: condition is assessed by periodic specialist measurement campaigns (in Germany the ZEB survey with measuring vehicles for the main network), manual walk-through inspections and citizen complaints. Municipal networks are rarely surveyed more often than every few years, defects are found late, and ratings differ between inspectors.
- AI value: cheap, frequent coverage of the whole network with ordinary cameras; consistent grading; early detection of deterioration when repair is still cheap; documented evidence for liability (Verkehrssicherungspflicht) and budget requests.
- Benchmarks: the public RDD2022 dataset (images from Japan, India, Czech Republic, Norway, US and China, with crack and pothole classes) is the standard yardstick; typical YOLO-family detectors reach mAP@50 roughly in the 0.6-0.7 range on it, i.e. good but not perfect [1]. No generally accepted ROI figure exists; measure your own baseline.
Variants
- Municipal road network: cameras on refuse trucks, street sweepers, buses or inspection cars.
- Motorway / state road operators: high-speed survey vehicles with LiDAR and calibrated cameras.
- Other public assets: traffic signs (missing, rotated, dirty), road markings, street lights, manholes and drain covers, guardrails, benches, playground equipment, graffiti, illegal dumping.
- Citizen reporting: photo submissions from "Mängelmelder"-style apps triaged and de-duplicated by a VLM.
- Cycling and footpath surveys; airport aprons and industrial yards (private operators).
- Private use: none beyond hobby mapping (OpenStreetMap/Mapillary contributions).
How it works
- camera + GPS/IMU on vehicle (or smartphone)
- frame sampling, privacy blurring (faces, plates) at the edge
- detection / segmentation of defects and assets (+ severity classes)
- geo-referencing, tracking across frames, de-duplication across passes
- aggregation per 10-100 m segment
- condition index / grade
- human review of low-confidence and high-severity items
- asset-management / GIS / work-order system (e.g. SAP, Cityworks-class, municipal GIS)
- repair prioritisation and trend dashboards
- Detector: YOLO-class or RT-DETR models trained on crack, pothole, patch, rutting and marking classes; segmentation for crack length/area; classifiers for sign state.
- Depth/3D: stereo, LiDAR or monocular depth estimates give pothole depth and rutting, which drive severity beyond what 2D images can show.
- VLM layer: open-vocabulary detection and VLMs handle the long tail ("leaning sign", "dumped mattress") and write human-readable defect descriptions; they are too slow for full-frame processing, so run them on flagged frames only.
- Human-in-the-loop: engineers validate samples, set severity thresholds and sign off the repair programme; AI proposes, the road authority decides.
Data preparation
Data are dashcam or smartphone video with GPS and IMU, optional LiDAR or 360-degree images, the road and asset register, the defect catalogue and earlier condition ratings (ZEB). In municipalities these sit in specialist software, GIS, file shares, paper plans and survey reports. Typical gaps: videos without reliable GPS time sync, asset registers not matched to the geometry, inconsistent damage descriptions, few labelled examples for local damage types, and personal data (faces, number plates) in images.
Preparation steps: (1) clean the asset register and align geometry in the GIS (DATA-003, PREP-005); (2) fix the defect catalogue to ZEB rules (PREP-019); (3) standardise capture (camera mount, speed, time sync); (4) label local examples with road engineers (DATA-023, PREP-018); (5) blur faces and plates automatically before storage (PREP-022, DOC-018 principle); (6) digitise old reports (PREP-006).
Tools: CVAT, QGIS, vendor tools. Rough share of total effort: 30-50 percent.
Implementation paths
Which path for whom
- Public — Path B: municipalities rarely run AI operations themselves; cloud APIs or EU-hosted services with an own condition-rating integration, procured under public-procurement rules, balance control and effort.
- Enterprise — Path B: road and network operators integrate detections into asset management and the ZEB rating through APIs and own pipelines.
- Developer — Path C: open-source road-damage models run on HW1 hardware and suit custom builds.
- SMB — Path A: small engineering offices and service providers buy a ready-made survey product with report output.
Path A - Buy (recommended for small engineering offices and service providers)
- Road-condition platforms: vialytic (DE, EU; smartphone on service vehicles, GDPR-conscious design), RoadBotics by Michelin (US), Roadway/Cyclomedia and Mapillary street imagery analytics (Mapillary: Meta-owned), Blyncsy, Vaisala RoadAI-class offerings, Cityflag-type citizen-reporting tools. Check: EU data hosting, anonymisation at the edge, export to your GIS/asset system, conformity with national rating schemes, per-km pricing.
- Choose A when: the goal is a network-wide inventory and ranking without ML staff.
Path B - Build on cloud APIs
- Frames to a multimodal API (Gemini, Claude, GPT, Mistral) with structured-output schemas for triage of citizen photos and long-tail asset checks; Azure AI Custom Vision / Vertex AI / SageMaker for training a managed detector; EU regions and DPA required.
- Orchestration with n8n or custom services; storage in PostGIS; maps in QGIS or a municipal GIS.
- Choose B for citizen-report triage and custom reporting, not for continuous full-video processing (cost).
Path C - Self-host (open source)
- Models: Ultralytics YOLO (AGPL, commercial licence available) or permissively licensed RT-DETR / RF-DETR; SAM 2 for segmentation-assisted labelling; Grounding DINO / OWLv2 for open-vocabulary assets; Qwen-VL class open VLMs for flagged frames; Depth Anything for monocular depth.
- Stack: ONNX/TensorRT on an NVIDIA Jetson Orin or similar on the vehicle, PostGIS, Label Studio or CVAT for labelling, OpenStreetMap/Mapillary for context.
- Hardware: HW1 (edge device in the vehicle) for inference; HW2 workstation for training and batch processing. Ops effort: moderate; the hard part is data governance and labelling.
Path D - Custom training
Worthwhile when national defect catalogues, local pavement types (cobblestones, concrete slabs, thin asphalt layers) or camera setups differ from public data. Start from a pretrained detector, fine-tune on a few thousand locally labelled images per class, and keep a fixed held-out region for evaluation. Public data: RDD2022, Crack500/CrackForest-style sets, Mapillary Vistas for signs.
State of the art (2026)
- Real-time YOLO/DETR detectors fine-tuned on RDD2022 are the baseline; newer lightweight variants target edge deployment [1][2]. Foundation models (SAM 2, open-vocabulary detectors, VLMs) cut labelling effort and help with rare defect types.
- Limits and failure modes: shadows, wet surfaces, patches and repairs confused with cracks; poor generalisation across countries and camera heights; low-severity cracks near the pixel resolution; severity (depth) is hard from 2D; cross-dataset accuracy drops notably; double counting across passes without good geo-referencing; GPS drift in urban canyons.
- Output is a screening indicator. Formal condition ratings for legal reporting may still require certified measurement methods; use AI to prioritise and to fill the gaps between campaigns.
Shared resources & combinations
- Same stack (object-detection + segmentation + edge-runtime): VIS-001 quality inspection, VIS-003 drone infrastructure inspection, VIS-012 traffic analytics, VIS-043 sewer CCTV.
- Data pipeline: VIS-030 anonymisation is a mandatory upstream step; VIS-021 and VIS-061 add satellite and InSAR context for subsidence; VIS-026 for measuring pothole volumes.
- Downstream: PRED-006 predictive maintenance models (deterioration forecasting, budget optimisation) consume the condition time series.
Effort, cost & KPIs
- Path A: weeks to pilot; subscription per km or per vehicle. A single pilot with 1-3 vehicles over one district is typical.
- Path B: 2-4 months for triage and reporting pipeline; API cost scales with images.
- Path C/D: 4-9 months incl. labelling, vehicle integration and validation; ongoing re-labelling as seasons and surfaces change.
- KPIs: recall of severe defects against a manual reference survey, false-alarm rate per km, km surveyed per month, time from detection to repair, share of network with data younger than 12 months, repair cost per km (early vs. late intervention), complaint backlog age.
People & change
Involve the head of the roads or civil engineering office as process owner, management or the mayor's office for budget and procurement, IT for GIS and storage, the DPO because public-road imagery contains people and vehicles, legal for procurement, and the staff council. Road crews and survey drivers are the end users; clerks maintaining the register are data owners.
Works council: depends. Survey vehicles with cameras may film employees, and driver or crew logs could allow performance evaluation; then §87(1) no. 6 BetrVG or the equivalent staff-council right (Personalrat, BPersVG and state laws) applies. Inform the council about the planned AI work process (§90(1) no. 3 BetrVG). A DPIA is likely advisable for systematic recording in public space; automatic blurring, short retention and AVV with every processor reduce the risk.
AI literacy: the AI Act (Art. 4) requires measures that support staff's AI literacy; short training for engineers on false positives and limits. Communicate to citizens and staff what is recorded and why. Pilot on one district, compare AI findings against a manual survey, and keep engineers in charge of repair priorities. See GUIDE-005, GUIDE-008 and GUIDE-009.
Risks, regulation & compliance
- EU AI Act: typically minimal risk. Annex III point 2 covers AI used as a safety component in managing and operating critical digital infrastructure and road traffic; offline condition surveys feeding maintenance planning are not safety components. Revisit if the system directly controls traffic or closes roads automatically. AI literacy (Art. 4) applies. No Art. 5 prohibited practice is involved unless the same cameras are repurposed for identifying people.
- GDPR: street video captures faces, number plates and bystanders. Blur at the edge before storage (VIS-030), minimise retention, use a documented legal basis (public task, Art. 6(1)(e)), run a DPIA for continuous mobile capture, and avoid secondary use (e.g. parking enforcement) without a separate legal basis. Works-council involvement if drivers' vehicles carry the cameras.
- Liability: documented detection of a defect creates knowledge; ensure a process to act on severe findings within statutory inspection duties.
- Road safety rules: Directive 2008/96/EC as amended by (EU) 2019/1936 requires network-wide road safety management for TEN-T and motorways; AI data can support, not replace, formal inspections [3].
- Mitigations: human sign-off, sampled audits, seasonal re-validation, vendor DPA and EU hosting.
Getting started
- Define the target: defect classes, severity levels and the rating scheme you must report in.
- Mount smartphones or dashcams on 1-2 service vehicles; capture a district and have an engineer label a reference sample.
- Pilot 2 Path A vendors against that reference and compare recall of severe defects.
- Complete the DPIA and enable edge anonymisation before any data leaves the vehicle.
- Connect output to the GIS/work-order system and review the first repair list with the maintenance team.
Sources
- Arya et al. - RDD2022: A multi-national image dataset for automatic Road Damage Detection - https://arxiv.org/abs/2209.08538
- Ultralytics YOLO documentation - https://docs.ultralytics.com/
- Directive (EU) 2019/1936 on road infrastructure safety management - https://eur-lex.europa.eu/eli/dir/2019/1936/oj
- Regulation (EU) 2024/1689 (AI Act) - https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- vialytic - https://www.vialytic.com/
- Mapillary - https://www.mapillary.com
- SAM 2 (Meta) - https://github.com/facebookresearch/sam2
- EDPB Guidelines 3/2019 on video devices - https://www.edpb.europa.eu/our-work-tools/our-documents/guidelines/guidelines-32019-processing-personal-data-through-video_en