Every new customer, supplier or subsidiary sends data in its own layout: different column names, units, date formats, code lists and EDI dialects. Mapping these to your schema is slow expert work that repeats with each partner. AI proposes the mapping, generates and tests the transformation, flags bad rows and remembers every correction, so onboarding drops from a project to a review task.
Problem & value
- Today: SaaS vendors build one-off CSV importers and support staff fix files by hand. B2B integration teams hand-craft EDI maps per trading partner from PDF implementation guides. Catalogue and price-list intake runs on Excel macros and ETL jobs that break when a supplier adds a column.
- AI value: semantic matching of headers and sample values (not just names), draft transformation code, automatic validation against the target schema, and a mapping memory that gets better with each partner. Humans approve instead of build.
- Delimitation: one-off migrations and UI-driven entry are AGENT-021; generating connector code is CODE-012; record deduplication and golden records are DATA-003. This case is the recurring partner-facing mapping and validation layer, usually embedded in a product or an integration platform.
Variants
- SaaS product import: end customers upload CSV/Excel into your product; an embedded importer maps and validates in-session.
- B2B/EDI partner onboarding: retailers, carriers and manufacturers map orders, ASN/despatch advice, invoices (X12, EDIFACT, VDA, Odette, Peppol UBL) per partner.
- Catalogue and price-list intake: supplier product files into a PIM/ERP, combined with classification (DATA-010).
How it works
- partner file / EDI message / API payload
- parse + profile (types, formats, distributions, delimiters, encoding)
- candidate matching (header + sample-value embeddings, LLM judgement, past mappings, standard dictionaries)
- transformation draft (rename, split/merge, unit and date normalisation, code-list lookups, derived fields as code/JSONata/XSLT)
- validation (schema, business rules, referential checks, sample dry-run)
- human review (side-by-side, confidence, diff to last version)
- publish partner mapping (versioned)
- production runs with monitoring
- corrections feed the mapping memory
- Matching: combine deterministic signals (exact names, synonyms, data types, regex profiles) with embeddings and an LLM that sees header, a few masked sample values and the target field descriptions. Output is a ranked proposal with confidence, never a silent write.
- Transformation: the LLM writes the rule in a constrained, testable language (JSONata, SQL, a mapping DSL) and runs it on samples. Generated rules are tested, not trusted.
- EDI: parse X12/EDIFACT into JSON (a deterministic parser), then map JSON to your canonical model with LLM assistance grounded in the partner implementation guide (RAG over PDFs, see DOC-014).
- Human-in-the-loop: low-confidence fields, any rule that changes amounts, quantities or units, and the first production run per partner need a person. Approved mappings run deterministically afterwards, without an LLM in the loop.
Data preparation
The inputs are partner files (CSV, Excel, XML, JSON) and EDI messages that arrive by mail, SFTP, portals or as attachments, plus your target schema from the ERP, PIM or database. Typical gaps: undocumented columns, mixed encodings and decimal formats, units and code lists that differ per partner, outdated implementation guides as PDFs, and earlier mappings that exist only in someone's head or in a script.
Preparation steps: (1) document the target schema with field descriptions, units, allowed values and examples (PREP-012); (2) collect a handful of real files per partner plus the approved result as gold examples (PREP-016, PREP-020); (3) write down code lists, units and synonyms (PREP-015); (4) convert implementation guides to searchable text (PREP-006, DOC-014); (5) mask sample values before sending them to an external model (PREP-022); (6) define who approves a mapping (PREP-004). Tools: Python with pandas, DuckDB or OpenRefine for profiling; an EDI parser for X12 and EDIFACT.
Effort share is moderate, roughly 30-40 percent: the target schema and the first approved mappings matter most, later partners get cheaper. See DATA-003, DATA-007 and DATA-008 for related cleaning and conversion steps.
Implementation paths
Which path for whom
- SMB (A): common formats and existing connectors are covered by EU EDI providers (Lobster, SEEBURGER, Babelway) or embedded importers at per-partner prices; building a mapping engine does not pay off.
- Enterprise (B): many partners, own canonical models and ERP landscapes call for an in-house service on cloud APIs (or the existing iPaaS) with a versioned mapping memory.
- Developer (C): open-weight models and a deterministic parser run on HW2 and keep partner files in your own environment, which suits product vendors that embed an importer.
Path A — Buy (recommended for SMBs)
- Embedded importers: Flatfile, OneSchema, CSVBox, Dromo (several with EU hosting options; check region and sub-processors). Fit when customers upload files into your SaaS.
- B2B / EDI: Stedi (API-first EDI with mapping tooling and AI-assisted mapping), Orderful, SPS Commerce, plus EU-rooted providers such as Lobster (DE), SEEBURGER (DE), Cleo, Babelway (BE) and OpenText B2B. Fit when onboarding many trading partners.
- Enterprise iPaaS with ML mapping: SAP Integration Suite (Integration Advisor with ML-assisted mapping proposals, SAP is EU), Boomi AI, Informatica CLAIRE, MuleSoft, Azure Logic Apps with Copilot assistance.
- Choose A when: formats are common and a connector exists; build effort should go into product, not mapping engines.
Path B — Build on cloud APIs
- Frontier LLMs with structured outputs (Claude, Gemini, GPT, Mistral in EU regions or from an EU provider) produce the mapping as JSON against your schema; embedding models (for example Mistral Embed, text-embedding models in EU regions) rank candidate fields.
- Orchestrate with n8n, Temporal or your backend; keep mappings as versioned config in Git or a database; run transformations with deterministic engines (JSONata, DuckDB SQL, dbt).
- Choose B when the importer is part of your product differentiation or formats are unusual.
Path C — Self-host (open source)
- Open-weight models for matching and rule drafting: Qwen3 family, Llama, Mistral Small / Devstral, served via vLLM or Ollama; embeddings such as BGE-M3 or multilingual-e5.
- Schema-matching and ETL tooling: Valentine (schema matching experiments), Airbyte, dbt, DuckDB, Apache Camel / Apache NiFi, Smooks and open EDI parsers (for example EDIReader, Bots, pyx12).
- Hardware: HW1 for matching with small models and parsers; HW2 for a 14–30B model that drafts transformation code.
Path D — Custom training
Worth it only for a single vertical with thousands of historical, approved mappings (for example a logistics platform with many carrier formats). Fine-tune a small model on (source profile, target schema, approved mapping) triples; reviewer corrections are the training data. Start with few-shot retrieval of past mappings, which usually gets most of the gain.
State of the art (2026)
- LLM schema matching is clearly better than name-based matchers on semantic headers and multilingual files, and academic work on LLM-based schema and entity matching is active, but accuracy falls on cryptic column names, codes without descriptions, and multi-column derivations. Treat vendor accuracy claims as marketing until measured on your data.
- Learning from corrections (mapping memory, few-shot retrieval of earlier partners) is the main practical differentiator versus one-shot matching.
- Code-generating mappings (the model writes the rule, a test runs it) are more reliable than letting a model transform rows directly, and are cheaper at volume.
- EDI is shifting to API/JSON-based tooling with AI drafting maps from implementation guides, yet complex partner-specific rules (conditional segments, qualifier semantics) still need an EDI expert's review.
- Failure modes: plausible but wrong mappings (day/month swaps, unit confusion, currency or decimal separators), silent truncation, drift when a partner changes the layout without notice, and prompt injection via cell contents in uploaded files.
Shared resources & combinations
- Shared stack: llm-general + embedding-text + workflow-automation with DATA-003 (matching and golden records), DATA-007 (catalogue and glossary used as target field descriptions), DATA-010 (product classification during catalogue intake) and CODE-012 (connector generation).
- Downstream: DATA-008 monitors the incoming feeds for schema drift and quality; DOC-006 supplies structured logistics documents that arrive as paper or PDF rather than EDI; DATA-015 uses cleaned supplier data for spend analytics.
Effort, cost & KPIs
- Path A: days to weeks; priced per importer, per message or per partner connection.
- Path B: 1–3 months for matching, review UI, validation and monitoring; LLM cost is small because only headers and samples are sent.
- Path C: 2–4 months plus operations.
- KPIs: time from first file to production per partner, share of fields mapped without correction, validation error rate per run, rows rejected or manually fixed, support tickets per import, mapping regressions after partner changes.
People & change
Involve the department lead (logistics, purchasing or customer onboarding) as process owner, and the end users who today map files by hand: they review proposals and know the partner quirks. IT and IT security handle SFTP, accounts, validation rules and the question which fields may leave the company. Data owners of the target system approve the schema. Suppliers and customers should be told when formats or test runs change.
Works council: usually not required, because the tool maps business data and does not monitor employees (§87(1) no. 6 BetrVG). If review actions are logged per person, discuss it briefly with the works council; the §90(1) no. 3 information duty applies when work processes change. Personal data (contact persons, driver or employee records in files) may occur: involve the DPO, conclude an AVV with every cloud processor (Art. 28 GDPR), and mask samples.
AI literacy: a short briefing on reading confidence scores and checking mapped amounts (GUIDE-006). Pilot with two partners of different difficulty. Typical concern: "the AI changes our numbers". Answer: rules are tested on samples, approved by a person, and run deterministically afterwards. See GUIDE-008, GUIDE-009 and GUIDE-013.
Risks, regulation & compliance
- EU AI Act: minimal risk; the system processes business data, not decisions about people. AI literacy (Art. 4) applies. If the mapped data feeds a high-risk system (for example HR or credit scoring), the Annex III obligations lie with that downstream system, but data governance (Art. 10) must be taken into account.
- GDPR: partner files often contain personal data (contacts, employees, customers, sole traders). Send only masked samples to external models, sign DPAs, prefer EU regions or self-hosting, apply data minimisation and retention limits, and keep the data lineage.
- Data Act (Regulation 2023/2854): gives users of connected products rights to data access and requires interoperability; this increases demand for standardised mapping and exchange.
- Security: treat uploaded files as untrusted (CSV/formula injection, malicious XML, prompt injection in cells); validate with schemas, sandbox generated code, log all mapping changes.
Getting started
- Pick the 5–10 most painful partner feeds and measure current onboarding time and error rate.
- Write the target schema with field descriptions, units, allowed values and examples; this is the most valuable input for any matcher.
- Test one Path A tool and one LLM prototype (headers plus masked samples) on those feeds; score field-level accuracy.
- Make the mapping a versioned artefact with tests, a review step and drift alerts.
- Feed every reviewer correction back into the mapping memory and track the share of unchanged proposals.
Sources
- Flatfile documentation — https://flatfile.com/docs/
- OneSchema documentation — https://docs.oneschema.co/
- Stedi documentation (EDI, mappings, guides) — https://www.stedi.com/docs
- SAP Integration Suite — https://help.sap.com/docs/integration-suite
- Boomi documentation — https://help.boomi.com/
- Valentine schema matching (open source) — https://github.com/delftdata/valentine
- UN/EDIFACT standards (UNECE) — https://unece.org/trade/uncefact/introducing-unedifact
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Regulation (EU) 2023/2854 (Data Act) — https://eur-lex.europa.eu/eli/reg/2023/2854/oj
- Regulation (EU) 2016/679 (GDPR) — https://eur-lex.europa.eu/eli/reg/2016/679/oj