Classic scrapers break whenever a site changes its layout, and every site needs its own selectors. AI extraction lets a model read the rendered page (text, DOM or screenshot) and return the fields you asked for as validated JSON, so one pipeline covers hundreds of sites. Typical uses: price and assortment monitoring, lead and company data, tender and regulatory watch, research corpora. The main constraints are not technical but legal: copyright, database rights, site terms, robots.txt and GDPR.
Problem & value
- Today: analysts copy data by hand, or developers write and maintain per-site scripts (Scrapy, Playwright, XPath/CSS selectors). Maintenance is the dominant cost: every redesign breaks a scraper, and long-tail sites are never worth the effort.
- AI value: schema-driven extraction ("give me product name, price, currency, availability") that tolerates layout changes, handles unstructured detail pages, and normalises output (units, currencies, languages). Models also turn pages into clean markdown for downstream RAG (see DOC-014).
- Cost trade-off: LLM extraction is slower and costlier per page than selectors. Hybrid designs are common: let a model generate or repair selectors once, then run cheap deterministic extraction and call the model only on failures or new layouts. No generally valid savings figure exists; measure on your own sites.
- Prefer APIs and licensed data first: if a site offers an API, open-data feed, sitemap with structured data (schema.org JSON-LD) or a data licence, use that instead of scraping.
Variants
- E-commerce / retail: competitor price and assortment monitoring, product attribute harvesting (feeds DATA-010).
- Sales & marketing: company and contact discovery from public sites (see AGENT-007, AGENT-011); highest GDPR exposure when natural persons are involved.
- Public sector & research: tender portals, legislation, open-government pages, web archives for text-and-data-mining (TDM).
- Finance / risk: news, filings and ESG disclosures monitoring.
- Developer / AI teams: crawling for RAG knowledge bases and agent web access.
- Private use: personal price trackers or research clippings; household exemption in GDPR is narrow.
How it works
- URL discovery (sitemap, search API, seed list)
- policy gate (robots.txt, terms/licence check, rate limits, personal-data rules)
- fetch (HTTP, or headless browser for JavaScript pages)
- clean (boilerplate removal
- markdown / pruned DOM / screenshot)
- extract (LLM or VLM against JSON schema, structured outputs)
- validate (types, ranges, cross-checks, source-text grounding, dedup)
- load (warehouse / PIM / CRM) + monitoring (drift, failure rate)
- human review of low-confidence records and schema changes
- Extraction: an LLM receives cleaned markdown or DOM plus a schema and returns JSON; schema- constrained decoding avoids malformed output. A VLM reads screenshots when the DOM is obfuscated, canvas-rendered or table-heavy.
- Selector induction: the model proposes CSS/XPath rules from sample pages; rules run without a model until validation fails ("self-healing" scrapers).
- Browser agents: for logins, pagination and forms, an agent drives the browser (see AGENT-003). Logins to third-party services are mostly contract-prohibited; do not circumvent access controls.
- Human-in-the-loop: sample-based audit, review of low-confidence fields, approval of new sources by someone who has checked terms and legal basis.
Data preparation
The data here comes from public websites, not from internal systems, so the preparation is mostly about defining the target and the destination. Typical gaps are no clear schema (which fields, which units), unclear legal status of sources, inconsistent product naming that blocks matching with the own catalogue, and no place to store results with source URL and timestamp.
Preparation steps: (1) list target sites and check their terms, robots.txt, copyright and database rights with legal, see GUIDE-014 for the regulatory frame; (2) write a JSON schema with field types, units and allowed values, see PREP-019; (3) prepare the internal reference data the results are matched against, such as own article lists, see PREP-005 and PREP-011; (4) collect 20-50 pages with correct extracted values as an evaluation set, see PREP-018; (5) clean and deduplicate results afterwards with DATA-003; (6) exclude personal data from the schema unless a legal basis exists, see PREP-025.
Tools: Playwright or Crawl4AI for rendering, a small LLM for extraction, a database or spreadsheet as target, and DOC-014 for PDF documents found on the way. Rough share of effort: about 30 percent, mostly schema design, matching to own data and monitoring changes. Check readiness with GUIDE-003.
Implementation paths
Which path for whom
- smb: Path B, because a small script or n8n flow with an LLM API fits the specific sites and fields better than a generic product, and few mature EU-hosted products exist.
- enterprise: Path B, because scraping feeds own pricing, sales or research systems and volume and legal control justify an internal pipeline on cloud APIs.
- developer: Path C, because open-source crawlers with a small local model remove per-page API costs and keep full control of rate limits and compliance.
Path A — Buy (off-the-shelf)
- AI scraping/extraction services: Firecrawl, Apify (Actors marketplace; Czech company, EU-based), Zyte (Irish origin; AI-assisted extraction API), Bright Data, Diffbot (knowledge-graph style extraction), Browse AI and Octoparse (no-code), Jina Reader (URL-to-markdown).
- Check: EU processing location, whether the vendor's proxy/data sourcing is ethically compliant, robots.txt support, data-processing agreement, who bears legal responsibility for targets (you do).
- Choose A when: a few dozen sources, no developers, results needed quickly.
Path B — Build on cloud APIs (recommended default)
- Fetch with Playwright or a managed browser/fetch API; clean with Trafilatura or Readability; extract with a frontier or mid-size model using structured outputs: Mistral (EU), Claude, Gemini, GPT via EU-region endpoints (Azure OpenAI, Vertex AI, Bedrock Frankfurt). Use small, cheap models for bulk pages and escalate hard pages to larger ones.
- Orchestrate with n8n, Airflow or Dagster; store raw snapshots (for audit/reprocessing) separately from extracted records. Use prompt caching and batch APIs to cut cost.
- Frameworks: ScrapeGraphAI, LangChain/LlamaIndex loaders, Crawl4AI (also usable in Path C).
Path C — Self-host (open source)
- Crawlers: Crawl4AI, Scrapy plus scrapy-playwright, Crawlee, Firecrawl (open-source core), Katana-style link discovery; cleaning with Trafilatura.
- Models: open-weight LLMs via vLLM or Ollama (Qwen3 family, Llama, Mistral, Gemma); Qwen-VL family for screenshots; small fine-tuned extractors for bulk (e.g. ReaderLM-style HTML-to-markdown/JSON models from Jina).
- Hardware: HW2 (one 24-32 GB GPU) handles ~7-14B models for moderate crawl volumes; a CPU-only setup works for selector-based extraction with occasional model calls. Ops effort: proxies, browser pools, scheduling, storage.
Path D — Custom training
Worth it only for very large, homogeneous crawls (e.g. millions of product pages) where a fine-tuned small model replaces an expensive API. Needs hundreds to thousands of page/JSON pairs, which the review queue produces naturally.
State of the art (2026)
- Standard pattern: page-to-markdown plus schema-constrained LLM extraction is a mainstream developer building block; hybrid "model writes the scraper, rules run it" is the cost-efficient evolution.
- Agentic browsing: browser agents and tool protocols (MCP-based web tools) let assistants navigate and extract live, but are slower, less deterministic and more costly than crawlers.
- Anti-bot escalation: sites increasingly block AI crawlers (robots.txt rules for named AI user agents, bot management, pay-per-crawl and licensing schemes). Evasion tooling exists but is legally and ethically hazardous; this catalogue does not recommend it.
- Limits and failure modes: hallucinated or "helpfully completed" fields, wrong price/variant selection on pages with several offers, truncated long pages, silently missing data after redesigns, prompt injection from page content aimed at the extraction or agent model, and cost blow-up on large crawls. Mitigate with source-grounding checks (extracted value must appear in page text), schema validation, anomaly alerts on field fill rates, and treating page content as untrusted input.
Shared resources & combinations
- Same stack (llm-general, vlm, web-scraping): DOC-014 (parsing PDFs/pages for RAG), DATA-022 (partner data/schema mapping), AGENT-003 (browser agents), DATA-003 (cleaning and deduplication of scraped records).
- Downstream: DATA-010 (product enrichment & classification), DATA-001 (conversational BI on the collected data), AGENT-011 (CRM enrichment), AGENT-007 (prospecting).
- Shared browser pool, proxy/egress policy and one vLLM server (HW2/HW3) can serve several of these.
Effort, cost & KPIs
- Path A: days to weeks; pricing per page/credit or per compute run.
- Path B: 1-2 months for a monitored pipeline; model cost scales with pages and tokens, so use hybrid selector reuse and caching.
- Path C: 2-4 months incl. browser/proxy operations and GPU hosting.
- KPIs: field-level accuracy against a labelled sample, coverage (share of target pages yielding complete records), cost per 1,000 pages, time to repair after a layout change, block/failure rate, freshness, share of sources with documented legal review.
People & change
The business sponsor (sales, purchasing or research lead) defines which data is worth collecting and who uses it. Legal and compliance must clear each source before the pilot, because copyright, database rights, site terms and competition law are the real risks. IT sets up the crawler infrastructure, including rate limits and identification; end users check sample outputs.
Works council: usually no, because the system collects public web data and does not monitor employees; it becomes relevant only if usage logs are evaluated per person (§87(1) no. 6 BetrVG), and §90(1) no. 3 still calls for timely information when work processes change. Public bodies inform the Personalrat in the same cases. Data protection: avoid personal data; if contact data of individuals is scraped, involve the DPO, check the legal basis and run a DPIA when done at scale.
AI literacy: support staff with a short briefing on what extracted data is reliable and what is not, see GUIDE-006, and on the company AI policy, see GUIDE-007. Start with one source type as pilot, show side-by-side results against the manual process, and keep a human review of a random sample each week. Address the concern about legal exposure with a documented source list and a stop switch for each site. Guides: GUIDE-009, GUIDE-011.
Risks, regulation & compliance
- EU AI Act: typically minimal risk; AI literacy (Art. 4) applies. Art. 5 caution: the prohibition on creating or expanding facial-recognition databases through untargeted scraping of facial images from the internet or CCTV (Art. 5(1)(e)) means image scraping for biometric databases is prohibited; this use case must not be used for it. Using scraped data to feed high-risk systems (Annex III, e.g. recruitment or credit scoring) shifts the classification of the downstream system. General-purpose AI model providers have copyright-policy duties (Art. 53), relevant if you scrape to train models.
- GDPR: public availability does not remove protection. A lawful basis (usually legitimate interest, Art. 6(1)(f), documented in a balancing test), transparency toward data subjects (Art. 14, with narrow exceptions), data minimisation, retention limits and special-category exclusion (Art. 9) apply. Regulators such as the CNIL and the EDPB (Opinion 28/2024 on AI models) have set out legitimate-interest expectations for web scraping. Prefer extracting company or product data, not personal data; filter or pseudonymise names, emails and profiles at ingestion.
- Copyright and database rights: EU DSM Directive Art. 3/4 allow text-and-data-mining for research and, unless rightsholders opt out in machine-readable form (e.g. robots.txt, TDM reservation protocol), for commercial purposes; the sui generis database right (Directive 96/9/EC) can restrict extraction of substantial parts of a database. Respect opt-outs.
- Contract and unfair competition: site terms of use, access controls (circumvention can be unlawful), and national unfair-competition law. Check CFAA-like local rules for non-EU targets.
- Security: isolate browsers, treat content as untrusted, never give extraction agents credentials or write access.
- Mitigations: source register with legal assessment, robots.txt/opt-out compliance, polite rate limits, identifiable user agent, provenance fields on every record, DPIA where personal data is involved at scale.
Getting started
- Define the business question and target schema; check whether an API, feed or licence exists first.
- Make a source register: owner, terms, robots.txt/TDM opt-out status, personal-data presence, legal basis.
- Prototype on 20 pages per source with page-to-markdown plus structured-output extraction; label a sample and measure field accuracy.
- Move stable sources to cached or generated selectors; keep the model for failures and new layouts.
- Add monitoring (fill rates, drift, cost) and a review queue before connecting results to decisions.
Sources
- Firecrawl documentation — https://docs.firecrawl.dev
- Crawl4AI (open-source LLM-friendly crawler) — https://github.com/unclecode/crawl4ai
- ScrapeGraphAI — https://github.com/ScrapeGraphAI/Scrapegraph-ai
- Scrapy documentation — https://docs.scrapy.org
- Apify documentation — https://docs.apify.com
- Trafilatura — https://github.com/adbar/trafilatura
- Regulation (EU) 2024/1689 (AI Act), Art. 5(1)(e) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- Directive (EU) 2019/790 (DSM), Art. 3-4 TDM — https://eur-lex.europa.eu/eli/dir/2019/790/oj
- EDPB Opinion 28/2024 on AI models and personal data — https://www.edpb.europa.eu/our-work-tools/our-documents/opinion-board-art-64/opinion-282024-certain-data-protection-aspects_en
- CNIL — AI and GDPR recommendations (legitimate interest, web scraping) — https://www.cnil.fr/en/ai-and-gdpr-cnil-publishes-new-recommendations-support-responsible-innovation
- Directive 96/9/EC (database right) — https://eur-lex.europa.eu/eli/dir/1996/9/oj