Test generation & QA automation

Most teams know their test suites are too thin, too slow or too brittle, but writing and maintaining tests is unglamorous work that loses against feature delivery. LLMs read code and requirements, write unit, API and end-to-end tests, run them, discard what does not compile or add value, and repair tests when the application changes. The gain is not "tests for free" but a cheaper path to a safety net, provided every generated test is verified by execution and by humans reviewing what it actually asserts.

Problem & value

• Today: developers write unit tests by hand, QA engineers script UI tests (Selenium, Playwright, Cypress) and manually run regression checks before releases. Legacy code often has no tests at all, which blocks refactoring and modernisation (CODE-005, CODE-021). UI tests break with every layout change and maintenance consumes a large part of QA capacity.
• AI value: bootstrap coverage for untested code, generate edge cases humans overlook, turn requirements or recorded user flows into executable tests, self-heal locators, triage flaky failures, and run exploratory testing with agents.
• Evidence: Meta's TestGen-LLM study reported that in its Instagram and Facebook evaluation a share of generated test classes improved coverage and that engineers accepted a majority of the tests shown to them in production; the key design idea was an automated filter chain (builds, passes, reliably passes, increases coverage) [1]. Benchmarks such as SWT-Bench show that generating fail-to-pass reproduction tests for real issues is feasible but far from solved [2]. Treat vendor productivity claims as unverified until measured on your own code.

Variants

• Unit-test generation for existing code (JVM, .NET, Python, TypeScript), often inside IDE or PR bot.
• Test-driven agents: coding agents (CODE-002) write the failing test first, then the fix.
• End-to-end / UI testing: natural-language or agent-driven browser tests, self-healing locators, visual regression with VLMs; mobile test automation.
• Test maintenance & flaky-test triage: repair of broken selectors, root-cause grouping of CI failures.

How it works

1. trigger (PR / nightly / requirements change)
2. context build (diff, covered/uncovered lines, related tests, specs, repo conventions)
3. LLM / agent proposes tests (unit, API, E2E)
4. automatic filter chain: compiles
5. runs
6. passes repeatedly (flakiness)
7. adds coverage
8. kills new mutants (mutation testing)
9. matches style/lint
10. failures fed back to the LLM for repair (bounded loop)
11. PR comment or commit with tests + coverage/mutation delta
12. human review (assertions meaningful? spec-conformant?)
13. merge

• Generation: a code-capable LLM with repository context; agentic variants execute the test runner and iterate. UI agents use the Playwright accessibility tree (and optionally screenshots via a VLM) instead of brittle pixel coordinates.
• Verification is the core: coverage alone is a weak signal; mutation testing (Stryker, PIT, mutmut) checks whether tests actually detect injected faults. Without it, LLMs tend to write tests that assert whatever the current (possibly buggy) code does.
• Human-in-the-loop: reviewers decide whether assertions encode intended behaviour, approve merges, and own the test oracle. Tests derived from the code under test cannot find a wrong specification.

Data preparation

Test generation draws on source code, existing tests and coverage reports, requirements or user stories (Jira, Confluence, Word), OpenAPI specs and CI logs. Typical gaps: requirements that exist only as prose or in people's heads, outdated API specs, no coverage measurement, flaky tests, test environments that differ from production, and test data copied from production with real customer or employee records.

Preparation steps: (1) generate or update OpenAPI specs and put them under version control; (2) measure baseline coverage and mutation score so progress is visible; (3) rewrite key requirements as testable acceptance criteria, see PREP-014; (4) build synthetic or pseudonymised test data, never raw production data, see PREP-022; (5) stabilise a test environment with seeded data and stable selectors (test IDs) for UI tests; (6) collect good existing tests as examples and templates, see PREP-016, and a set of known bugs as evaluation cases (PREP-018). Past defect reports are a source of labels (PREP-020).

Effort share: medium, about 25 to 35 percent, mostly test data and environment work.

Implementation paths

Which path for whom

• SMB: B - tests must fit the own code base, test framework and CI; coding assistants (see CODE-001) plus an API-based generation script cover this more cheaply than niche QA products.
• Enterprise: B - cloud APIs in EU regions combined with the existing test platform and CI give control over test data, quotas and traceability for large suites.
• Developer: C - open-weight coder models with Playwright and a local agent generate and repair tests offline at no per-token cost.

Path A — Buy (off-the-shelf)

• Unit/PR-level: Diffblue Cover (UK; Java, uses reinforcement learning rather than only prompting an LLM, tests are kept only if they compile and pass), Qodo (Israel; IDE and PR test generation, open-source Qodo Cover), GitHub Copilot, JetBrains AI Assistant (Czech vendor), Cursor, Amazon Q Developer.
• End-to-end / QA platforms: mabl, Tricentis (Testim, Tosca; Austrian-founded), Katalon, testRigor, Functionize, SmartBear, BrowserStack and Sauce Labs AI features, Applitools (visual AI). Check: EU data residency, how test artefacts and screenshots (may contain personal data) are stored, CI integration, exportable tests (avoid lock-in to proprietary formats).
• Choose A when: a team lacks capacity to build tooling and the stack matches a vendor's strength.

Path B — Build on cloud APIs (recommended default)

• Models: frontier coding models (Claude, GPT, Gemini) or Mistral's Codestral/Devstral (EU provider) via EU-region endpoints; use reasoning models for tricky edge cases.
• Agents: Playwright's official Test Agents and Playwright MCP server (planner, generator, healer) let an agent explore an app and emit Playwright tests [3]; coding agents such as Claude Code or Copilot agent mode run test commands in CI sandboxes.
• Pipeline: GitHub Actions / GitLab CI job calling the model, running the filter chain above, using JaCoCo/coverage.py/Istanbul for coverage and Stryker/PIT for mutation score [4].
• Choose B when: you want control over prompts, filters and repository conventions at modest cost.

Path C — Self-host (open source)

• Models: open-weight code models, e.g. Qwen3-Coder, Devstral, DeepSeek-Coder-V2 and Codestral-class models via vLLM or Ollama; HW2 runs ≤30B quantised models for unit tests, HW3 for larger agentic coding.
• Tools: Qodo Cover (open source, TestGen-LLM-style loop) [5], Playwright, Pynguin/EvoSuite (search-based test generation, no LLM required), Schemathesis and RESTler (API fuzzing), Stryker, PIT, mutmut, SonarQube Community for quality gates.
• Choose C when: source code may not leave the premises (defence suppliers, finance, health).

Path D — Custom training

Rarely justified. Fine-tuning on a company's own test conventions and past accepted tests can improve style fit and reduce rework; RL against test execution/mutation rewards is a research frontier. Needs thousands of reviewed (code, test) pairs and strong evaluation discipline.

State of the art (2026)

• Agentic test authoring is mainstream: agents run the code, read failures and iterate; first-party agent support in Playwright and IDE agents made natural-language E2E generation practical [3].
• Filter-then-accept (TestGen-LLM pattern) is the proven way to deal with hallucinated tests [1]; mutation score is increasingly used as the acceptance criterion instead of line coverage.
• Issue-reproduction tests (SWT-Bench) became a standard component of coding-agent pipelines because a failing-then-passing test is the best evidence that a fix works [2].
• Limits and failure modes: tautological assertions, over-mocking, tests that lock in bugs, flaky generated tests (timing, ordering), coverage inflation, weak performance on complex domain logic, concurrency, and non-deterministic UIs; no test oracle without a specification; cost of agent loops in CI.

Shared resources & combinations

• Same stack (llm-code + agent-framework): CODE-001 IDE assistants, CODE-002 coding agents (tests as verification), CODE-003 AI code review, CODE-014 bug reproduction and fixing, CODE-021 dependency upgrades and refactoring (tests as safety net), CODE-005 legacy modernisation (characterisation tests).
• Browser agents (computer-use): CODE-017 accessibility testing, CODE-024 game playtesting.
• Evaluation of AI apps: CODE-018 for LLM application evals; SEC-014 for adversarial testing of AI systems.
• Safety-critical: CODE-020, CODE-019 (testbenches and verification) need qualified toolchains.

Effort, cost & KPIs

• Path A: days to weeks; per-seat or per-test-run SaaS pricing (verify on vendor pages).
• Path B: 2–6 weeks for a CI pipeline with filters and reporting; token cost scales with repository context and number of repair iterations.
• KPIs: branch coverage and mutation score on changed code, share of generated tests accepted by reviewers, flaky-test rate, test maintenance hours, escaped defects, CI duration, cost per accepted test, time to reproduce a reported bug.

People & change

Involve the QA lead or development lead as process owner, developers and testers as daily users, IT for test environments and CI, information security and the DPO because test data and recorded UI sessions can contain personal data, and management for the target (coverage, release speed). Business departments contribute acceptance criteria; data owners release test data (GUIDE-003).

Works council: usually no, because the tools analyse code and test results, not people. It becomes relevant if tester productivity is reported per person or if session recordings capture employees; then section 87(1) no. 6 BetrVG applies. Inform the works council in good time anyway under section 90(1) no. 3, because test work processes change. Public sector: staff council.

AI literacy: support staff under Art. 4 AI Act with training on reviewing generated tests (do they assert the right behaviour, or just mirror the code?) and on the limits of agentic UI testing (GUIDE-006). Pilot on one service or one UI flow for four to six weeks and compare defects found, maintenance effort and coverage (GUIDE-009). Common concerns: manual testers fear for their jobs, developers distrust trivial tests. Position testers as reviewers and strategists, keep humans accountable for the test strategy and release decisions, and review generated tests like normal code (GUIDE-008).

Risks, regulation & compliance

• EU AI Act: minimal risk for developer tooling; AI literacy duty (Art. 4) applies. Not prohibited under Art. 5. If the tested software is itself high-risk or a medical device (Annex III / MDR), AI-generated tests do not remove conformity, validation and documentation duties; keep qualified verification evidence.
• GDPR: do not feed production data into prompts or test fixtures; use synthetic or anonymised data. Screenshots and traces from E2E runs may contain personal data; sending code to third-country APIs requires a DPA and transfer basis. Prefer EU regions or self-hosting.
• Security and IP: secrets in repositories, prompt injection through code comments or web pages visited by UI agents, licence contamination; run agents in sandboxes with least privilege and no production credentials.
• Mitigations: execution-based filters, mutation testing, mandatory review, test-data policy, audit log.

Getting started

1. Baseline: coverage, mutation score (on one module), flaky rate and QA hours per release.
2. Pilot an IDE/PR tool (Path A or B) on a module with some tests; require that each accepted test compiles, passes reliably and kills new mutants.
3. Add characterisation tests before any legacy refactoring or migration.
4. Pilot Playwright agents for 5–10 critical user journeys, exporting plain Playwright code under version control.
5. Define a review checklist (assertion meaning, mocking, data privacy) and track KPIs monthly.

Sources

1. Alshahwan et al., Meta — Automated Unit Test Improvement using Large Language Models at Meta (TestGen-LLM) — https://arxiv.org/abs/2402.09171
2. Mündler et al. — SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents — https://arxiv.org/abs/2406.12952
3. Playwright — Test Agents documentation — https://playwright.dev/docs/test-agents
4. Stryker Mutator — mutation testing documentation — https://stryker-mutator.io/docs/
5. Qodo Cover (open source) — https://github.com/qodo-ai/qodo-cover
6. Diffblue Cover documentation — https://docs.diffblue.com/
7. PIT mutation testing for Java — https://pitest.org/
8. Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj