Many organisations run critical processes on mainframe terminals, thick-client ERPs, vendor portals and government web forms that offer no usable API. Classic RPA scripts these screens with fixed selectors and breaks whenever a button moves. A computer-use agent looks at the screen, decides the next click or keystroke and adapts to layout changes. It cuts development time for long-tail automations, but it is slower, costlier and less deterministic than scripts, so it is best used inside a governed, human-supervised framework.
Problem & value
- Today: staff re-key data between systems, download reports from portals, and process the same forms daily. Existing RPA bots (UiPath, Blue Prism, Automation Anywhere, Microsoft Power Automate desktop flows) cover the stable, high-volume core, but each bot needs a developer, and UI changes break selectors. The long tail of low-volume processes is never automated.
- AI value: the task is described in natural language or as an SOP; the agent perceives the UI via screenshots (plus accessibility tree or DOM where available) and acts. It tolerates cosmetic UI changes, pop-ups and unexpected dialogs, and development drops from days to hours or minutes.
- Evidence is mixed: a 2026 study found RPA faster and more reliable at run time than a Claude computer-use agent, while the agent cut development time substantially [1]; Blue Prism argues RPA stays essential for thick-client and mainframe systems [2]. Design hybrid: scripts for the stable core, agents for the long tail and exceptions.
Variants
- Browser agent for web portals: supplier, carrier and authority portals without API.
- Desktop agent for thick clients: SAP GUI, Citrix/RDP apps, run in an isolated VM per task.
- Terminal/mainframe agent: 3270/5250 emulators in banking and insurance.
- Agent-assisted RPA: the agent generates or repairs a deterministic script once; the script then runs cheaply.
- Public sector: bridging legacy specialist applications (Fachverfahren) to portals.
- Private use: consumer browser agents for bookings and forms; see AGENT-019.
How it works
- trigger (queue item / email / schedule / human request)
- task planner (goal + SOP + allowed apps and domains)
- sandboxed environment (VM, container or remote browser, scoped service account)
- loop: observe (screenshot + a11y tree/DOM)
- reason
- act (click/type/key/scroll/tool call)
- verify result (screen state, extracted value, postcondition check)
- guardrails (allowlist, action policy, injection filter, spend/step limits)
- human approval for irreversible steps (submit, pay, delete, send)
- result + full trace (screenshots, actions, model reasoning) to audit log
- Control loop: each step is a model call (screenshot in, action out); tasks take tens of steps, so cost scales with length. Prefer API/tool calls where available; use the screen only for gaps.
- Human-in-the-loop: approval gates before irreversible or financial actions, takeover when the agent is stuck, and sampling review of completed runs.
- Hybridisation: deterministic postcondition checks wrap the agent so errors are caught.
Data preparation
A computer-use agent needs less training data than other use cases but more process knowledge. Relevant inputs are the target application (legacy ERP client, supplier or authority portal, bank or insurer web UI), the task descriptions and the master data the agent will type in, typically held in ERP, Excel or mail attachments. Typical gaps: undocumented click paths and exceptions, shared logins without named accounts, MFA or captchas blocking automation, inconsistent source data (typos, duplicates, missing mandatory fields) and no test environment, so experiments happen on production.
Concrete steps: (1) record the process step by step with screenshots and 20 real cases including failures; (2) check first whether an API, export or file interface exists, which is always more robust than clicking (PREP-005); (3) clean and validate the input records before the agent sees them (PREP-011); (4) set up named service accounts with least privilege and a test or sandbox tenant; (5) define success checks per step (expected screen text, record count) and build a small evaluation set (PREP-018); (6) classify what data may be shown to the model (PREP-024). Tools: Playwright for recording, a password vault, a test VM. Data effort is low, around 10-20%; process documentation and access setup dominate. See GUIDE-003.
Implementation paths
Which path for whom
- SMB: A. A managed RPA or computer-use product with sandboxed browsers saves building secure infrastructure; the process is typically a single legacy portal.
- Enterprise: B. Strategic processes and volume justify building on cloud APIs inside a governed sandbox with central credential vaulting, audit and kill switches.
- Developer: B. Frontier computer-use models via API are still clearly ahead of self-hostable ones; self-hosting needs HW3 and is rarely worth it.
- Public sector: B. Use EU-region or sovereign cloud APIs with strict allow-lists; buy only certified products.
Path A — Buy
- RPA platforms with agentic UI control: UiPath (UI Agent, bring-your-own-model incl. Anthropic computer use; check the vendor docs for current preview/GA status) [3]; Microsoft Power Automate / Copilot Studio computer use [4]; Automation Anywhere; Blue Prism (SS&C). EU: Camunda (Berlin) for orchestration, H Company (Paris) agent models [5]; check EU hosting of every vendor.
- Agent products: OpenAI ChatGPT agent, Anthropic Claude browser/desktop agents, Google Gemini.
- Choose A when: you own an RPA platform and need vaulting, orchestration and audit out of the box.
Path B — Build on cloud APIs
- Computer-use APIs: Anthropic computer use tool (screenshot, mouse, keyboard in an agent loop) [6]; OpenAI computer-use tool in the Responses API [7]; Google Gemini computer use. Use EU regions (Claude via AWS Bedrock/Vertex EU, Azure OpenAI EU data zone); Mistral (EU) for text sub-steps.
- Browser frameworks: Playwright, browser-use [11], Stagehand, Skyvern; hosted remote browsers.
- Orchestration: queue + worker (Temporal, Camunda, n8n), one isolated VM/container per run, secrets in a vault, traces into AGENT-025 tooling.
Path C — Self-host (open source)
- Open-weight GUI models: Qwen-VL family (Qwen2.5-VL/Qwen3-VL), UI-TARS (ByteDance), OpenCUA, H Company Holo models (EU); served with vLLM or Ollama.
- Stack: browser-use/Playwright, Docker or Firecracker sandboxes, Temporal/n8n/Camunda.
- Hardware: HW3 for 30B+-class GUI models; HW2 for simple browser flows with lower reliability.
- Choose C when: screens show regulated data (health, finance, administration) that must stay in-house.
Path D — Custom training
Fine-tune a small GUI model on recorded trajectories (screenshots + actions) of your own applications to raise accuracy and cut per-step cost. Needs thousands of validated trajectories and an evaluation harness; rarely justified before prompt-and-SOP approaches are exhausted.
State of the art (2026)
- Benchmarks: OSWorld (real desktop tasks) is the reference [8]; vendor-reported scores rose from low double digits in 2024 to above 70% by 2026, but are self-reported and harness-dependent. Also: WebArena, Mind2Web, AndroidWorld.
- Products: Microsoft reports computer use in Copilot Studio as generally available in 2026 [4]; major model vendors ship computer-use tools in their APIs [6][7]; RPA incumbents integrate third-party models rather than training their own.
- Techniques: hybrid perception (screenshot + accessibility tree), grounding models, reasoning planners, and "record once, replay with self-healing" to avoid model calls on every run.
- Limits: slow and costly versus scripts; mis-clicks on dense UIs; loops on unexpected dialogs; weak with CAPTCHAs, MFA and drag-and-drop; non-determinism complicates SLAs. Prompt injection is the central security risk: text on a page, PDF or email can steer the agent into exfiltrating data; vendors document it as unsolved [6][9].
Shared resources & combinations
- Same runtime: AGENT-021 (data entry & migration) and AGENT-004 (order entry) use it as fallback when no API exists; AGENT-002 supplies triggers, queues, approvals; AGENT-013 can use it as a tool; AGENT-025 provides observability; AGENT-006 for ERP screens; CODE-005 when wrapping the legacy system with an API is the better long-term fix.
Effort, cost & KPIs
- Path A: weeks for a pilot if an RPA platform exists; licences plus model credits.
- Path B: 2-4 months for a hardened platform; cost driver is tokens per step (screenshots).
- Path C: 3-6 months plus GPU operations.
- KPIs: task success on a fixed evaluation set, straight-through rate, cost per task, mean time to repair after UI change, downstream error escape rate, guardrail-blocked incidents.
People & change
Involve the department lead and the employees who currently do the clicking: they own the exceptions and must validate the agent's work in the pilot. IT and especially IT security are essential, since the agent acts with real credentials in real applications; they define sandbox, allow-listed domains, vaulted secrets and logging (GUIDE-013). The DPO checks personal data on screens (screenshots go to the model provider), the AVV and whether a DPIA is needed. Management approves the risk appetite and the rule that no payment, deletion or external submission happens without approval.
Works council: yes in practice. The agent works under user or service accounts and records screens and actions, so it is objectively suited to monitor behaviour (§87(1) no. 6 BetrVG, BAG); §90(1) no. 3 applies to the changed work processes, and an AI expert may be engaged under §80(3). Agree that logs are used for security and quality only, not for individual performance review (GUIDE-004). Public sector: Personalrat.
Choose one stable, low-risk process and a pilot group of 2-4 users with an exit criterion (GUIDE-009). Typical concerns: replacing staff, errors in live systems and being watched. Address them by showing the agent's run log, starting in read-only or draft mode, keeping a visible stop button, and training staff to review and take over (GUIDE-006, GUIDE-008). Measures to support AI literacy should cover prompt-injection on web pages.
Risks, regulation & compliance
- EU AI Act: the technology is general-purpose, so risk depends on the process [10]. Back-office data transfer is typically minimal risk (AI literacy, Art. 4). It becomes high-risk under Annex III if the agent makes or materially influences decisions on employment, creditworthiness, access to essential public or private services or benefits. Where it interacts directly with natural persons in a way that is not obvious, transparency under Art. 50 applies. No Art. 5 prohibition is triggered by the technology itself; avoid using it for prohibited practices such as social scoring.
- GDPR: the agent sees whatever is on screen, including special-category data; apply data minimisation (restrict applications and fields), a DPA with providers, EU hosting or self-hosting, and a DPIA for large-scale or sensitive processing. Art. 22 limits solely automated decisions with legal effect.
- Security: prompt injection, credential misuse and over-broad permissions. Mitigations: dedicated least-privilege service accounts (no personal logins), network/domain allowlists, isolated disposable VMs, no standing access to payment or admin functions, human approval for irreversible actions, treat all screen content as untrusted input [9]. NIS2 and DORA entities must cover the agent in their ICT risk management and third-party oversight.
- Other: automating third-party portals may breach their ToS; keep tamper-evident traces and one owner per agent.
Getting started
- Inventory candidate processes: no API, medium volume, tolerant of occasional human review; rank by effort saved versus risk. Keep RPA for stable high-volume flows.
- Build an evaluation set of 20-50 real task instances in a test environment with mock data and measure success, steps, cost and time per task.
- Pilot with one tool from Path A and one from Path B in a sandboxed VM with a scoped service account and approval gates on every write action.
- Add guardrails (domain/app allowlists, injection filtering, step and spend limits) and tracing before widening scope.
- Run in shadow mode next to the human process, review traces, then move to supervised production with sampling-based review.
Sources
- Are LLM agents the new RPA? A comparative benchmark study of agentic automation and RPA (Cogent Business & Management, 2026) — https://www.tandfonline.com/doi/full/10.1080/23311975.2026.2724760
- Blue Prism — The future of RPA: trends and predictions 2026 — https://www.blueprism.com/resources/blog/future-of-rpa-trends-predictions/
- UiPath documentation (search "UI Agent") — https://docs.uipath.com/
- Deck — The State of Computer Use Agents in 2026 (includes Copilot Studio computer-use status; vendor blog) — https://deck.co/blog/state-of-computer-use-agents-2026
- H Company (Paris) — https://www.hcompany.ai/
- Anthropic — Computer use tool documentation — https://docs.claude.com/en/docs/agents-and-tools/tool-use/computer-use-tool
- OpenAI — Computer use guide — https://platform.openai.com/docs/guides/tools-computer-use
- OSWorld benchmark — https://os-world.github.io/
- OWASP — Top 10 for LLM Applications (prompt injection) — https://genai.owasp.org/llm-top-10/
- Regulation (EU) 2024/1689 (AI Act) — https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- browser-use project — https://github.com/browser-use/browser-use