# CUA Agent Kit tested: AI finds the clicks, code repeats

Canonical: https://www.kartikaydhar.com/blog/cua-agent-kit-computer-use-to-playwright
Published: 2026-11-20T03:49:20.1+00:00
Updated: 2026-11-18T02:54:21.601927+00:00
Category: Tool Breakdowns

I replayed CVS Health's ready-made CUA Agent Kit sample 5 times with no AI model. All 5 passed in about a second, but its password masking has a gap.

Image: Illustration: CUA Agent Kit tested: AI finds the clicks, code repeats

CUA Agent Kit is a free, open-source starter kit from CVS Health for automating screen-based work in a browser or on Windows. An AI agent works out the steps of a job from plain-English instructions, then a coding assistant turns that recorded run into ordinary code that repeats the job with no AI model at all. Think of a weekly export you pull by hand from a vendor dashboard that has no API. For a PM, the appeal is that you pay for the AI once while it learns the job, and after that every run is free, quick and repeats the same steps.

I replayed the sample job that CVS ships already turned into code, five times. All five passed, in about a second each, with no AI model involved. Then I changed the sample app three ways to see what breaks it. I didn't run the AI discovery, the Codex step, the repair loop or the Windows version, because discovery needs a paid OpenAI API key and codifying needs Codex. So I can't say how many tries it takes the AI to learn a job, or how good the code Codex writes is.

CUA stands for computer-use agent: an AI model that operates a computer the way a person does, by looking at screenshots and choosing where to click and what to type.

## What I tested

- Version: the latest code on GitHub (commit 965ee25, 12 November 2026), package version 0.1.0. It is released under the Apache 2.0 licence, which means it is free to use, including commercially.
- Setup: three commands from the README: download a copy of the code from GitHub (cloning the repo), install the Python packages, and download Chromium, the open-source browser behind Chrome. About 12 seconds in total, with about 680 MB of browser files. No account or sign-up.
- Task I ran: the kit ships a fake healthcare app called Care Intake (fictional patients only) and a workflow CVS has already turned into code. It signs in, finds referral REF-1001, books an appointment and checks the confirmation page. I ran it five times in headless mode, meaning the browser ran without opening a window, with no OpenAI key set. Then I made three screen changes to the app, one at a time.
- Time to first result: once setup was done, the ready-made sample replayed in about a second. That is the replay of a job CVS had already codified. Teaching it a new job of your own goes through discovery and Codex first, which I didn't run.
- What worked: 5 of 5 runs passed the final check. The kit timed them at 1.093, 0.955, 0.909, 0.976 and 0.922 seconds. Each run saved a screenshot of every step and a log, about half a megabyte per run. The kit's own 40 tests all passed. After my screen edits were undone, the replay passed again.
- What broke or surprised me: the password masking in the run summary has a gap, and the saved browser recording keeps the password in plain text. One screen change made the replay report a failure even though the booking worked. Because I only replayed CVS's sample, I haven't seen how well the AI learns a job or repairs a broken one.
- Would I use it at work? Yes, for repeat jobs on stable internal web tools that have no API, once I had tried discovery and Codex on one real job and an engineer had reviewed the code. Not yet for screens that change often, and only after fixing how secrets are stored.

Image: Illustration: What I tested

The last screenshot from my fifth replay. The code checks this exact summary line before it marks the run as passed.

## What problem is it solving?

Plenty of work still happens by clicking through screens. The proper fix is an API (a way for software to talk to other software directly), but many tools don't have one, or it doesn't cover every action.

The older answer is RPA, short for robotic process automation: software robots that click through screens on a fixed script. The README says those tools "required significant time and investment for every new workflow." Computer-use agents fix that, since you describe the job in plain English. But an agent that thinks through the same clicks every day costs money on every run (you pay per token, the chunks of text a model reads and writes), and it can take a different path each time.

The kit's answer is to use the AI once to learn the job and then let plain code do it from then on. The README credits Kelsey Hightower's Zero Token Architecture talk at PlatformCon 2026 for the idea.

## How does it work?

Image: Illustration: How does it work?

### 1. Discover and record

You write the job in plain English in one file. An OpenAI computer-use model then does the job in a real browser while the kit records screenshots, each action, what it clicked on, and a trace. A trace is a step-by-step recording of a browser session that you can replay later to see exactly what happened. This step needs an OpenAI API key, the password-like code that lets a program use OpenAI's paid models. Without one, the kit stops with a clear message.

### 2. Codify

Next you ask Codex, OpenAI's coding assistant, to use the kit's codify skill on the successful runs. The skill tells it to swap screen positions for stable ways of finding things, such as a button's label. It also says: "Do not invent selectors." A selector is the rule code uses to find an item on a page, like "the button named Search".

For websites the output is Playwright code. Playwright is a free Microsoft library for controlling a browser from code. For Windows apps it is pywinauto, a free Python library that clicks and types in Windows programs through their built-in controls. Here is a short piece of the sample's codified Playwright code:

date = page.get_by_label(
    "Appointment date", exact=True)
await date.fill("2027-04-15")
await expect(date).to_have_value("2027-04-15")

It finds the field by its label, types the date, then checks the field holds that date before moving on.

### 3. Replay

The codified workflow runs with one command and no model. That makes it deterministic, as long as the screens don't change: the same input gives the same steps and the same result every time, because no AI is making choices. Each run still saves screenshots, a log and a trace.

## What did my runs show?

Each replay did the same six steps in the same order, passed the same final check, and finished in about a second, within 0.2 seconds of each other. For a PM, that sameness is what you'd be buying: a daily job with no model bill per run, a predictable run time, and no chance it clicks somewhere new. When it fails, the screenshots show the step where it stopped.

## What happens when the screen changes?

I edited the sample app three ways and replayed after each change.

### A renamed button fails loudly

I renamed the "Confirm appointment" button to "Confirm booking". The replay filled in the date and time, then waited for a button called "Confirm appointment" that no longer existed. After 30.9 seconds it hit its timeout, the longest the code will wait for something to appear before it calls the step a failure. The error named the button it was looking for, and the last screenshot showed the new label, so the cause was easy to spot.

Image: Illustration: A renamed button fails loudly

The last screenshot before the replay timed out. The date and time are filled in, but the button now says "Confirm booking", so the code never found the button it was told to click.

### Moving fields around didn't break it

I put the time field above the date field and changed the hidden ids that the page's own code uses for those fields. The replay still passed in 1.2 seconds. It finds fields by their visible labels, "Appointment date" and "Appointment time", so a layout change that keeps the labels is fine.

### A changed test id caused a false failure

A test id is an invisible tag developers add to part of a page so automated checks can find it. I changed the one on the confirmation summary. The booking went through and the page said "Appointment confirmed", but the final check looked for the old tag and marked the run failed after 6.0 seconds. The error even printed the confirmed booking it had missed. That beats a silent wrong booking, but it would still alert someone for nothing.

Fixing a broken job means giving the new traces to Codex to update the code, or rerunning AI discovery if the job itself changed, then having an engineer review it. I didn't run that repair loop, so I can't say how well it works.

## Where is the catch with passwords?

Redaction means blacking out sensitive text, like a password, before a file is saved or shared. The kit masks the sign-in details in run.json, the summary file each run writes, but badly. The sample login is username "demo" and password "demo-password", and this is what run.json saved:

Sign in with username [REDACTED] and
password [REDACTED]-password.

The masking replaced "demo" first, including inside the password, so "demo-password" was never matched and the rest stayed visible. It also over-masks: in the error from my false-failure test, the page's "Reset demo" button came out as "Reset [REDACTED]".

The trace file is worse. I found the password in plain text in several places: in the record of the step that typed it, in that step's log line, and in the page snapshots, which save what was typed into the password box. The step-by-step event log was clean. The README does warn that traces "can contain action arguments, credentials, cookies or session tokens."

From reading the code, not from a test: discovery sends your task description, with the username and password filled in, to the OpenAI model unmasked. The masking only touches what gets saved. The README says so too.

My practical tip: don't share run folders as they are, and never attach a trace to a ticket or a chat. If an engineer needs to see a failure, send the screenshot of the failing step, and use a throwaway test account while you try the kit.

## What does it cost?

The kit is free. Replays cost nothing beyond the machine they run on. Discovery uses the OpenAI API, paid per token, and the default model in the config file is gpt-6-sol. Codifying uses Codex, OpenAI's coding assistant; I didn't check which plan it needs. The AI cost lands up front, and again each time a screen change breaks the code.

## How to start

You need uv, a free Python package installer. Setup is three commands (the cd line just moves into the downloaded folder):

git clone https://github.com/cvs-health/cua-agent-kit
cd cua-agent-kit
uv sync
uv run playwright install chromium

Then put the README's three sample lines in a file called .env in the folder:

DISCOVERY_BACKEND=browser
START_URL=http://127.0.0.1:8081
WORKFLOW_ACCOUNT_JSON='{"username":"demo","password":"demo-password"}'

Start the sample app in one command window with uv run python sample_app/app.py, then run the replay in another:

PLAYWRIGHT_HEADLESS=true uv run cua-agent-run

Success looks like the "Appointment confirmed" page in the last screenshot, and "status": "succeeded" in the run.json inside artifacts/deterministic/.

Trying a job of your own takes more. You need an OpenAI API key for discovery, Codex for the codify step, and an engineer to review the code it writes before it runs anywhere real. It suits someone comfortable typing commands, or a PM pairing with an engineer.

## What this means for product and GTM teams

This is the RPA playbook with the expensive part swapped out. An agent figures out the clicks, a coding assistant writes the script, and what runs in production is code a person can review.

The jobs I'd point it at first are the boring repeat ones on tools with no API, like that weekly vendor-dashboard export, or a release check that a signup flow still works. I've tested a few tools in this space: agent-browser lets an agent drive a browser live, tester-army's e2e turns agent clicks into test lines, and unwalled records how an app looks up data so an assistant can repeat it. CUA Agent Kit is closest to the last two.

Two cautions. First, it is a starter kit. CVS says the hybrid mode, where a model handles only the steps that need judgment, is something you'd build yourself. Second, the security notes matter more than the speed. Before I pointed it at a real system, I'd change the masking so longer secrets are replaced first and keep traces out of shared storage.

## Frequently asked questions

### Do I need an OpenAI key to use CUA Agent Kit?

Only to discover a new workflow. Replaying a codified workflow needs no key and makes no model calls. I ran the bundled sample five times with no key set.

### What is the difference between a computer-use agent and RPA?

RPA follows a fixed script that someone wrote by hand. A computer-use agent decides each click from screenshots and plain-English instructions. This kit uses the agent to write the script, then runs the script like RPA.

### Does it work with desktop apps?

Yes, on Windows, through pywinauto. It needs a Windows machine running the kit's server. I only tested the browser sample.

### Is it safe to use with real logins?

Be careful. Discovery sends the sign-in details to the model, trace files keep the password in plain text, and the run summary masked it badly in my test. Use a test account and keep run folders private.

## Sources

- cvs-health/cua-agent-kit on GitHub (README and code, commit 965ee25, checked 18 November 2026)
- Choosing an approach (browser, Windows, AI-driven and hybrid modes)
- codify-playwright-workflow skill
- Playwright for Python documentation
- pywinauto documentation
- uv documentation
- Kelsey Hightower, Zero Token Architecture, PlatformCon 2026
