# ui-diff tested: visual QA that caught a contrast bug

Canonical: https://www.kartikaydhar.com/blog/ui-diff-visual-qa-pull-requests
Published: 2026-11-21T04:08:15.341+00:00
Updated: 2026-11-21T02:40:40.42029+00:00
Category: Tool Breakdowns

I ran ui-diff on a two-page demo. It caught a 1.91:1 contrast drop, and Cursor's agent correctly explained a misleading focus-order note.

When a coding agent changes your app, the code diff doesn't show what users will see. ui-diff takes before-and-after pictures of the pages that changed and flags new accessibility problems, so a person can check them before anything ships. In my test it caught a text color change that made a paragraph hard to read.

It is a free, open-source (MIT) tool by Murat Alpay. This is visual regression QA: a check that compares screenshots of the new version against the old one, so you can see what moved. That kind of check matters more now that coding agents change UI often; the GitHub repo went up on 19 November 2026. It can also post one comment on a pull request, a proposed change on GitHub. Nothing is posted until you run a separate publish command.

## What I tested

- Version: ui-diff 0.1.0 from the GitHub repo, commit 3075b4c. MIT-licensed, 58 stars when I checked the repo.
- Setup: A local two-page demo on a feature branch. The command-line run used no API keys and no model calls. It found my two pages on its own and checked that my setup was ready. I also ran it through Cursor's agent with the skill installed by install.sh --cursor.
- Task I ran: Compare the feature branch with the main branch. On the home page I had made the call-to-action button larger (more padding, rounder corners, bigger type) and changed its label from "Start free trial" to "Start your free 14-day trial". I also lightened an intro paragraph from dark gray (#444) to light gray (#bbb). The pricing page was left alone.
- Time to first result: The first command-line run took 19 seconds. A confirmation on the same demo finished in 15.2 seconds and produced the same numbers. Through Cursor's agent, the successful run took 17.4 seconds. On that fresh machine, install.sh --cursor took 2.6 seconds, and downloading Chromium took 7.2 seconds.
- What worked: It flagged the changed home page, left the untouched pricing page alone, and caught one new contrast failure. The agent's summary described the button and the contrast drop, and it correctly explained the keyboard-order quirk.
- What broke or surprised me: The automatic summary says the keyboard focus order changed, even though the report marks the order as not reordered. Following the skill's advice, a run with --themes light,dark failed with Unknown theme "dark", because the setup file only listed a light version of the pages. I did not run ui-diff publish.
- Would I use it at work? Yes, as a local check after a coding agent changes a UI, or in CI (the automatic checks that run on every change), with a person reading the preview before anything is posted. I would rather use the agent-written summary than the automatic one.

## How do you try it?

You need git and Node.js 20.10 or newer. On Windows, the README says to use WSL2. No API key is required for a local run. The GitHub CLI (gh) is only needed later, if you publish a comment. The README says the first capture downloads a browser engine, about 150 MB.

Download the tool from GitHub. For a command-line run, point an alias at skills/ui-diff/bin/ui-diff.mjs; the first run installs the tool's own packages. install.sh --claude is only needed if you want a coding assistant to load the skill; use --cursor or --codex for those. Then, in your app folder, run init (it finds your pages) and run (it takes and compares the screenshots).

git clone https://github.com/imuratalpay/ui-diff.git ~/src/ui-diff
alias ui-diff='node ~/src/ui-diff/skills/ui-diff/bin/ui-diff.mjs'

cd ~/src/my-app
ui-diff init
ui-diff run

Open .ui-diff/comment.local.md for the comment preview and .ui-diff/compare/ for the images. Nothing is posted until you run ui-diff publish. The README says cloud agents need the skill committed in the project. In my Cursor session the skill was not auto-loaded; the agent read the installed SKILL.md.

## What did the run actually catch?

run first found the two pages I had changed. It made temporary copies of both versions, started a server for each, and took a before and an after picture of each page, on a laptop-sized screen and a phone-sized screen. That's four comparisons.

The home-page change was small in pixel terms, 0.82% on a laptop-sized screen and 0.39% on a phone-sized screen, concentrated in one area on each (8,374 pixels on the laptop-sized shot). That matches what I changed: the button grew and the intro paragraph got lighter. Pricing, which I had not touched, came back unchanged.

axe, a widely used accessibility checker, compared issues on the new pages with the old ones and flagged one new serious problem. The intro paragraph, light gray (#bbbbbb) on white (#ffffff) at 16px normal weight, was 1.91:1. WCAG, the common web accessibility standard, asks for at least 4.5:1 for normal text. The old dark gray (#444) on white is about 9.7:1, which clears that bar. Lightening that paragraph would have shipped a contrast failure.

Image: Illustration: What did the run actually catch?

## What did the coding agent's summary add?

The agent's summary is the one I'd hand a product manager: it explains what changed in plain words and corrects the tool's own misleading note. The automatic summary is three short bullets. It correctly notes the 0.82% change and the new contrast issue, then says "Keyboard focus order changed" on both screen sizes. That overstates what happened.

It describes the bigger button and the new label. It puts the contrast failure first, under "Worth a look", with the ratio and the 4.5:1 minimum, and notes the sentence was readable before. On keyboard order it is careful: "The order did not actually reorder: the button's accessible name changed, so 'Start free trial' is listed as removed and 'Start your free 14-day trial' as added." In plain words, the button's label changed, so the tool counted the old button as gone and the new one as new. Nothing actually moved.

It also explains why the pricing page was captured: that page shares the stylesheet, but it does not use the edited button and intro styles, and zero pixels changed. It lists what it did not check, including a dark theme, animation recordings, and a check of design colors and spacing.

A few caveats. The button sizes it quotes, 229 by 38 pixels growing to 357 by 49, were measured from the screenshot pixels, not reported by ui-diff. And when the skill's notes said to add --themes light,dark because a color had changed, that command failed with Unknown theme "dark", because the setup file only listed a light version of the pages.

## How does the check work?

It uses Playwright to drive Chromium, the same engine as Google Chrome, to take the screenshots; pixelmatch to compare them pixel by pixel; and axe-core 4.14 to check accessibility. In my run, init recognized a static HTML site, wrote a start command of python3 -m http.server, and listed the routes / and /pricing.html. The laptop-sized viewport was 1,280 by 800 pixels, and the phone-sized one was 390 by 844.

A pixel diff lays the two screenshots on top of each other and counts the pixels that do not match. ui-diff then builds a composite image: before, after, and a highlighted view of the changed pixels, plus a zoomed crop of the largest changed area.

The accessibility delta is the change in axe findings, not a full audit of the site from scratch. It was already on in my config. The tool can also make animation recordings or run a check of design colors and spacing; I did not turn those on, beyond the failed dark-theme attempt above.

The comment is meant to be one sticky note on the pull request, updated in place on later runs rather than stacked. Screenshots are not sent to a public image host. The README says publish commits them to a separate storage branch in the same repo (ui-diff-assets) and refuses a public repository unless you pass --allow-public-repo. I did not run publish. I did run render on the agent's summary, and it wrote a comment preview without uploading anything.

[[U

Image: Illustration: How does the check work?

PLOAD_FIG2]]

## What does it cost, and what does it not do?

The tool is free and open source under the MIT license. A local run needs no API key and makes no model calls. You do need Node.js 20.10 or newer, git, and a one-time browser download on the first capture if it is not already there. It is early: the published version is 0.1.0, and there were no GitHub Releases when I checked.

It will not tell you whether a change was intentional. A bigger button and a lighter intro paragraph are both "changed pixels." The contrast failure is the part a product reviewer can act on without guessing.

## What this means for product and GTM teams

If a coding agent is editing a landing page or an app settings screen, the pull request often looks like a pile of CSS. A screenshot pass in the 15 to 19 second range, on my two-page demo, gives you a picture of what actually moved, and it can catch a typography tweak that fails accessibility. A real app with more pages and slower startup will take longer.

The contrast result is the one I would want on a launch checklist. Lightening body text is a common polish pass. In this demo it took the intro paragraph from a passing contrast ratio to a clear failure, which should be fixed before the page goes out.

Keep a person in the loop before publish. The local preview is the right default: you can see the images and the findings without writing to GitHub. When you do publish, remember the images live in the same repo, visible to anyone with read access. If the automatic report says a control was removed and added, check whether it was only renamed.

This sits next to other browser checks a coding agent can run. ghost-qa scores a running site for bugs and accessibility, and agent-browser lets an assistant click through a live page. ui-diff is narrower: it is the before-and-after visual check on the pages a branch touched, plus the accessibility delta between those two versions.

## Frequently asked questions

### What is ui-diff?

A free, open-source (MIT) agent skill and TypeScript command-line tool by Murat Alpay. It takes before and after screenshots, compares them pixel by pixel, can run an accessibility delta, and can post one sticky comment on a GitHub pull request.

### Do I need an AI assistant or an API key?

No. I ran init, doctor, run and render from the command line with no API key and no model. I also ran it through Cursor's agent with the skill installed, which produced a clearer UX summary than the automatic fallback. The skill is optional.

### What is a pixel diff, and what is a contrast ratio?

A pixel diff compares two screenshots and counts the pixels that differ. A contrast ratio is how different a text color is from its background; 4.5:1 is the usual minimum for normal-sized text. In my run, the new intro-paragraph color was 1.91:1 on white.

### Does it post to GitHub on its own?

No. ui-diff run writes a local preview only. Nothing is posted until you run ui-diff publish, which needs the GitHub CLI signed in. I did not run publish.

### How is this different from ghost-qa?

ghost-qa audits a site you already have running and writes a scored bug report. ui-diff compares two versions of the same pages, shows the visual diff, and reports what got better or worse in accessibility. I would use them together: ui-diff on the pull request, ghost-qa as a pass over the running app.

## Sources

- ui-diff repository: https://github.com/imuratalpay/ui-diff
- ui-diff site: https://skills.muratalpay.me/ui-diff
- License (MIT, Copyright 2026 Murat Alpay): https://github.com/imuratalpay/ui-diff/blob/main/LICENSE
- axe-core color-contrast rule (v4.14): https://dequeuniversity.com/rules/axe/4.14/color-contrast
- WCAG 2.1 Understanding, 1.4.3 Contrast (Minimum), 4.5:1 for normal text: https://www.w3.org/WAI/WCAG21/Understanding/contrast-minimum.html
- Playwright: https://playwright.dev
- pixelmatch: https://github.com/mapbox/pixelmatch
