# ghost-qa tested: 13 planted bugs, a clean page still failed

Canonical: https://www.kartikaydhar.com/blog/ghost-qa-browser-qa-coding-agents
Published: 2026-11-20T03:30:00.516+00:00
Updated: 2026-11-20T03:01:55.902361+00:00
Category: Tool Breakdowns

I ran ghost-qa 2.1.2 on a two-page app with 13 planted bugs. It caught all 13 and one unplanned bug, scored the app 21/100, and still failed a clean page.

ghost-qa is a free, open-source (MIT) tool from VibeFoundry. It opens a headless browser, a browser with no visible window, on a site you already have running, checks the pages in three screen sizes and in light and dark mode, and writes a scored bug report a coding agent can use before it says the work is done.

The project is brand new. The repo was created on 7 November 2026, version 2.1.2 shipped on 17 November, and it had 48 stars when I tested it. I installed that release and pointed it at two tiny local apps I wrote. On a storefront with 13 planted bugs, it found all 13, plus one I had not planted, scored the app 21 out of 100 (grade F), and finished in 24.9 seconds. A cleaned-up version of the same pages scored 90 (grade A-) and still failed, because the default settings treat mid-severity issues as blocking. Turning the fail threshold up to critical made that same clean run exit successfully.

## What I tested

- Version: ghost-qa 2.1.2 from npm, GitHub commit 19cc3ab. TypeScript, MIT-licensed, 48 stars when I checked the repo. Playwright, the library it uses to drive the browser, pulled in Chromium.
- Setup: Two local two-page apps, with no API keys and no model calls. The built-in doctor setup check passed (Node.js 24.22.1, Chromium, and a quick blank-page test).
- Task I ran: The buggy app and the clean page, then compare to diff the buggy and clean reports, a --fail-on critical rerun of the clean page, and a listing of the MCP tools. MCP is the Model Context Protocol, a way for an AI assistant to call tools over a local connection.
- Time to first result: The buggy audit printed 24.9 seconds. The clean audit printed 18.4 seconds.
- What worked: It caught 13 of 13 planted bugs, including a console error I had put in the test app. Its click runner also triggered addToCart is not defined, a 14th real bug I had not planted. Each finding has a stable id and a selector, a path to the element on the page.
- What broke or surprised me: The clean page still failed (exit code 1) at 90/100, on eight touch-target flags (links and inputs too small to tap easily on a phone). A plain-text 404 page was flagged for 1:1 contrast in dark mode, and for the same 606-pixel sideways overflow as the home banner, even though that page has no banner. Every command also asked me to star the GitHub repo.
- Would I use it at work? Yes, as a done-check after a coding agent changes a UI, with --fail-on set on purpose. I would not drop the default command into CI, the automatic checks that run on every change, without deciding what should fail the build.

## How do you try it?

You need Node.js and a site already running on your machine, for example at http://localhost:3000. No API key is required. On first use, Playwright, the library ghost-qa uses to drive the browser, downloads Chromium.

Check the setup first:

npx ghost-qa doctor

Then run an audit:

npx ghost-qa test http://localhost:3000

## What does ghost-qa actually check?

It launches headless Chromium, crawls internal links, and runs eight checks, which the project calls "traps": console errors, network and asset failures, layout (including overflow and touch-target size), accessibility, performance, links, SEO and metadata, and light/dark theme differences. The default viewports, the page widths it tests, are desktop at 1440x900, tablet at 768x1024, and mobile at 375x812, each in light and dark mode. It also clicks and fills some controls, unless you pass --no-interactions.

Every run produces a 0-100 "QA Health Score" and a letter grade. Each of the eight checks starts at 100 and loses 25 points per critical finding, 10 per major, and 3 per minor. The overall score is the average of those eight. My two reports matched that math: the clean app scored 20 on layout and 100 on the other seven checks, which averages to 90. The report lands as report.json, a markdown file, a patch prompt for an agent, and an HTML dashboard. Help lists --fail-on critical, major (the default), or minor. The exit code, the number a script uses to decide success or failure, is 0 for pass and 1 for fail.

## What did it catch on the broken app?

The buggy run visited three routes (home, about, and a missing page), lasted 24.9 seconds, and failed (exit code 1). The score was 21/100, grade F, from 122 findings: 11 critical, 66 major, and 45 minor. Nine interactions passed, nine were skipped for safety, and none were marked failed.

I had planted 13 bugs, and every one showed up:

- Broken links and server errors: a "Deals" link that went nowhere, a "Careers" link to a missing page, a logo file that did not exist, and a cart API that returned an error on all three screen sizes.
- Missing labels and low contrast: no main heading (the page jumped in at a lower heading), an icon button with no name for assistive tools, a logo with no alt text, an email field with no label (a placeholder is not enough, and the tool says so), body text at 1.92:1 contrast against a 4.5:1 requirement, and a duplicate id.
- Mobile overflow: a 600-pixel-wide banner that stuck out 606 pixels past a 375-pixel phone screen.
- JavaScript crashes: a function that ran on page load and did not exist, and a console error on the about page, "Failed to load profile widget", which I had put in the test app.

When the runner clicked the unnamed button, the page also threw addToCart is not defined. That is the 14th real bug, and I had not planted it. I connected the button to a function I forgot to write.

Image: Illustration: What did it catch on the broken app?

## Why did a clean page still fail?

The clean page is the same two pages with the planted bugs fixed. The audit visited home and about, lasted 18.4 seconds, and scored 90/100, grade A-. Seven of the eight checks scored 100. Layout scored 20, from eight major findings, all of one type: WCAG 2.5.5 touch-target flags, on tablet and mobile only. WCAG is the Web Content Accessibility Guidelines. Rule 2.5.5, Target Size, asks for a 44 by 44 CSS-pixel minimum. It is a Level AAA recommendation, and the tool also cites Apple's Human Interface Guidelines. The "About" link was 42 by 17 pixels, "Home" was 43 by 17, and the email input was 205 by 21. Under the default --fail-on major, that is enough for the command to fail (exit code 1) and print FAIL.

I reran the same clean page with --fail-on critical. The score stayed 90/100, and the eight findings were still in the report. The command exited 0, marked the report as passed, and printed PASS, in 18.3 seconds. The findings do not go away. You only get a green exit if you choose that threshold on purpose.

This is the same kind of "look at it in a real browser" check I have written about with Hallmark, which still leaves some website checks to a browser, and with agent-browser, another way to drive a real browser from an agent. ghost-qa is narrower: it audits the pages it can crawl, then hands you a list.

## Why did 14 bugs turn into 122 findings?

The long report is mostly repetition and extra checks, not 122 separate defects. 48 of the 122 findings were SEO and metadata: missing page tags search engines and social apps look for, like page descriptions and social preview tags, repeated across routes and screen sizes. Most functional bugs were also filed once per viewport, so a single cart API error became three critical lines. Some issues were filed twice: the missing logo was both a broken image and a broken file request, and the 1.92:1 text showed up as an accessibility failure and as a theme failure in both light and dark mode.

Image: Illustration: Why did 14 bugs turn into 122 findings?

Two extras on the plain-text 404 page are hard to defend as product bugs. In dark mode, ghost-qa flagged Chrome's own "not found" text at 1:1 contrast, three times. It also reported the same 606-pixel sideways overflow as the home banner, even though the 404 page is only the words "not found". An agent told to "fix" those would be chasing the browser, or a leftover measurement, not the app. Each line still has an id and a selector, so an agent can group by type if you tell it to. A 122-line report, dumped raw into a chat, will fill up the AI's prompt and invite drive-by SEO edits.

## Where does this sit in a coding-agent loop?

The intended loop is: change the code, start a local server, run ghost-qa test, fix what you can, and run ghost-qa compare against the earlier JSON.

Image: Illustration: Where does this sit in a coding-agent loop?

compare on my buggy report versus the clean report said the score moved from 21 (F) to 90 (A-), a gain of 69, with 122 findings fixed, 8 new regressions, and 0 remaining. The eight "new regressions" are the clean page's touch-target flags. The verdict stayed "NOT YET VERIFIED". The tool only sets verified when the score is 95 or above, with no new regressions and no blocking findings left. A 90 with eight majors does not clear that bar, and compare itself failed (exit code 1).

There is also a native MCP server: npx ghost-qa mcp. I checked the MCP server locally and confirmed it lists three tools: ghost_qa_audit_url, ghost_qa_compare, and ghost_qa_doctor. I did not wire them into a model. If you already use MCP-based checks, this is the same idea as tester-army's e2e helper, which turns agent clicks into test lines, pointed at a live app instead of a recorded script. One caution: init (I did not run it) writes a protocol into CLAUDE.md, AGENTS.md, and .cursorrules, plus other agent config files. Review that in a repo you already care about. Every command I ran, including doctor, printed a prompt to star the GitHub repo, plus a gh repo star one-liner. The HTML dashboard does the same. In a CI log, that is just noise.

## What this means for product and GTM teams

If a coding agent is shipping landing pages or app UI, a no-model browser pass before a human looks is worth having. ghost-qa is free, it runs on your own computer before anything goes live, and in my test it did not miss a planted functional or accessibility bug. Plan for a long report: 13 planted bugs plus one surprise became 122 lines, 48 of them metadata. Tell the agent to fix critical and major functional issues first, and to group by type and selector.

Decide the fail threshold before you add this to CI. The default will fail a tidy page that only has 44 by 44 touch-target issues, as my 90/100 clean page did. --fail-on critical made that page exit 0, which is not a claim that the page meets WCAG 2.5.5. Treat "verified" as the tool's own bar: compare keeps saying "NOT YET VERIFIED" until you are at 95 or above with no majors left. A grade of A- can still be a failed check.

## Frequently asked questions

### What is ghost-qa?

A free, open-source (MIT) command-line tool and MCP server from VibeFoundry. It opens headless Chromium on a running URL, checks the pages, and writes a scored bug report.

### Does it need an API key or a paid model?

No. Version 2.1.2 makes no model calls. I ran the audits, the compare command, doctor, and the MCP tool list with no keys.

### Why did a clean page still fail?

The default --fail-on major setting treats mid-severity findings as blocking. My clean page only failed WCAG 2.5.5 touch-target checks on tablet and mobile, scored 90/100, and still failed (exit code 1).

### Can I make a clean page pass in CI?

Yes. On the same clean page, --fail-on critical kept the eight findings in the report, scored 90/100, and exited 0 with a PASS. You can also enlarge the links and inputs to 44 by 44 pixels and keep the default.

### What does the MCP server add?

It exposes the same audit, compare, and doctor steps as tools an AI assistant can call: ghost_qa_audit_url, ghost_qa_compare, and ghost_qa_doctor. I checked the MCP server locally and confirmed those names. I did not run a full agent session through them.

## Sources

- ghost-qa repository: https://github.com/avivgerz-hue/ghost-qa
- npm package, version 2.1.2: https://www.npmjs.com/package/ghost-qa
- WCAG 2.1 Understanding, 2.5.5 Target Size (Level AAA, 44 by 44 CSS pixels): https://www.w3.org/WAI/WCAG21/Understanding/target-size.html
- Apple Human Interface Guidelines, Buttons (44 by 44 pt hit region, cited by the tool): https://developer.apple.com/design/human-interface-guidelines/buttons
