Automated accessibility scanners such as axe-core and Lighthouse are worth running on every build. They’re fast, they never get bored, and they catch real problems: images without alt text, form fields without labels, text with too little contrast, duplicate IDs, invalid ARIA attributes. The trouble starts when a clean scan gets reported as “the page is accessible”. A scanner can only check what can be computed from the page’s markup and styles. It can’t press Tab, it can’t tell whether a label makes sense, and it doesn’t know what’s supposed to happen when you press Escape.
This article runs the comparison on one component, a confirmation dialog, with real output: what axe reports, what keyboard tests in a real browser report, and what fixes both.
What a scanner can and can’t decide
Many WCAG success criteria split into a part a machine can check and a part it can’t. “Images have a text alternative” is checkable: the alt attribute exists or it doesn’t. “The text alternative describes the image” isn’t: alt="image" passes the scan. “All functionality is operable from a keyboard” can’t be decided by inspecting markup at all, because it depends on what scripts do when keys are pressed.
So scanners are good at presence and syntax: is there a name, is the ARIA valid, is the contrast ratio above 4.5:1. Behavior, meaning and experience need tests that act like users: keyboard tests in a real browser for interaction, and people with assistive technology for whether the result is actually usable.
A confirmation dialog that passes axe
Here’s a “Delete account” confirmation built the way many dialogs are built: a div that’s shown and hidden.
<main>
<h1>Account settings</h1>
<button id="delete">Delete account</button> <a href="#help">Help</a>
</main>
<div id="confirm" class="modal" hidden>
<h2>Delete account?</h2>
<p>This permanently removes your data.</p>
<button class="close" onclick="closeModal()">×</button>
<button onclick="closeModal()">Cancel</button>
<button>Delete</button>
</div>
<script>
const modal = document.getElementById('confirm');
document.getElementById('delete').onclick = () => { modal.hidden = false; };
function closeModal() { modal.hidden = true; }
</script>The test opens the page in Chrome through Playwright, clicks “Delete account”, and runs axe with the WCAG 2.1 A and AA rules:
test('axe finds no WCAG A/AA violations with the dialog open', async () => {
await openDialog();
const results = await new AxeBuilder({ page }).withTags(['wcag2a', 'wcag2aa', 'wcag21a', 'wcag21aa']).analyze();
assert.deepEqual(results.violations.map((v) => `${v.id}: ${v.help}`), []);
});✔ axe finds no WCAG A/AA violations with the dialog open (801.969ms)Zero violations. Even the icon-only close button passes, because its text content × gives it an accessible name. A screen reader announces it as “times button” or “multiplication sign button”, depending on the reader, which is technically a name and practically useless. Playwright’s accessibility snapshot of the open page shows what assistive technology receives:
- heading "Delete account?" [level=2]
- paragraph: This permanently removes your data.
- button "×"
- button "Cancel"
- button "Delete"There’s no dialog in that tree. To a screen reader user, a heading and three buttons have appeared somewhere at the end of the page, with nothing announcing that a decision is required.
Testing it the way keyboard users meet it
The WAI-ARIA Authoring Practices describe what a modal dialog must do, and each requirement turns into a test:
test('the dialog is exposed with its title as its name', async () => {
await openDialog();
assert.equal(await page.getByRole('dialog', { name: 'Delete account?' }).count(), 1);
});
test('opening the dialog moves focus into it', async () => {
await openDialog();
assert.ok(await focusIsInsideDialog(), `focus is on ${await focusedText()}`);
});
test('Tab never reaches the page behind the dialog', async () => {
await openDialog();
for (let i = 0; i < 6; i++) {
await page.keyboard.press('Tab');
const inside = await focusIsInsideDialog();
const onBody = await page.evaluate(() => document.activeElement === document.body);
assert.ok(inside || onBody, `Tab ${i + 1} moved focus to ${await focusedText()}`);
}
});
test('Escape closes the dialog and returns focus to the button that opened it', async () => {
await openDialog();
await page.keyboard.press('Escape');
assert.equal(await page.locator('#confirm').isVisible(), false, 'dialog is still visible');
assert.equal(await focusedText(), 'button "Delete account"');
});focusIsInsideDialog checks whether document.activeElement is inside #confirm, and focusedText describes the focused element for the failure message. Against the div version:
✔ axe finds no WCAG A/AA violations with the dialog open (801.969ms)
✖ the dialog is exposed with its title as its name (172.3115ms)
✖ opening the dialog moves focus into it (193.392459ms)
✖ Tab never reaches the page behind the dialog (199.692833ms)
✖ Escape closes the dialog and returns focus to the button that opened it (202.041917ms)
AssertionError [ERR_ASSERTION]: focus is on button "Delete account"
AssertionError [ERR_ASSERTION]: Tab 1 moved focus to a "Help"
AssertionError [ERR_ASSERTION]: dialog is still visibleEach message describes what a keyboard user would experience. After opening the dialog, focus is still on “Delete account”, behind it. Pressing Tab goes to the “Help” link on the page underneath, so a keyboard user can wander around a page they can’t see past the overlay. Escape does nothing. A sighted mouse user would never notice any of this.
A note on the Tab test: it allows focus to land on body because in a real browser, tabbing past the last control of a native modal dialog moves focus to the browser’s own interface, which the page reports as body. Tracing the native version below in Chrome, focus went Cancel, Delete, body, Close, Cancel, Delete. What must never happen is focus reaching a control behind the dialog.
The native dialog version
The HTML dialog element, opened with showModal(), handles most of this in the browser:
<dialog id="confirm" aria-labelledby="confirm-title">
<h2 id="confirm-title">Delete account?</h2>
<p>This permanently removes your data.</p>
<form method="dialog">
<button value="cancel" aria-label="Close">×</button>
<button value="cancel" autofocus>Cancel</button>
<button value="delete">Delete</button>
</form>
</dialog>
<script>
const dialog = document.getElementById('confirm');
document.getElementById('delete').onclick = () => dialog.showModal();
</script>✔ axe finds no WCAG A/AA violations with the dialog open (748.404958ms)
✔ the dialog is exposed with its title as its name (173.999ms)
✔ opening the dialog moves focus into it (207.851125ms)
✔ Tab never reaches the page behind the dialog (238.268792ms)
✔ Escape closes the dialog and returns focus to the button that opened it (190.751ms)showModal() gives the element the dialog role, makes the rest of the page inert so Tab can’t reach it, closes on Escape, and in current Chrome returns focus to the previously focused element on close. aria-labelledby names the dialog after its heading. autofocus on Cancel puts initial focus on the least destructive action, a deliberate choice for a delete confirmation. And aria-label="Close" replaces “times” with a word that means something. The accessibility snapshot now shows dialog "Delete account?" containing button "Close".
These tests are worth keeping even with the native element, because they protect behavior from future changes: someone adding a custom close handler that forgets focus, or swapping the element for a component library’s version.
What still needs a person
The keyboard tests cover the mechanics. Several important questions still need a human, ideally including people who use assistive technology daily:
- Screen reader experience. Is the announcement on open understandable? Does “Delete account?, dialog” followed by the description make the consequence clear? Test with at least one screen reader and browser pairing your users have, such as NVDA with Firefox or Chrome, or VoiceOver with Safari.
- Meaning. Do labels, link text and error messages make sense out of context? “Click here” and “Invalid input” pass every scanner.
- Zoom and reflow. At 200% and 400% zoom, does the dialog still fit, or do the buttons end up off screen?
- Other conditions: visible focus indicators in every state, motion that respects reduced-motion settings, captions on video, timeouts users can extend.
A workable split is: axe on every page in CI; keyboard and accessibility-tree tests for each interactive component, as above; a manual checklist pass with a screen reader before releasing new components or flows.
Interview practice
A product manager shows you a Lighthouse accessibility score of 100 for the new checkout and asks if the accessibility work is done. What do you tell them?
Answer
I’d say the score is good news but not a sign-off. Lighthouse runs a subset of axe-core rules, so a 100 means none of the automatically checkable problems it looks for were found on the pages and states it loaded. It doesn’t test whether the checkout works with a keyboard, whether focus moves sensibly when the payment step opens or an error appears, whether error messages are announced to screen readers, or whether labels make sense. It also only sees the state of the page at load, not the address form after a validation error or the card-details modal.
I’d propose three additions before calling it done. First, keyboard tests in Playwright for the checkout flow: completing a purchase with only the keyboard, focus moving to the first error after a failed submit, and the payment dialog handling focus and Escape correctly. Second, axe scans of each step in its interesting states, not only on page load. Third, a manual pass with a screen reader through the whole purchase, plus a check at 200% zoom. If we can, a short session with a user who relies on assistive technology finds things none of us will. I’d also mention that WCAG conformance is assessed against success criteria, many of which need human judgement, so no automated score can establish it.