Axe Plugin - Authentication Support

[hanna-skryl]

User story

As a Code PushUp user, I want to test accessibility of authenticated pages using axe-core so that I can ensure my login-protected web applications meet WCAG compliance requirements.

Acceptance criteria

• setupScript option added to AxePluginOptions schema
• setupScript option accepts file paths in plugin configuration
• Setup script runs once before URL analysis, receiving a Playwright Page instance
• Setup script can perform any Playwright operations, including navigation, form filling, clicking, waiting for elements, cookie manipulation, localStorage access, and multi-tab handling
• Authentication state persists across all URL analyses via shared browser context
• Plugin aborts with descriptive error when:
  • setup script is not found
  • setup script lacks default function export
  • setup script execution throws
• Setup execution logged via logger.task() for progress visibility
• Unit tests cover config validation, path resolution, script validation, and context sharing
• Integration test demonstrates authenticated page analysis workflow

Setup script contract

• Scripts must export a default async function
• Receives a Playwright Page instance within a shared browser context
• Access to page.context() available for cookie/storage manipulation
• Existing timeout option from AxePluginOptions applies to setup navigation

Script execution flow

1. Browser launch: chromium.launch({ headless: true }) from playwright-core
2. Context creation: Single shared context via browser.newContext()
3. Setup execution:
   • Resolve script path (relative to process.cwd())
   • Dynamic import of setup module using the resolved path
   • Create page: context.newPage()
   • Execute: await setupFn(page)
   • Close setup page (context retains auth state)
4. URL analysis (for each URL):
   • Create page in shared context (inherits auth state)
   • Navigate and run AxeBuilder.analyze() from @axe-core/playwright
   • Close page
5. Cleanup

• The browser session is maintained across all URL analyses to preserve authentication state (cookies, localStorage, etc.)
• The browser closes only after all URLs have been analyzed, regardless of success or failure

Error handling

Error Type	Behavior	Log Level
Script not found	Abort plugin	error via thrown exception
Missing default export	Abort plugin	error via thrown exception
Setup execution failure	Abort plugin	error via thrown exception
URL navigation/analysis failure	Skip URL, continue	warn (existing behavior)

Setup failures cause the plugin to abort, as continuing would test unauthenticated pages, producing misleading results.

Error messages:

• Setup script not found: {resolvedPath}
• Setup script must export a default function: {resolvedPath}
• Setup script failed: {stringifyError(error)}

Testing

Unit tests

• Configuration validation accepts/rejects setupScript values
• Path resolution (relative and absolute)
• Script validation (missing file, missing export, non-function export)
• Setup function receives Page instance before navigation to target URLs
• Context sharing preserves auth state across multiple URL analyses

E2E tests

• A test fixture exists with the following structure:

e2e/plugin-axe-e2e/mocks/fixtures/auth-setup/
├── test-app/
│ ├── login.html           Login form
│ └── protected.html       Protected page (requires auth cookie)
├── axe-setup-script.ts    Playwright script that logs in
└── code-pushup.config.ts  Config using axe plugin with setupScript

• The protected.html file contains accessibility violations that can only be detected after authentication
• The axe-setup-script.ts navigates to login.html, fills the form, submits it, and waits for successful authentication
• The code-pushup.config.ts uses the axe plugin with setupScript: './axe-setup-script.ts' pointing to the authentication script
• The E2E test starts a local HTTP server that serves the test app and validates authentication via cookies
• When the collect command runs, it executes the setup script, successfully authenticates, navigates to the protected page, and analyzes it with axe
• The generated report.json contains audit results from the authenticated protected page
• The test verifies that without the setup script, the protected page would not be accessible (returns 401 or redirects to login)

Documentation

• The README.md includes a "Testing authenticated pages" section with setup script examples
• Code examples are provided for common scenarios:
  • Basic username/password login with environment variables
  • OAuth/SSO flow (e.g., GitHub login)
  • Multi-step navigation to nested authenticated pages
  • Testing multiple authenticated pages with a shared session
• Documentation includes environment variable setup instructions for local development (.env file), GitHub Actions (secrets configuration), and GitLab CI (variables configuration)
• The README clarifies that setup scripts use Playwright API for authentication, then @axe-core/playwright runs the axe analysis

Documentation examples

Basic login:

// axe-setup.ts
import type { Page } from 'playwright-core';

export default async function setup(page: Page): Promise<void> {
  await page.goto('https://example.com/login');
  await page.fill('[name="username"]', process.env['TEST_USERNAME']);
  await page.fill('[name="password"]', process.env['TEST_PASSWORD']);
  await page.click('button[type="submit"]');
  await page.waitForURL('**/dashboard');
}

Cookie-based auth:

import type { Page } from 'playwright-core';

export default async function setup(page: Page): Promise<void> {
  await page.context().addCookies([
    {
      name: 'session_token',
      value: process.env['SESSION_TOKEN']!,
      domain: 'example.com',
      path: '/',
    },
  ]);
}

CI configuration (GitHub Actions):

- name: Run Code PushUp
  uses: code-pushup/github-action@v0
  env:
    LOGIN_URL: https://staging.example.com/login
    TEST_USER: ${{ secrets.E2E_TEST_USER }}
    TEST_PASSWORD: ${{ secrets.E2E_TEST_PASSWORD }}

// code-pushup.config.ts
import axePlugin from '@code-pushup/axe-plugin';

export default {
  plugins: [
    await axePlugin(
      [
        'https://example.com/dashboard',
        'https://example.com/profile',
        'https://example.com/settings'
      ],
      { setupScript: './axe-setup.ts' }
    )
  ],
}

[matejchalk]

Again, very well documented. 👍

A few suggestions:

1. The main plugin config schema is extended to include an optional setupScript field

   After setup completes successfully, the runner navigates to each configured URL and runs axe analysis

   1 script for all URLs might not be flexible enough for some scenarios. Specific pages may require their own interactions to reach some common application state - e.g., opening a quick search modal on the homepage. Apps may also have different content per user (e.g. based on access level), so may require different logins per URL.

   It would still be useful to have 1 script applied to all URLs, but maybe we could combine that with an option to define a script per URL as well? Perhaps something like:

   axePlugin({
     'https://example.com/': { setupScript: './home-setup.ts', weight: 2 },
     'https://example.com/admin': { setupScript: './admin-setup.ts', weight: 1 },
   }, { setupScript: './base-setup.ts' })

2. Documentation examples

   I'm missing a simple Accessibility category configuration in these examples. The only category example is a custom Critical accessibility category, which is fine as an advanced example, but I would always show the simple "where most users should start" example first.

[hanna-skryl]

I'm missing a simple Accessibility category configuration in these examples. The only category example is a custom Critical accessibility category, which is fine as an advanced example, but I would always show the simple "where most users should start" example first.

The easiest approach is to create a mergeAxeCategories() function similar to the one Lighthouse implements and expose it to users. It will automatically generate an "Accessibility" category comprising the audits/groups returned based on the selected preset, and if needed, handle multiple URLs.

Something like this:

const axe = await axePlugin('https://example.com', { preset: 'wcag21aa' });

export default {
  plugins: [axe],
  categories: mergeAxeCategories(axe),
};

[matejchalk]

The easiest approach is to create a mergeAxeCategories() function similar to the one Lighthouse implements and expose it to users. It will automatically generate an "Accessibility" category comprising the audits/groups returned based on the selected preset, and if needed, handle multiple URLs.

A helper function for merging multiple URLs will definitely be needed, yes. I would approach this like we did for the Lighthouse plugin.

One minor difference is that, unlike Lighthouse, Axe results should typically only map to 1 high-level category. So I'd consider adjusting the helper function accordingly. 🤔

// original - looks like it could be multiple categories 😕 
export default {
  plugins: [axe],
  categories: mergeAxeCategories(axe),
};

// 💡 something like this is explicitly singular
export default {
  plugins: [axe],
  categories: [mergeAxeCategory(axe)],
};

// 💡 alternatively (or additionally), we could provide only the refs
// more advanced, but could be useful for a category combining other plugins 🤔 
export default {
  plugins: [axe, lighthouse, eslint],
  categories: [
    {
      slug: 'accessibility',
      title: 'Accessibility',
      refs: [
        ...mergeAxeGroupRefs(axe, 10), // 👈 possible weight multiplier?
        { type: 'group', plugin: 'lighthouse', slug: 'accessibility', weight: 3 },
        { type: 'group', plugin: 'eslint', slug: 'angular-eslint-template-a11y', weight: 1 },
      ],
    }
  ],
};

[hanna-skryl]

@matejchalk Thanks for the suggestion! I completely agree.

I'm currently working on the core Axe plugin functionality (#1134) and have extracted shared multi-URL handling logic to utilities. Between doing that work and reading through this discussion, I realized category merging really deserves its own focused issue. I've created #1139 to track it.

It makes perfect sense that the ideal solution would support aggregating accessibility categories across multiple plugins, not just Axe alone, but that's a more complex problem. I'd implement basic Axe category aggregation first, and then we could discuss how to aggregate a category from multiple plugins (I'm not sure it has been attempted before).