Automate UI testing

[jlewi]

We are iterating on lots of UI paper cuts.

How can we begin to automate UI testing?

Can we define scenarios or use cases and then drive the app through them (either with AI or without)?
We define assertions which we then use to ensure know regressions?
Could we produce videos/screenshots automatically which we upload?

For example in #68 we saw a bug where the cell output is cut off so could we define a scenario like the following

• Open test-long-output.json
  • This notebook should produce very verbose output e.g. 100 lines of output so that the cell won't fit on the single page
• Run the cell

Verify

• The cell output doesn't show on a single page
• There is a vertical scroll bar for the document
• You can scroll down to see the entire cell output

cc @hamelsmu

[hamelsmu]

@jlewi there is this cool thing called Codex

From the post

With the Codex app, you can also set up Automations that let Codex work in the background on an automatic schedule. Automations combine instructions with optional skills, running on a schedule you define. When an Automation finishes, the results land in a review queue so you can jump back in and continue working if needed.

At OpenAI, we’ve been using Automations to handle the repetitive but important tasks, like daily issue triage, finding and summarizing CI failures, generating daily release briefs, checking for bugs, and more.

Here are the docs. We should try it

[jlewi]

I think I'm looking for more detail such as

• how do we define the test scenarios? Are these text files? TS files?
• how do we define the assertions?
• do we use agent browser ? Playwright?

[hamelsmu]

Proposal: Automated UI Testing for Quill Notebook

(@hamelsmu's agent, reporting research findings)

@jlewi — here's a concrete proposal addressing your questions, informed by three approaches: (1) Ryan Carson's agent-driven E2E article, (2) OpenAI Codex Automations, and (3) the Ralph Playbook pattern.

---

Answering your questions directly

How do we define test scenarios?

Markdown files (.runme.md) with structured acceptance criteria. We already have test fixtures at web/app/test/fixtures/notebooks/ — the proposal is to extend this pattern with dedicated UX test scenarios. Each scenario is a markdown section with:

• A user journey description (e.g. "Open notebook with 120-line output, verify scrolling works")
• Acceptance criteria as a checklist — critically, all criteria must be verifiable via DOM inspection (not human judgment)

Example format:

## Scenario: Long cell output scrolls correctly
- Open `long-output-test.runme.md` in the app
- Run the "generate-120-lines" cell
- [ ] Output container has `overflow: auto` (not `hidden`)
- [ ] `scrollHeight > clientHeight` on the output div (content overflows)
- [ ] Vertical scrollbar is visible
- [ ] Scrolling to bottom reveals last line of output
- [ ] Cell card border-radius is not clipped by output

How do we define assertions?

Two tiers:

1. DOM/CSS assertions (primary) — verified via agent-browser eval running JS against the live page. These check computed styles, dimensions, scroll properties, element existence. Example: agent-browser eval "getComputedStyle(document.querySelector('.cell-card')).overflow" → assert not hidden.

2. Visual assertions (secondary) — screenshots compared by an LLM-as-judge (the Ralph Playbook's "perceptual quality test" pattern). For subjective criteria like "layout looks correct" or "no visual regressions", take a screenshot and have the AI evaluate it against criteria.

Do we use agent-browser? Playwright?

agent-browser for the primary path. Here's why, per Ryan Carson's article:

Traditional E2E relies on brittle CSS selectors, hardcoded test sequences, and fails on any UI change. Agent-Driven E2E uses accessibility tree snapshots and adapts to semantic changes.

agent-browser already gives us:

• snapshot -i — accessibility tree with interactive element refs (no CSS selectors needed)
• eval — arbitrary JS for DOM assertions
• screenshot — for visual verification
• click/type — for user journey simulation

Playwright as a complement for deterministic CI gating (e.g. checking that the app boots without console errors). We already have test/browser/test-smoke.sh doing this.

---

Three options for scheduling/automation

Option A: launchd + agent-browser (Carson's approach)

A shell script runs daily on a Mac mini (or CI runner), launches the app, runs agent-browser through each scenario, and files GitHub issues on failure.

• Pros: Simple, works today, we already have the scripts (test-smoke.sh, test-notebook-ui.sh)
• Cons: Requires a dedicated machine, no built-in review queue

Option B: Codex Automations

Codex Automations let you schedule recurring tasks that run in isolated worktrees. We could create an automation that:

1. Reads scenario files from web/app/test/fixtures/scenarios/
2. Launches the app, runs each scenario via agent-browser
3. Results land in the Codex review queue (inbox) for triage

• Pros: Built-in review queue, runs in isolated worktrees, can reference skills via $skill-name
• Cons: Requires Codex app running locally, not CI-native yet, newer/less battle-tested

Option C: Ralph Loop + Claude Code (or Amp)

The Ralph pattern structures test scenarios as user stories in a prd.json with acceptance criteria. A bash loop spawns fresh AI instances to execute each scenario, and the AI validates its own results using browser dev tools.

• Pros: Autonomous, self-healing (adapts to UI changes), accumulates learnings in progress.txt
• Cons: More complex setup, designed for development loops not pure testing

---

Recommended approach: Option A now, Option B later

Start with Option A — we're 80% there already:

1. We have Claude Code skills for this: ux-review (visual testing checklist) and runme-notebook-test (notebook validation)
2. We have test fixtures: long-output-test.runme.md, layout-test.runme.md, etc.
3. We have working scripts: test-smoke.sh, test-notebook-ui.sh
4. Next step: Create a scenarios/ directory with markdown scenario files, and a runner script that iterates through them using agent-browser

Graduate to Option B when Codex Automations matures — the review queue and scheduling are compelling for daily regression catches.

Borrow from Option C the key insight: acceptance criteria must be machine-verifiable (DOM-inspectable), not require human judgment.

---

Concrete next steps

1. Create web/app/test/scenarios/ directory with 5-10 scenario .md files covering: cell output scrolling, cell rendering, add-cell UX, notebook loading, console interaction
2. Write a run-scenarios.sh script that: starts the app, iterates scenario files, runs agent-browser assertions, captures screenshots, reports pass/fail
3. Wire into just test-ux command
4. Later: schedule via Codex Automation or launchd for daily regression catching