🤖 Playwright AI Self-Healing Agent

An autonomous AI agent that automatically detects, analyzes, and fixes failing Playwright tests in GitHub Actions workflows. This agent uses OpenAI's GPT models to understand test failures, generate targeted fixes, verify them through CI, and create draft pull requests—all without human intervention.

🚀 Quick Start

# Install dependencies
pip install -r requirements.txt

# Set up environment variables
export GITHUB_TOKEN="ghp_your_github_token"
export OPENAI_API_KEY="sk_your_openai_key"

# Run the agent
python main.py https://github.com/owner/repo/actions/runs/12345678

✨ Features

• 🔍 Automatic Failure Detection: Monitors GitHub Actions workflows and identifies all test failures
• 🧠 AI-Powered Analysis: Uses OpenAI to analyze failure logs and understand root causes
• 🛠️ Surgical Code Fixes: Generates precise, targeted fixes for failing tests
• 🔄 Self-Healing Loop: Automatically retries fixes up to 3 times, learning from previous failures
• ✅ CI Verification: Creates branches and validates fixes through GitHub Actions
• 📋 Draft PR Creation: Automatically creates draft PRs with verified fixes
• 🏢 Monorepo Support: Handles complex monorepo structures with path normalization
• 🎯 Multi-Issue Resolution: Processes multiple distinct failures independently

🏗️ Architecture

┌─────────────────┐
│  GitHub Actions │
│   (Failed Run)  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  GitHubClient   │◄─── Fetches logs, metadata, diffs
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  LogProcessor   │◄─── Extracts relevant failure info
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│    AIAgent      │◄─── Analyzes & generates fixes
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Self-Healing   │
│      Loop       │
├─────────────────┤
│ 1. Generate Fix │
│ 2. Create Branch│
│ 3. Push Changes │
│ 4. Run CI       │
│ 5. Verify       │
│ 6. Retry if ❌  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│   Draft PR      │
│   ✅ Success    │
└─────────────────┘

🚀 Getting Started

Prerequisites

• Python 3.8+
• GitHub Personal Access Token with appropriate permissions:
  • repo (full control)
  • workflow (to trigger workflows)
• OpenAI API Key

Installation

1. Clone the repository:

   git clone <repository-url>
   cd playwright-ai-agent

2. Install dependencies:

   pip install -r requirements.txt

3. Configure environment variables: Create a .env file in the root directory:

   GITHUB_TOKEN=ghp_your_github_token_here
   OPENAI_API_KEY=sk-your_openai_api_key_here

Usage

1. Run the agent with a failed workflow URL:

   python main.py https://github.com/owner/repo/actions/runs/12345678

2. Monitor the process: The agent will:

   • 🔍 Identify all distinct failures
   • 🛠️ Process each issue independently
   • 🔄 Retry failed fixes up to 3 times
   • ✅ Create draft PRs for successful fixes

📦 Project Structure

playwright-ai-agent/
├── main.py                 # Entry point and orchestration logic
├── requirements.txt        # Python dependencies
├── .env                    # Environment variables (create this)
└── agent/
    ├── __init__.py
    ├── ai_agent.py        # OpenAI integration for analysis & fix generation
    ├── github_client.py   # GitHub API wrapper
    ├── log_processor.py   # Log parsing and cleanup
    └── models.py          # Pydantic models for structured data

🔧 Configuration

Retry Settings

Adjust the maximum retry attempts in main.py:

MAX_RETRIES = 3  # Number of fix attempts per issue

AI Model

Configure the OpenAI model in agent/ai_agent.py:

self.model = "gpt-5-mini"  # or "gpt-4", "gpt-4-turbo", etc.

CI Timeout

Modify the CI verification timeout in main.py:

timeout = 900  # 15 minutes (in seconds)

🧪 How It Works

1. Failure Identification

The agent parses GitHub Actions logs and identifies distinct test failures with:

• File path and line number
• Error category (timeout, assertion, etc.)
• Stack traces and error messages

2. Intelligent Fix Generation

Using OpenAI, the agent:

• Analyzes the failure context
• Reviews the original code
• Generates a surgical fix following best practices
• Avoids generic solutions (e.g., blanket waitForTimeout calls)

3. Verification Loop

For each fix:

Generate Fix → Create Branch → Push Code → Trigger CI → Wait
                                                          │
                                                          ▼
                                                      Success?
                                                      │     │
                                                     Yes    No
                                                      │     │
                                                Draft PR  Retry
                                                          (with error context)

4. Monorepo Handling

The agent automatically normalizes file paths:

• Log path: tests/login.spec.ts
• Actual path: packages/playwright/tests/login.spec.ts

📊 Example Output

🤖 Starting Agent for Run: 21857019114 (push)
🔍 Identifying all distinct failures...
📊 Found 2 issues to address.

🛠️  Processing Issue in: tests/auth/login.spec.ts
   🧠 Attempt 1/3: Generating surgical fix...
   🧪 Waiting for CI verification on branch: ai-fix-a3b92f...
   ⏳ CI still in progress... (30s)
   ⏳ CI still in progress... (60s)
   ✅ SUCCESS: Fix verified by CI.
   🚀 Draft PR Created: https://github.com/owner/repo/pull/123

🛠️  Processing Issue in: tests/checkout/payment.spec.ts
   🧠 Attempt 1/3: Generating surgical fix...
   🧪 Waiting for CI verification on branch: ai-fix-c7d45e...
   ❌ FAIL: Attempt 1 did not resolve the issue.
   🔍 Fetching new logs for re-analysis...
   🧠 Attempt 2/3: Generating surgical fix...
   ...

🛡️ Best Practices & Safety

• Draft PRs: All PRs are created as drafts to allow human review before merging
• Branch Isolation: Each fix gets a unique branch to avoid conflicts
• CI Gating: Fixes are only proposed after passing CI verification
• Retry Logic: Failed fixes are retried with additional context from new logs
• Monorepo Aware: Handles complex repository structures

🔐 Security Considerations

• Never commit .env file to version control
• Use GitHub tokens with minimal required permissions
• Rotate API keys regularly
• Review all draft PRs before merging
• Consider rate limiting for API calls

🤝 Contributing

Contributions are welcome! Areas for improvement:

• Support for additional test frameworks (Jest, Cypress, etc.)
• Parallel processing of multiple failures
• Cost tracking for OpenAI API calls
• Webhook-based automatic triggering
• Enhanced diff analysis for regression detection
• Support for flaky test identification

📝 License

[Add your license here]

🙏 Acknowledgments

• Built on OpenAI's GPT models
• Utilizes GitHub REST API
• Designed for Playwright test automation

---

Note: This agent uses AI to generate code fixes. Always review generated changes before merging to production.