Visual Failure Analysis Tool

Automated Review and Reference-Update Workflow for Robot Framework Compare Image Failures

Overview

The Visual Failure Analysis Tool is a lightweight validation utility developed during an internship at Ampere Software Technology (Renault Group).
It streamlines the HMI visual validation pipeline by automating the extraction and review of visual mismatches detected by the Compare Image keyword in Robot Framework.

The tool helps testers:

• Automatically extract failed image-comparison entries from Robot Framework XML logs.
• Visualize reference, actual, and delta images in a PyQt6 interface.
• Decide whether a failure corresponds to a software defect or a false positive.
• Perform bulk reference-image updates with full traceability.

This tool reduces manual triage time and improves the maintainability of visual test suites in CI pipelines.

Features

Automated XML Log Parsing

• Scans Robot Framework XML logs (output*.xml).
• Detects failed Compare Image keywords.
• Extracts:
  • Actual screenshot
  • Reference image
  • Pixel delta image
• Saves them inside structured per-test folders under output/.

PyQt6 GUI for Human-in-the-Loop Review

• Displays:
  • Test Case ID
  • Failed keyword index
  • Element name & reference path
  • Reference / Actual / Delta images
• Zoom slider with horizontal & vertical scrolling.
• Checkbox-based marking of false positives.
• Bulk reference updates with:
  • Safety checks
  • Audit logging
  • Automatic destination folder creation

Traceability

Every update is logged in:

output/update_reference_images.log

Updated reference images are saved in:

output/updated_reference_images/

Requirements

• Ubuntu 22.04 or later
• Python 3.10 or later
• Dependencies: PyQt6 (see requirements.txt)

Installation

Recommended: use a virtual environment.

python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Usage

1. Put Robot Framework XML logs (output*.xml) in input/.
2. Run the GUI from the project root:

python3 src/gui_main.py

3. The parser extracts images into output/, and the GUI loads them into the review table.
4. Selected updates are moved to output/updated_reference_images/ and logged in output/update_reference_images.log.

Notes

• If input/ or output/ is missing, the scripts create folders as needed.
• The GUI expects each failed keyword to contain three images (actual, reference, delta) in the log message.