Matrix Benchmarking

(documentation generated by Claude Code)

A comprehensive benchmarking and performance analysis framework designed for OpenShift/Kubernetes environments. Matrix Benchmarking provides tools for downloading, parsing, analyzing, and visualizing performance data with support for long-term storage and advanced statistical analysis.

Features

Core Functionality

• Benchmark Execution: Run performance benchmarks in Kubernetes/OpenShift environments
• Data Collection: Download and scrape benchmark data from multiple sources
• Result Parsing: Parse Prometheus metrics and other benchmark outputs
• Data Analysis: Statistical analysis with multiple methods (standard deviation, Z-score, Hunter anomaly detection)
• Visualization: Generate interactive web-based reports and visualizations
• Long-term Storage: Schema generation and data management for historical analysis

Supported Data Sources

• Middleware CI: Scrape data from middleware continuous integration systems
• OpenShift CI: Integration with OpenShift CI pipelines
• S3 Storage: Download benchmark data from S3-compatible storage
• Prometheus: Direct integration with Prometheus metrics

Analysis Methods

• Hunter: Advanced anomaly detection algorithm
• Standard Deviation: Statistical variance analysis
• Z-Score: Standardized score analysis
• Custom Methods: Extensible framework for additional analysis approaches

Installation

Prerequisites

• Python 3.7+
• Access to Kubernetes/OpenShift cluster (for benchmark execution)
• Prometheus instance (for metrics collection)

Dependencies

Install required Python packages:

pip install -r requirements.txt

Key dependencies include:

• pandas - Data manipulation and analysis
• dash - Web-based interactive visualizations
• plotly - Plotting and charting
• fire - CLI framework
• prometheus_api_client - Prometheus integration
• pydantic - Data validation
• opensearch-py - OpenSearch integration
• boto3 - AWS S3 integration

Usage

Matrix Benchmarking provides a command-line interface with multiple subcommands:

Basic Commands

# Run benchmarks
python -m matrix_benchmarking benchmark --workload <workload_name> --run

# Download benchmark data
python -m matrix_benchmarking download --source <source_type>

# Parse benchmark results
python -m matrix_benchmarking parse --results-dir <path_to_results>

# Generate visualizations
python -m matrix_benchmarking visualize --results-dir <path_to_results>

# Analyze data with long-term storage
python -m matrix_benchmarking analyze_lts --method <analysis_method>

Long-term Storage Operations

# Upload data to long-term storage
python -m matrix_benchmarking upload_lts --data-dir <path_to_data>

# Download from long-term storage
python -m matrix_benchmarking download_lts --output-dir <output_path>

# Generate storage schema
python -m matrix_benchmarking generate_lts_schema --config <config_file>

Getting Help

# General help
python -m matrix_benchmarking --help

# Command-specific help
python -m matrix_benchmarking benchmark --help

Configuration

Matrix Benchmarking uses YAML configuration files for:

• Benchmark definitions
• Data source configurations
• Analysis parameters
• Visualization settings

Example configuration structure:

workload:
  name: "performance_test"
  parameters:
    # Workload-specific parameters

data_sources:
  prometheus:
    url: "http://prometheus:9090"

analysis:
  methods:
    - hunter
    - stdev
    - zscore

Architecture

Module Structure

• matrix_benchmarking/ - Main package
  • benchmark.py - Benchmark execution logic
  • download.py - Data downloading orchestration
  • parse.py - Result parsing utilities
  • visualize.py - Visualization generation
  • analyze_lts.py - Long-term storage analysis
  • upload_lts.py - Long-term storage uploads
  • download_lts.py - Long-term storage downloads
  • generate_lts_schema.py - Schema generation

Submodules

• downloading/ - Data acquisition

  • scrape/ - Web scraping utilities
    • middleware_ci.py - Middleware CI integration
    • ocp_ci.py - OpenShift CI integration
    • s3.py - S3 storage integration

• exec/ - Execution utilities

  • kube.py - Kubernetes operations
  • prom.py - Prometheus queries
  • common.py - Shared execution logic

• parsing/ - Data parsing

  • prom.py - Prometheus metrics parsing

• plotting/ - Visualization

  • ui/ - User interface components
    • web.py - Web-based dashboards
    • report.py - Report generation

• store/ - Data storage

  • simple.py - Simple file-based storage
  • prom_db.py - Prometheus database integration

• analyze/ - Analysis methods

  • method/ - Statistical analysis implementations
    • hunter.py - Hunter anomaly detection
    • stdev.py - Standard deviation analysis
    • zscore.py - Z-score analysis

Web Interface

Matrix Benchmarking includes a web-based interface for:

• Interactive data exploration
• Real-time visualization updates
• Report generation and sharing
• Performance trend analysis

Access the web interface by running the visualize command with web output enabled.

Development

Project Structure

matrix_benchmarking/
├── matrix_benchmarking/          # Main package
│   ├── analyze/                  # Analysis methods
│   ├── downloading/              # Data acquisition
│   ├── exec/                     # Execution utilities
│   ├── parsing/                  # Data parsing
│   ├── plotting/                 # Visualization
│   └── store/                    # Data storage
├── utils/                        # Utility modules
├── requirements.txt              # Python dependencies
└── README.md                     # This file

Extending Analysis Methods

To add a new analysis method:

1. Create a new file in matrix_benchmarking/analyze/method/
2. Implement the analysis interface
3. Register the method in the analysis framework
4. Add tests and documentation

Adding Data Sources

To support a new data source:

1. Create a scraper in matrix_benchmarking/downloading/scrape/
2. Implement the downloading interface
3. Add configuration support
4. Update documentation

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests for new functionality
5. Submit a pull request

License

This project is part of the TOPSAIL project under the OpenShift PSAP organization.

Support

For issues, questions, or contributions, please refer to the OpenShift PSAP team or create an issue in the repository.