Skip to content

Support custom metrics via YAML configuration #19

@aviralgarg05

Description

@aviralgarg05

Summary

Allow users to define custom metrics via YAML configuration instead of Python code.

Tasks

  • Define YAML schema for custom metrics (name, formula, thresholds).
  • Parse and register metrics from agentunit.yaml or similar config file.
  • Support simple formulas referencing built-in metrics (e.g., (faithfulness + tool_success) / 2).
  • Document the schema in docs.

Acceptance criteria

  • Custom metrics defined in YAML appear in results.
  • Invalid YAML produces clear error messages.
  • Documentation includes examples.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions