-
Notifications
You must be signed in to change notification settings - Fork 12
Open
Labels
enhancementNew feature or requestNew feature or requesthelp wantedExtra attention is neededExtra attention is needed
Description
Summary
Allow users to define custom metrics via YAML configuration instead of Python code.
Tasks
- Define YAML schema for custom metrics (name, formula, thresholds).
- Parse and register metrics from
agentunit.yamlor similar config file. - Support simple formulas referencing built-in metrics (e.g.,
(faithfulness + tool_success) / 2). - Document the schema in docs.
Acceptance criteria
- Custom metrics defined in YAML appear in results.
- Invalid YAML produces clear error messages.
- Documentation includes examples.
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or requesthelp wantedExtra attention is neededExtra attention is needed