Skip to content

Conversation

@nikivanstein
Copy link
Contributor

Motivation

  • Add a new problem class that lets users supply a 25-function suite and evaluates how well that suite discriminates between five Nevergrad optimizers using a Friedman-ranks IQR metric.
  • Transform the provided evaluation script into a BLADE Problem so suites can be authored as solution code and scored inside the platform.

Description

  • Add iohblade/problems/ng_friedman_suite.py implementing NG_FriedmanSuite with prompts, example/format guidance, validation (_validate_suite), benchmarking (run_benchmark + NG_Evaluator) and scoring (get_friedman_val).
  • The problem accepts problems and meta_dims (25 entries each) as authored code and wraps each function with ioh on [-5,5]^d, runs the Nevergrad optimizers, collects logs via ioh.logger.Analyzer and iohinspector, and computes the mean IQR of Friedman ranks as the fitness.
  • Add required runtime dependencies and imports strings for evaluation environments (nevergrad, ioh, iohinspector, polars, scipy) and configure the problem interface (func_name = "problems", empty init_inputs/func_inputs/func_outputs).
  • Export NG_FriedmanSuite from iohblade/problems/__init__.py so it is visible under the problems package.

Testing

  • Ran formatting and import-order fixes with uv run isort iohblade/ and code formatting with uv run black iohblade/, which reformatted files and updated the new module.
  • Ran the test suite with uv run pytest tests/; collection failed due to missing external packages (ModuleNotFoundError during collection). The immediate missing modules reported include cloudpickle, ioh, numpy, httpx, mlflow, and matplotlib, so pytest could not complete.
  • Unit/e2e tests were not able to exercise the new problem because the environment lacks the runtime dependencies required to run NG_FriedmanSuite (nevergrad/ioh/iohinspector/etc.).

Codex Task

@nikivanstein nikivanstein changed the base branch from main to competition/benchdesign January 30, 2026 10:25
@nikivanstein nikivanstein merged commit 0f1462f into competition/benchdesign Jan 30, 2026
8 checks passed
@nikivanstein nikivanstein deleted the feature/codex-add-new-problem-class-to-iohblade branch January 30, 2026 10:25
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d2c5740481

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@codecov
Copy link

codecov bot commented Jan 30, 2026

Codecov Report

❌ Patch coverage is 0% with 122 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
iohblade/problems/ng_friedman_suite.py 0.00% 118 Missing ⚠️
iohblade/problems/__init__.py 0.00% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants