Refactor ep upload to use Fireworks SDK + Auto-generate flags
#379
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…eworks-sdk-auto-generate-flags # Conflicts: # development/CONTRIBUTING.md # docs/cli_reference/cli_overview.mdx # eval_protocol/auth.py # eval_protocol/cli_commands/utils.py # tests/test_auth.py
dphuang2
changed the title
ep upload to use Fireworks SDK + Auto-generate flags
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…eworks-sdk-auto-generate-flags
…eworks-sdk-auto-generate-flags
…les and remove API_KEY filtering logic
…eworks-sdk-auto-generate-flags
- Introduced `_prompt_select_secrets` function to allow users to select environment variables for upload as secrets. - Implemented fallback selection method for non-interactive environments. - Updated `upload_command` to utilize the new secret selection logic. - Enhanced user experience with improved prompts and error handling. - Added `_get_questionary_style` function for consistent CLI styling.
| ) | ||
|
|
||
| # Auto-generate flags from SDK Fireworks().evaluators.create() signature | ||
| create_evaluator_fn = Fireworks().evaluators.create |
There was a problem hiding this comment.
Fireworks SDK client created at CLI parser build time
The Fireworks() client is instantiated during parser construction (in _configure_parser), which happens for every CLI invocation including --help. This means users who haven't configured their FIREWORKS_API_KEY environment variable will be unable to run eval-protocol --help or any command without credentials. The add_args_from_callable_signature function only needs the method signature for introspection, so instantiating a live client with potential network/auth requirements is unnecessary and breaks the CLI for unconfigured users.
Additional Locations (1)
dphuang2
deleted the
dhuang/dxe-442-refactor-ep-upload-to-use-fireworks-sdk-auto-generate-flags
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Modernizes evaluator creation and secret management via the Fireworks SDK and simplifies the CLI surface.
upload): adds auto-generated flags fromFireworks().evaluators.create(with aliases/help overrides), consolidates--force, and introduces interactive/non-interactive secret selection (Questionary style, masking) pulling from.envand env varsEvaluator.create()now uses SDK for get/delete/create, signed URL retrieval, and upload validation; removes large amounts of legacy code (preview/multi-metric loaders, folder parsing)secrets.get/create/update/delete) and handles NotFound/500 cases via typed errorsdeploy,preview,deploy-mcp, andrunpaths and their tests; streamlines arg parsing/helpfireworks-aito1.0.0a20Written by Cursor Bugbot for commit a785e66. This will update automatically on new commits. Configure here.