feat(utils): add OpenAI provider support

[zhewang2001]

Added OpenAI provider support for utils/litellm.py and modified the generate_openai logic in eval/generate.py.
If generate uses openai backend with openai/deepseek/deepseek-r1 and then eval uses openai/o4-mini, the original approach would cause conflicts in getting api_base, thus leading to generate failure

[gemini-code-assist]

Summary of Changes

Hello @zhewang2001, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly improves the handling of OpenAI models within the system, particularly addressing issues where different OpenAI backends used for generation and evaluation could lead to API configuration conflicts. By centralizing and dynamically adjusting API key and base URL settings based on the model name, the changes ensure seamless operation across various OpenAI models, including specific handling for 'O-series' models' parameter requirements. This also refines environment variable loading for better control and consistency.

Highlights

• Improved OpenAI Model Configuration: The pull request introduces enhanced support for OpenAI models by refining how API keys and base URLs are managed. This change dynamically configures API parameters based on the model string, preventing conflicts when different OpenAI backends are used for generation and evaluation processes.
• Enhanced Model Compatibility: Specific adjustments have been made to handle 'O-series' OpenAI models (o1-, o3-, o4-) by ensuring that parameters like temperature and stop are correctly omitted or managed according to these models' unique API requirements.
• Refined Environment Variable Loading: The load_dotenv() call has been relocated from a global scope to specific functions, allowing for more precise control over when environment variables are loaded, particularly for bedrock models.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[Copilot]

Pull Request Overview

This PR adds OpenAI provider support to handle conflicts when using different OpenAI-based models across generation and evaluation phases. The changes ensure proper API configuration based on model naming patterns and handle special temperature requirements for O-series models.

• Adds OpenAI provider detection and configuration logic to utils/litellm.py
• Updates eval/generate.py to validate OpenAI model naming and handle API parameters dynamically
• Implements special handling for O-series models (o1, o3, o4) that don't support custom temperature/stop parameters

Reviewed Changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 4 comments.

File	Description
utils/litellm.py	Adds OpenAI provider support with conditional API configuration and O-series model parameter handling
eval/generate.py	Updates OpenAI generation logic with model validation and dynamic API parameter assignment

[gemini-code-assist]

Code Review

This pull request adds support for OpenAI providers in utils/litellm.py and eval/generate.py. The changes correctly address the issue of conflicting api_base configurations.

My review focuses on improving maintainability by addressing code duplication. I've identified two pieces of logic that are repeated across both modified files: one for configuring the OpenAI provider settings (api_key, api_base), and another for handling special parameters for O-series models. I've suggested refactoring these into shared helper functions. This will not only reduce redundancy but also fix a potential bug where the stop parameter was handled inconsistently for O-series models between the two files. Overall, the changes are good, and with the suggested refactoring, the code will be more robust and easier to maintain.