Add OpenJudge integration guide for VERL with async reward evaluation. #53
Conversation
Summary of ChangesHello @chr6192, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a new documentation file that serves as a detailed guide for integrating the OpenJudge reward system into VERL training. The primary goal is to enhance the flexibility and performance of reward computation by leveraging OpenJudge's modular grading capabilities and asynchronous evaluation, which is particularly beneficial for incorporating LLM-based assessments without creating performance bottlenecks. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds a comprehensive integration guide for using OpenJudge with VERL. The documentation is well-structured, detailed, and provides clear examples for users to follow. I've identified a few minor issues in the code snippets, such as missing imports and inconsistent naming, which could prevent the examples from running correctly. Addressing these will improve the clarity and usability of this excellent guide.
| results = await runner.arun_multiple_datasets(datasets) | ||
|
|
||
| # 4. Parse and aggregate | ||
| return self._aggregate_scores(results) |
There was a problem hiding this comment.
This example calls self._aggregate_scores(results), but the more detailed implementation in Section 4.2 uses _parse_and_aggregate. Using a consistent method name across examples will reduce confusion for readers.
| return self._aggregate_scores(results) | |
| return self._parse_and_aggregate(results) |
| try: | ||
| runner_results = await runner.arun_multiple_datasets(datasets) | ||
| except Exception as e: | ||
| logger.error(f"Grading failed: {e}") | ||
| # Return default scores on error | ||
| return self._create_error_results(prompt_to_samples, error=str(e)) | ||
|
|
There was a problem hiding this comment.
The logger object is used here for error logging, but it's not imported in the code example. This will cause a NameError when an exception occurs. Please add from loguru import logger at the top of the MyRewardFunction code snippet to ensure the example is runnable.
| ```bash | ||
|
|
||
| # Define paths | ||
| REWARD_FUN="${PATH_TO_DR}/reward/openjudge_reward_function.py" |
There was a problem hiding this comment.
The placeholder ${PATH_TO_DR} is used here but is not defined or explained, which could be confusing. Using a more descriptive placeholder like ${PATH_TO_YOUR_PROJECT} would improve clarity. It's also helpful to add a comment explaining that this path needs to be replaced.
| REWARD_FUN="${PATH_TO_DR}/reward/openjudge_reward_function.py" | |
| REWARD_FUN="${PATH_TO_YOUR_PROJECT}/reward/openjudge_reward_function.py" |
| def _dump_generations(self, messages, inputs, outputs, scores, reward_extra_infos_dict, dump_path, all_details=None): | ||
| """ | ||
| Dump training samples with details. | ||
|
|
||
| Extracts details from reward_extra_info["details"] if not provided separately. | ||
| """ | ||
| os.makedirs(dump_path, exist_ok=True) | ||
| filename = os.path.join(dump_path, f"{self.global_steps}.jsonl") | ||
|
|
||
| # Extract details from dict if not provided | ||
| if all_details is None and "details" in reward_extra_infos_dict: | ||
| all_details = reward_extra_infos_dict["details"] | ||
| # Remove from dict to avoid duplication | ||
| reward_extra_infos_dict = { | ||
| k: v for k, v in reward_extra_infos_dict.items() | ||
| if k != "details" | ||
| } | ||
|
|
||
| # Prepare data | ||
| n = len(inputs) | ||
| base_data = { | ||
| "messages": messages, | ||
| "input": inputs, | ||
| "output": outputs, | ||
| "score": scores, | ||
| "step": [self.global_steps] * n, | ||
| } | ||
|
|
||
| # Add reward_extra_info fields | ||
| for k, v in reward_extra_infos_dict.items(): | ||
| if len(v) == n: | ||
| base_data[k] = v | ||
|
|
||
| # Add details if available | ||
| if all_details is not None and len(all_details) == n: | ||
| base_data["details"] = all_details | ||
|
|
||
| # Write JSONL | ||
| lines = [] | ||
| for i in range(n): | ||
| entry = {k: v[i] for k, v in base_data.items()} | ||
| lines.append(json.dumps(entry, ensure_ascii=False)) | ||
|
|
||
| with open(filename, "w") as f: | ||
| f.write("\n".join(lines) + "\n") | ||
| ``` |
There was a problem hiding this comment.
The _dump_generations method in this example uses the os and json modules, but they are not imported. This will lead to a NameError. To make the example correct, please add import os and import json at the beginning of the MyRayPPOTrainer code snippet.
| A: Create fresh `GradingRunner` per training step (see [Section 5.1](#critical-event-loop-management)) | ||
|
|
||
| **Q: Getting zero scores?** | ||
| A: Enable debug logging: `logger.add(sys.stderr, level="DEBUG")` |
There was a problem hiding this comment.
The troubleshooting advice suggests using logger.add(sys.stderr, level="DEBUG"), but this will fail without the necessary imports for logger and sys. To make this guidance more helpful, please include the required imports in the example.
A clearer example would be:
# Add these imports at the top of your script
import sys
from loguru import logger
# Then enable debug logging
logger.add(sys.stderr, level="DEBUG")
OpenJudge Version
0.2.0
Description
Add OpenJudge integration guide for VERL with async reward evaluation.
Checklist
Please check the following items before code is ready to be reviewed.
pre-commit run --all-filescommand