fix: correct org and dataset paths #18
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR corrects dataset and organization paths throughout the codebase, updating references from "purpcorn" to "purpcode" and fixing dataset names to match their actual identifiers on Hugging Face.
- Updates dataset paths in SFT configuration files to use correct organization name and dataset identifiers
- Fixes organization references in evaluation code from "purpcorn" to "purpcode"
- Adds documentation for pre-generated datasets with correct paths and expands usage instructions
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| sft/ctxdistill_qwen32b.yaml | Updates dataset path to correct organization and removes .jsonl extension |
| sft/ctxdistill_qwen14b.yaml | Updates dataset path to correct organization and removes .jsonl extension |
| eval/compile_xscode/cwe2ovrf.py | Corrects organization name from "purpcorn" to "purpcode" in codeguru dataset references |
| README.md | Adds documentation for pre-generated datasets and expands rejection sampling instructions |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @ganler, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request primarily focuses on fixing incorrect organization and dataset paths across the codebase and enhancing documentation. My aim is to ensure that all references to datasets and organizations are accurate, thereby improving the reliability of data loading processes and providing clearer instructions for users.
Highlights
- Path Corrections: Corrected organization name from purpcorn to purpcode in eval/compile_xscode/cwe2ovrf.py for codeguru-rules dataset references.
- Dataset Path Updates: Updated dataset paths in sft/ctxdistill_qwen14b.yaml and sft/ctxdistill_qwen32b.yaml to reflect the correct and more specific dataset names (-57k and -55k suffixes).
- Documentation Enhancement: Improved the README.md by adding information about pre-generated datasets and making the "Rejection Sampling from Scratch" instructions collapsible for better readability.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request updates organization names from purpcorn to purpcode and corrects dataset paths across several files, including documentation, Python scripts, and YAML configurations. The changes look good and align with the PR's goal. I've added a couple of suggestions to improve clarity in the README and simplify a function call in a Python script.
| * [`Qwen2.5-14B-Instruct-1M`](https://huggingface.co/datasets/purpcode/ctxdistill-verified-Qwen2.5-14B-Instruct-1M-57k) via best of 8 | ||
| * [`Qwen2.5-32B-Instruct`](https://huggingface.co/datasets/purpcode/ctxdistill-verified-Qwen2.5-32B-Instruct-55k) via best of 4 |
There was a problem hiding this comment.
For clarity, it would be better if the link text for the pre-generated datasets matched the actual dataset names. The current text refers to the base model names, which might be confusing as the dataset names in the URLs are different.
| * [`Qwen2.5-14B-Instruct-1M`](https://huggingface.co/datasets/purpcode/ctxdistill-verified-Qwen2.5-14B-Instruct-1M-57k) via best of 8 | |
| * [`Qwen2.5-32B-Instruct`](https://huggingface.co/datasets/purpcode/ctxdistill-verified-Qwen2.5-32B-Instruct-55k) via best of 4 | |
| * [`purpcode/ctxdistill-verified-Qwen2.5-14B-Instruct-1M-57k`](https://huggingface.co/datasets/purpcode/ctxdistill-verified-Qwen2.5-14B-Instruct-1M-57k) via best of 8 | |
| * [`purpcode/ctxdistill-verified-Qwen2.5-32B-Instruct-55k`](https://huggingface.co/datasets/purpcode/ctxdistill-verified-Qwen2.5-32B-Instruct-55k) via best of 4 |
| collection = create_cwe_information() | ||
| if vuln_rules_type == "codeguru": | ||
| collection = create_codeguru_information("purpcorn/codeguru-rules") | ||
| collection = create_codeguru_information("purpcode/codeguru-rules") |
There was a problem hiding this comment.
The argument "purpcode/codeguru-rules" passed to create_codeguru_information is now identical to the function's default dataset_path value. You can simplify this call by removing the redundant argument.
| collection = create_codeguru_information("purpcode/codeguru-rules") | |
| collection = create_codeguru_information() |
No description provided.