feat(eval): unify sec code oracles in one pipeline #16
Conversation
nirav0999
commented
Aug 9, 2025
nirav0999
commented
- removed dead code
- integrated secure code oracles API from the main repo
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @nirav0999, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
I've implemented a significant refactor to unify the secure code oracle evaluation pipeline. This change centralizes the process of analyzing generated code for security vulnerabilities, integrating various static analysis tools into a single, efficient workflow. The primary goal is to streamline the evaluation process, improve performance through parallelization and intelligent batching, and enhance the overall maintainability of the secure code assessment framework.
Highlights
- Unified Secure Code Oracle Pipeline: I've refactored and centralized the secure code oracle evaluation process into a unified pipeline. This change streamlines how we analyze generated code for potential vulnerabilities, making the process more efficient and maintainable.
- Parallelized Static Analysis with Batch Processing: The new pipeline now supports parallel execution of static analysis tools like CodeGuru and CodeQL. I've implemented robust batching mechanisms that can split code samples by either the number of code blocks or lines of code, significantly improving performance for large datasets.
- Integration of CodeGuru and CodeQL Oracles: I've integrated the existing CodeGuru and CodeQL oracle APIs directly into this new pipeline, ensuring that their capabilities are fully leveraged within the unified evaluation framework. This includes handling their results and filtering them based on severity.
- Introduction of New Utility Functions: Several utility functions have been added to support the new pipeline, including methods for extracting code snippets, handling temporary directories, managing AWS regions, checking minimum severity levels, and performing base64 encoding/decoding and file zipping.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Pull Request Overview
Integrates secure code oracle functionality from the main repository into a unified pipeline, replacing placeholder TODO comments with actual implementation. The change removes dead code and consolidates security analysis capabilities.
- Implements comprehensive secure code evaluation pipeline with support for multiple static analyzers
- Adds utility functions for AWS integration, severity checking, and file processing
- Provides batch processing capabilities for parallel analysis of code samples
Reviewed Changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.
| File | Description |
|---|---|
| eval/oracles/secure_code_oracles_utils.py | Adds utility functions for AWS region detection, severity validation, base64 encoding/decoding, and file compression |
| eval/oracles/secure_code_oracles.py | Implements complete secure code evaluation pipeline with batch processing, parallel execution, and support for CodeGuru/CodeQL analyzers |
There was a problem hiding this comment.
Code Review
This pull request introduces a new pipeline for running secure code oracles, unifying them into a single workflow. The changes are substantial, adding new modules for data processing, batching, and parallel execution of static analyzers. My review focuses on improving the robustness, maintainability, and correctness of this new pipeline. I've provided suggestions to handle edge cases like empty inputs, improve path handling for cross-platform compatibility, refactor for clarity and consistency, and make parts of the pipeline more configurable. Addressing these points will make the new evaluation logic more resilient and easier to maintain.
