download: cdn friendly & getfile support sign & optimize listbucket2 & support intelligent tiering

[YangSen-qn]

变更背景描述

jira issue链接

主要变更点

• fix bug
• new feature
• 不兼容变更

Checklist

• 已自测
• 已更新Readme

[gemini-code-assist]

Summary of Changes

Hello @YangSen-qn, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the storage management capabilities by integrating 'Intelligent Tiering' as a new storage class, allowing for more dynamic and cost-effective data management. It also refines the download process for better compatibility with CDNs and strengthens security for private bucket downloads. Concurrently, the bucket listing functionality has been optimized for improved performance and flexibility, alongside general improvements to flow control and error handling.

Highlights

• Intelligent Tiering Support: Introduced support for 'Intelligent Tiering' storage type across various commands including asyncfetch, listbucket2, chlifecycle, batchchtype, fput, qupload, qupload2, rput, and sync. This includes updating command-line flags, API structures, and documentation to recognize and utilize the new storage type.
• CDN Friendly Downloads and GetFile Signing: Improved download functionality to be more CDN-friendly by changing network file information retrieval from HEAD requests to GET requests with a 'Range: bytes=0-0' header, which can leverage CDN caching. Additionally, getfile operations for private buckets now correctly include signing for enhanced security, and the 'public' flag in download commands has been deprecated.
• Optimized Bucket Listing (listbucket2): Refactored the bucket listing mechanism to support pagination and provide more flexible retrieval of bucket information. The allBuckets function now uses a callback handler and distinguishes between detailed (single call) and paginated (for bucket names) listing, improving efficiency for different use cases.
• Flow Control and Error Handling Improvements: Enhanced the robustness of flow control mechanisms by adding checks for empty work data in the producer and scanner errors in the work provider, preventing potential issues during data processing.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

本次 PR 包含多项功能新增和优化，整体实现不错。主要包括：

• 在多个命令中增加了对“智能分层存储”的支持。
• 优化了下载流程，使其对 CDN 更友好，并为 getfile URL 添加了签名支持。
• 优化了 listbucket2 命令，改用分页 API，提升了列举大量存储空间时的性能和内存效率。

代码变更大部分是好的，但我发现了一个由于复制粘贴导致的严重 bug，可能会导致错误的生命周期规则被设置。另外，在一个命令的帮助信息中有一个小拼写错误。请修复这些问题。

[gemini-code-assist]

Summary of Changes

Hello @YangSen-qn, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the storage management and download capabilities of the system. It introduces comprehensive support for Intelligent Tiering storage, allowing users more granular control over object storage classes. Additionally, download operations are made more efficient and secure through CDN-friendly practices and automatic URL signing for private buckets. The bucket listing mechanism has also been optimized for better performance and scalability.

Highlights

• Intelligent Tiering Storage Support: Introduced support for 'Intelligent Tiering' storage type across various commands and APIs, including async fetch, bucket listing, lifecycle management, and upload operations. This adds a new storage option (type 5) for managing object lifecycles and storage classes.
• CDN Friendly Downloads: Modified the GetNetworkFileInfo function to use a GET request with a Range: bytes=0-0 header instead of a HEAD request. This change is intended to be more CDN-friendly, allowing CDNs to cache responses and reduce direct origin requests, especially for file size checks.
• Signed Downloads for Private Buckets: Enhanced the getfile download method to automatically sign URLs for private buckets, ensuring secure access to objects that are not publicly accessible.
• Optimized Bucket Listing (listbucket2): Refactored the bucket listing functionality to support pagination (apiVersion=v4) and a callback-based approach, improving performance and flexibility when retrieving bucket information, especially for detailed listings.
• Deprecation of --public Flag: The --public flag in download commands has been deprecated. The system now dynamically determines if a bucket is public or private to handle download operations appropriately.
• Improved Flow Error Handling: Added checks in the work producer flow to handle cases where no data is read or scanner errors occur, preventing potential panics and providing more robust error reporting.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[YangSen-qn]

@xgopilot /review

[gemini-code-assist]

Code Review

这个 Pull Request 包含多项重要更新。首先，它增加了对智能分层存储类型的支持，并相应地更新了多个命令和文档。其次，为了对 CDN 更友好，优化了获取网络文件信息的方式，从 HEAD 请求改为带 Range 头的 GET 请求。此外，getfile API 现在支持对私有空间资源进行签名访问。最后，对 listbucket2 命令进行了重构，通过支持分页来优化获取存储空间列表的性能。

代码整体质量不错，但发现一个严重的复制粘贴错误，可能导致生命周期规则设置不正确。另外，在一个命令的帮助信息中有一个小拼写错误。建议修复这些问题。

[xgopilot]

Code Review Summary

Critical Issues Found: This PR introduces intelligent tiering storage support but contains 1 critical bug that breaks the feature, plus several security and performance concerns.

🚨 Must Fix Before Merge

1. iqshell/storage/object/lifecycle.go:48-49 - Wrong variable (ToDeepArchiveAfterDays instead of ToIntelligentTieringAfterDays) prevents intelligent tiering lifecycle from working
2. iqshell/common/utils/file.go:142 - Missing HTTP status validation allows potential security bypass via crafted error responses
3. iqshell/common/utils/file.go:167 - Logic error checking wrong variable (contentLength vs etag)

⚠️ Should Fix

4. iqshell/storage/object/operations/lifecycle.go:90 - Missing validation for ToIntelligentTieringAfterDays (inconsistent with batch operation)
5. iqshell/storage/object/operations/lifecycle.go:117 - Incomplete job ID hash missing ToArchiveIRAfterDays parameter
6. iqshell/storage/object/download/operations/batch.go:122 - N+1 API calls for bucket info (recommend caching)

📝 Documentation Gaps

• Missing --to-intelligent-tiering-after-days documentation in docs/chlifecycle.md and docs/batchchlifecycle.md

Recommendation: Fix critical bugs before merging. The intelligent tiering feature is well-integrated across commands but needs these corrections to function properly.

---

👉 Want me to re-run this review? Just type /review in the comments! For more usage examples, visit CodeAgent GitHub Repository.

[xgopilot]

Code Review Summary

Reviewed PR adding intelligent tiering storage support. Found 1 critical bug, several important issues, and performance/security concerns.

Critical Issues:

• Lifecycle code uses wrong variable (ToDeepArchiveAfterDays instead of ToIntelligentTieringAfterDays) - breaks the feature
• Missing HTTP status validation in file info retrieval - security risk

Important Fixes Needed:

• N+1 API calls for bucket info (10,000+ redundant calls in batch operations)
• Incomplete lifecycle validation and logging
• ETag validation logic error

See inline comments for details and suggested fixes.

[qiniu-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: YangSen-qn

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [YangSen-qn]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[YangSen-qn]

/test qiniu-qshell-presubmit-integration-test