bless-crawl plugin support

[zees-dev]

Description

This pull request introduces a new plugin, BlessCrawl, for distributed web scraping.
It includes changes to the codebase, configuration, and documentation to integrate and support this new functionality.
The most important changes are grouped into three themes: feature integration, configuration updates, and documentation enhancements.

Depends on: blocklessnetwork/sdk-rust#23

Feature Integration:

• Added the BlessCrawl plugin, including its implementation in Rust (src/crawl/mod.rs) and JavaScript wrapper (src/crawl/crawl.js). The plugin supports scraping webpages, mapping links, and crawling websites with customizable options. [1] [2]
• Exposed the BlessCrawl plugin to the runtime by modifying the initialize_runtime function in src/lib.rs. This includes binding the plugin to the global context and evaluating its JavaScript module. [1] [2]
• Added a new example file, examples/crawl.js, demonstrating the usage of the BlessCrawl plugin with various operations such as scraping, mapping, and crawling.

Configuration Updates:

• Updated Cargo.toml to include the crawl feature in the default features and added a patch for the blockless-sdk crate.
• Updated the GitHub Actions workflow (.github/workflows/release.yml) to include the crawl feature in the build matrix.

Documentation Enhancements:

• Updated the README.md to document the BlessCrawl plugin, including its purpose, integration into the architecture diagram, and a new example command for compiling JavaScript using the plugin. [1] [2] [3] [4] [5]

[Copilot]

Pull Request Overview

This pull request adds support for a new distributed web scraping plugin called BlessCrawl. It introduces the BlessCrawl implementation in Rust, a JavaScript wrapper, runtime integration updates, documentation enhancements, and configuration changes.

• New Rust module and JavaScript binding for BlessCrawl.
• Runtime initialization and Cargo.toml updates for plugin integration.
• Documentation and example updates to illustrate usage.

Reviewed Changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/lib.rs	Initializes and exposes the BlessCrawl plugin in the global context.
src/crawl/mod.rs	Implements scraping, mapping, and crawling functionality.
src/crawl/crawl.js	Provides a JavaScript class wrapper for BlessCrawl.
examples/crawl.js	Demonstrates example usage of the BlessCrawl plugin.
README.md	Updates to documentation and architecture diagrams.
Cargo.toml	Adds the crawl feature to the default features and applies a dependency patch.
.github/workflows/release.yml	Updates the build matrix to include the crawl feature.