diff --git a/docs/Protocol/Protocol_v2/Sequencer/EventCollector.md b/docs/Protocol/Protocol_v2/Sequencer/EventCollector.md index 472736b..3098b91 100644 --- a/docs/Protocol/Protocol_v2/Sequencer/EventCollector.md +++ b/docs/Protocol/Protocol_v2/Sequencer/EventCollector.md @@ -45,7 +45,7 @@ When a match is identified, it indicates the end of the submission window, promp ### Batch Submission After the batches are organized, they are iterated over, and the relevant data is transformed into submission details. These details are then placed into the Finalizer Queue for further processing. -The [**Finalizer**](https://github.com/PowerLoom/submission-sequencer-finalizer), an auto-scaled component that follows, retrieves and processes data from the Finalizer Queue, completing the batch processing pipeline. This architecture enables the system to scale effectively, managing submission tasks in parallel and optimizing both performance and throughput. +The [**Finalizer**](https://github.com/powerloom/submission-sequencer-finalizer), an auto-scaled component that follows, retrieves and processes data from the Finalizer Queue, completing the batch processing pipeline. This architecture enables the system to scale effectively, managing submission tasks in parallel and optimizing both performance and throughput. ## On-Chain Updates via Relayer diff --git a/docs/Protocol/Protocol_v2/relay.md b/docs/Protocol/Protocol_v2/relay.md index 9c5e0fa..a29a8d3 100644 --- a/docs/Protocol/Protocol_v2/relay.md +++ b/docs/Protocol/Protocol_v2/relay.md @@ -9,7 +9,7 @@ Circuit relayers are currently not being supported for the pre-mainnet and mainn The trusted list of sequeuncer interfaces can be found on our Github repository in this JSON file: -https://github.com/PowerLoom/snapshotter-lite-local-collector/blob/feat/trusted-relayers/sequencers.json +https://github.com/powerloom/snapshotter-lite-local-collector/blob/feat/trusted-relayers/sequencers.json ::: ## Recap diff --git a/docs/Protocol/Protocol_v2/sequencer.md b/docs/Protocol/Protocol_v2/sequencer.md index 9599290..651313f 100644 --- a/docs/Protocol/Protocol_v2/sequencer.md +++ b/docs/Protocol/Protocol_v2/sequencer.md @@ -8,7 +8,7 @@ sidebar_position: 1 :::warning The sequencer listening interfaces for specific data markets are listed in the following trusted sequencer JSON file hosted on the Powerloom Github repository: -https://github.com/PowerLoom/snapshotter-lite-local-collector/blob/feat/trusted-relayers/sequencers.json +https://github.com/powerloom/snapshotter-lite-local-collector/blob/feat/trusted-relayers/sequencers.json DO NOT attempt to connect to any other sequencer interfaces supplied by anyone claiming to represent Powerloom if they are not listed in the above file. ::: @@ -100,7 +100,7 @@ The libp2p listening interface is loadbalanced on Layer 4 of the networking stac :::info * Read more: [Sequencer: Libp2p Listener](/Protocol/Protocol_v2/Sequencer/Listener.md) -* [Github](https://github.com/PowerLoom/libp2p-submission-sequencer-listener) +* [Github](https://github.com/powerloom/libp2p-submission-sequencer-listener) ::: ### Autoscaled dequeuers @@ -109,7 +109,7 @@ These dequeuers are autoscaled by an event driven architecture and intermediated :::info * Read more: [Sequencer: Dequeuer](/Protocol/Protocol_v2/Sequencer/Dequeuer.md) -* [Github](https://github.com/PowerLoom/sequencer-dequeuer) +* [Github](https://github.com/powerloom/sequencer-dequeuer) ::: ### Event collector @@ -126,7 +126,7 @@ This works as the system clock of the sequencer setup as a whole. :::info * Read more: [Sequencer: Event Collector](/Protocol/Protocol_v2/Sequencer/EventCollector.md) -* [Github](https://github.com/PowerLoom/submission-sequencer-event-collector/) +* [Github](https://github.com/powerloom/submission-sequencer-event-collector/) ::: ### Autsocaled Batch Finalizers @@ -138,5 +138,5 @@ These are another set of autoscaled pods that are responsible for :::info * Read more: [Sequencer: Batch Finalizer](/Protocol/Protocol_v2/Sequencer/Finalizer.md) -* [Github](https://github.com/PowerLoom/submission-sequencer-batch-finalizer/) +* [Github](https://github.com/powerloom/submission-sequencer-batch-finalizer/) ::: diff --git a/docs/Protocol/Specifications/Epoch.md b/docs/Protocol/Specifications/Epoch.md index b58641a..f1cf7e6 100644 --- a/docs/Protocol/Specifications/Epoch.md +++ b/docs/Protocol/Specifications/Epoch.md @@ -65,4 +65,4 @@ The Force Consensus service operates slightly differently than the Epoch Generat ![Force consensus mechanism](https://raw.githubusercontent.com/PowerLoom/onchain-consensus/feat/force_consensus_only_relevant_projects/docs/images/force_consensus.png) -- [Epoch Generator Source Code on Github](https://github.com/Powerloom/onchain-consensus/blob/63d09aa9ab1d98a2fed55e05b7760c12692fea83/epoch_generator.py) +- [Epoch Generator Source Code on Github](https://github.com/powerloom/onchain-consensus/blob/63d09aa9ab1d98a2fed55e05b7760c12692fea83/epoch_generator.py) diff --git a/docs/Protocol/Specifications/Snapshotter/components.md b/docs/Protocol/Specifications/Snapshotter/components.md index b05602d..757fd16 100644 --- a/docs/Protocol/Specifications/Snapshotter/components.md +++ b/docs/Protocol/Specifications/Snapshotter/components.md @@ -12,19 +12,19 @@ The System Event Detector tracks events triggered on the protocol state contract ## Process Hub Core -The Process Hub Core, defined in [`process_hub_core.py`](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/process_hub_core.py), serves as the primary process manager in the snapshotter. -- Operated by the CLI tool [`processhub_cmd.py`](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processhub_cmd.py), it is responsible for starting and managing the `SystemEventDetector` and `ProcessorDistributor` processes. -- Additionally, it spawns the base snapshot and aggregator workers required for processing tasks from the `powerloom-backend-callback` queue. The number of workers and their configuration path can be adjusted in [`config/settings.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json). +The Process Hub Core, defined in [`process_hub_core.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/process_hub_core.py), serves as the primary process manager in the snapshotter. +- Operated by the CLI tool [`processhub_cmd.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processhub_cmd.py), it is responsible for starting and managing the `SystemEventDetector` and `ProcessorDistributor` processes. +- Additionally, it spawns the base snapshot and aggregator workers required for processing tasks from the `powerloom-backend-callback` queue. The number of workers and their configuration path can be adjusted in [`config/settings.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json). ## Processor Distributor -The Processor Distributor, defined in [`processor_distributor.py`](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py), is initiated using the `processhub_cmd.py` CLI. +The Processor Distributor, defined in [`processor_distributor.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py), is initiated using the `processhub_cmd.py` CLI. - It loads the preloader, base snapshotting, and aggregator config information from the settings file. -- It reads the events forwarded by the event detector to the `f'powerloom-event-detector:{settings.namespace}:{settings.instance_id}'` RabbitMQ queue bound to a topic exchange as configured in `settings.rabbitmq.setup.event_detector.exchange`([code-ref: RabbitMQ exchanges and queue setup in pooler](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py)). -- It creates and distributes processing messages based on the preloader configuration present in `config/preloader.json`, the project configuration present in [`config/projects.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json), and [`config/aggregator.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/aggregator.example.json). +- It reads the events forwarded by the event detector to the `f'powerloom-event-detector:{settings.namespace}:{settings.instance_id}'` RabbitMQ queue bound to a topic exchange as configured in `settings.rabbitmq.setup.event_detector.exchange`([code-ref: RabbitMQ exchanges and queue setup in pooler](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py)). +- It creates and distributes processing messages based on the preloader configuration present in `config/preloader.json`, the project configuration present in [`config/projects.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json), and [`config/aggregator.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/aggregator.example.json). - For [`EpochReleased` events](/Protocol/Specifications/Epoch#1-epoch_released), it forwards such messages to base snapshot builders for data source contracts as configured in `config/projects.json` for the current epoch information contained in the event. ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L1077-L1115 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L1077-L1115 ``` @@ -34,44 +34,44 @@ https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d0 Preloaders often fetch and cache large volumes of data, such as all the transaction receipts for a block on the data source blockchain. In such cases, a single worker is often insufficient to feasibly fetch the data for timely base snapshot generation and subsequent aggregate snapshot generations to reach a consensus. -To address this, workers are defined as `delegate_tasks` in [`config/preloader.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/preloader.json). The [Process Hub Core](#process-hub-core) then launches a specific number of workers, as defined in the primary settings file, [`config/settings.json`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json), under the key `callback_worker_config.num_delegate_workers`. +To address this, workers are defined as `delegate_tasks` in [`config/preloader.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/preloader.json). The [Process Hub Core](#process-hub-core) then launches a specific number of workers, as defined in the primary settings file, [`config/settings.json`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json), under the key `callback_worker_config.num_delegate_workers`. ```python reference -https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/preloader.json#L19-L25 +https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/preloader.json#L19-L25 ``` ```python reference -https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json#L86-L90 +https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json#L86-L90 ``` Delegation workers operate over a simple request-response queue architecture over RabbitMQ. ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py#L243-L254 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py#L243-L254 ``` One of the preloaders bundled with this snapshotter peer is tasked with fetching all the transaction receipts within a given epoch's block range and because of the volume of data to be fetched it delegates this work to a bunch of delegation worker -* The Preloader: [snapshotter/utils/preloaders/tx_receipts/preloader.py](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/preloaders/tx_receipts/preloader.py). -* The Delegation Workers: [snapshotter/utils/preloaders/tx_receipts/delegated_worker/tx_receipts.py](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/preloaders/tx_receipts/delegated_worker/tx_receipts.py) +* The Preloader: [snapshotter/utils/preloaders/tx_receipts/preloader.py](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/preloaders/tx_receipts/preloader.py). +* The Delegation Workers: [snapshotter/utils/preloaders/tx_receipts/delegated_worker/tx_receipts.py](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/preloaders/tx_receipts/delegated_worker/tx_receipts.py) As a common functionality shared by all preloaders that utilize delegate workers, this logic is present in the generic class `DelegatorPreloaderAsyncWorker` that all such preloaders inherit. Here you can observe the workload is sent to the delegation workers ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/generic_delegator_preloader.py#L191-L210 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/generic_delegator_preloader.py#L191-L210 ``` Upon sending out the workloads tagged by unique request IDs, the delegator sets up a temporary exclusive queue to which only the delegation workers meant for the task type push their responses. ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/generic_delegator_preloader.py#L159-L186 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/generic_delegator_preloader.py#L159-L186 ``` The corresponding response being pushed by the delegation workers can be found here in the generic class `DelegateAsyncWorker` that all such workers should inherit from: ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/delegate_worker.py#L74-L84 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/delegate_worker.py#L74-L84 ``` ## Callback Workers @@ -81,11 +81,11 @@ The callback workers are the ones that build the base snapshot and aggregation s They listen to new messages on the RabbitMQ topic exchange as described in the following configuration, and the topic queue's initialization is as follows. ```python reference -https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json#L33-L55 +https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/settings.example.json#L33-L55 ``` ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py#L182-L213 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/init_rabbitmq.py#L182-L213 ``` Upon receiving a message from the processor distributor after preloading is complete, the workers do most of the heavy lifting along with some sanity checks and then call the `compute()` callback function on the project's configured snapshot worker class to transform the dependent data points as cached by the preloaders to finally generate the base snapshots. @@ -96,12 +96,12 @@ Upon receiving a message from the processor distributor after preloading is comp ## RPC Helper -Extracting data from the blockchain state and generating the snapshot can be a complex task. The `RpcHelper`, defined in [`utils/rpc.py`](https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/rpc.py), has a bunch of helper functions to make this process easier. It handles all the `retry` and `caching` logic so that developers can focus on efficiently building their use cases. +Extracting data from the blockchain state and generating the snapshot can be a complex task. The `RpcHelper`, defined in [`utils/rpc.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/rpc.py), has a bunch of helper functions to make this process easier. It handles all the `retry` and `caching` logic so that developers can focus on efficiently building their use cases. ## Core API -This component is one of the most important and allows you to access the finalized protocol state on the smart contract running on the anchor chain. Find it in [`core_api.py`](https://github.com/PowerLoom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/core_api.py). +This component is one of the most important and allows you to access the finalized protocol state on the smart contract running on the anchor chain. Find it in [`core_api.py`](https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/core_api.py). In the end, diff --git a/docs/Protocol/Specifications/Snapshotter/preloading.md b/docs/Protocol/Specifications/Snapshotter/preloading.md index 1749036..f35ba47 100644 --- a/docs/Protocol/Specifications/Snapshotter/preloading.md +++ b/docs/Protocol/Specifications/Snapshotter/preloading.md @@ -6,10 +6,10 @@ sidebar_position: 1 Preloaders play a crucial role in fetching low-level data, such as block details and transaction receipts. This ensures that subsequent base snapshot building can proceed without unnecessary redundant queries against the data source smart contracts. -Each project type within the project configuration, found in [config/projects.json](https://github.com/Powerloom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf8070/config/projects.example.json#L3-L12), can specify the preloaders that their base snapshot builds depend on. Once the dependent preloaders have completed their fetches, the Processor Distributor subsequently triggers the base snapshot builders for each project type. +Each project type within the project configuration, found in [config/projects.json](https://github.com/powerloom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf8070/config/projects.example.json#L3-L12), can specify the preloaders that their base snapshot builds depend on. Once the dependent preloaders have completed their fetches, the Processor Distributor subsequently triggers the base snapshot builders for each project type. ```json reference -https://github.com/Powerloom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf8070/config/projects.example.json#L3-L12 +https://github.com/powerloom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf8070/config/projects.example.json#L3-L12 ``` ![Preloading](/images/preloading.png) @@ -21,7 +21,7 @@ The preloaders implement one of the following two generic interfaces: ### `GenericPreloader` ```python reference -https://github.com/Powerloom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf8070/snapshotter/utils/callback_helpers.py#L109-L126 +https://github.com/powerloom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf8070/snapshotter/utils/callback_helpers.py#L109-L126 ``` ### `GenericDelegatorPreloader` @@ -29,13 +29,13 @@ https://github.com/Powerloom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf807 These preloaders are tasked with fetching large volumes of data and utilize delegated workers. They submit large workloads over a request queue and wait for the results to be returned over a response queue. ```python reference -https://github.com/Powerloom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf8070/snapshotter/utils/callback_helpers.py#L129-L161 +https://github.com/powerloom/pooler/blob/5e7cc3812074d91e8d7d85058554bb1175bf8070/snapshotter/utils/callback_helpers.py#L129-L161 ``` ## Shipped preloaders Currently, the snapshotter template includes three generic preloaders: -* [Block details](https://github.com/Powerloom/pooler/blob/33f838a2fd7a1ad335f5d0ab00c6fb5828821282/snapshotter/utils/preloaders/block_details/preloader.py) - Prefetches block details for all blocks in an epoch and caches them in Redis. -* [Eth Price](https://github.com/Powerloom/pooler/blob/feat/single_snapshotter/snapshotter/utils/preloaders/eth_price/preloader.py) - Prefetches ETH price for blocks in an epoch and caches it in Redis. -* [Transaction receipts](https://github.com/Powerloom/pooler/blob/feat/single_snapshotter/snapshotter/utils/preloaders/tx_receipts/preloader.py) - Prefetches all transaction details present in each epoch and caches the data in Redis. Since fetching all block transactions is a substantial workload, it utilizes the delegated workers architecture to parallelize and fetch data quickly and reliably. \ No newline at end of file +* [Block details](https://github.com/powerloom/pooler/blob/33f838a2fd7a1ad335f5d0ab00c6fb5828821282/snapshotter/utils/preloaders/block_details/preloader.py) - Prefetches block details for all blocks in an epoch and caches them in Redis. +* [Eth Price](https://github.com/powerloom/pooler/blob/feat/single_snapshotter/snapshotter/utils/preloaders/eth_price/preloader.py) - Prefetches ETH price for blocks in an epoch and caches it in Redis. +* [Transaction receipts](https://github.com/powerloom/pooler/blob/feat/single_snapshotter/snapshotter/utils/preloaders/tx_receipts/preloader.py) - Prefetches all transaction details present in each epoch and caches the data in Redis. Since fetching all block transactions is a substantial workload, it utilizes the delegated workers architecture to parallelize and fetch data quickly and reliably. \ No newline at end of file diff --git a/docs/Protocol/Specifications/Snapshotter/snapshot-build.md b/docs/Protocol/Specifications/Snapshotter/snapshot-build.md index 17028e5..f865868 100644 --- a/docs/Protocol/Specifications/Snapshotter/snapshot-build.md +++ b/docs/Protocol/Specifications/Snapshotter/snapshot-build.md @@ -10,18 +10,18 @@ sidebar_position: 2 As briefly introduced in the section on Snapshotter implementations that [leverage Git Submodules for specific computation logic](/build-with-powerloom/snapshotter-node/architecture), the modules are specified in the configuration for project types under the key `processor`. ```json reference -https://github.com/Powerloom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json#L15-L28 +https://github.com/powerloom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json#L15-L28 ``` Let's take the example of the snapshot builder configured for the project type `zkevm:owlto_bridge` and locate it in the `snapshotter-computes` repo, in the `zkevm_quests` branch ```python reference -https://github.com/Powerloom/snapshotter-computes/blob/29199feab449ad0361b5867efcaae9854992966f/owlto_bridge.py#L1-L31 +https://github.com/powerloom/snapshotter-computes/blob/29199feab449ad0361b5867efcaae9854992966f/owlto_bridge.py#L1-L31 ``` As observed, it implements the `compute()` interface expected from Snapshotter implementations inheriting `GenericProcessorSnapshot`. ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/callback_helpers.py#L179-L196 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/callback_helpers.py#L179-L196 ``` @@ -39,14 +39,14 @@ The data sources are specified against the `projects` key in the configuration s The Snapshotter node attempts to retrieve data sources corresponding to the `projects` key from the protocol state. ```python reference title="Processor Distributor synchronizing projects from protocol" -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L321-L332 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L321-L332 ``` 2. If the `projects` key is non-existent - 1. data sources can also be dynamically added on the protocol state contract which the [processor distributor](/Protocol/Specifications/Snapshotter/components#processor-distributor) [syncs with](https://github.com/Powerloom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/processor_distributor.py#L1107) + 1. data sources can also be dynamically added on the protocol state contract which the [processor distributor](/Protocol/Specifications/Snapshotter/components#processor-distributor) [syncs with](https://github.com/powerloom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/processor_distributor.py#L1107) ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L738-L751 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L738-L751 ``` @@ -65,7 +65,7 @@ Bulk Mode is highly effective in situations where: - Once a certain state change is observed, no further changes need to be recorded. Example use cases include monitoring on-chain activities and tracking task or quest completion statuses on the blockchain. ```json reference title="Project configuration for bulk mode" -https://github.com/Powerloom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json#L17-L27 +https://github.com/powerloom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json#L17-L27 ``` This allows for flexibility to filter through all transactions and blocks without the need for predefined data sources. @@ -74,7 +74,7 @@ The `Processor Distributor` generates a `SnapshotProcessMessage` with bulk mode ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L717-L730 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/processor_distributor.py#L717-L730 ``` :::info @@ -84,7 +84,7 @@ Since common datapoints like block details, transaction receipts, etc., are prel Whenever a data source is added or removed by the [signaling ecosystem](/Protocol/data-sources#data-source-signaling), the protocol state smart contract emits a `ProjectUpdated` event with the following data model. ```python reference -https://github.com/Powerloom/pooler/blob/5892eeb9433d8f4b8aa677006d98a1dde0458cb7/snapshotter/utils/models/data_models.py#L102-L105 +https://github.com/powerloom/pooler/blob/5892eeb9433d8f4b8aa677006d98a1dde0458cb7/snapshotter/utils/models/data_models.py#L102-L105 ``` @@ -96,7 +96,7 @@ https://github.com/Powerloom/pooler/blob/5892eeb9433d8f4b8aa677006d98a1dde0458cb ### Project ID generation ```python reference -https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/snapshot_worker.py#L51-L71 +https://github.com/powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d03/snapshotter/utils/snapshot_worker.py#L51-L71 ``` ![Base snapshot project ID generation](/images/base_snapshot_project_id.png) @@ -104,7 +104,7 @@ https://github.com/Powerloom/pooler/blob/634610801a7fcbd8d863f2e72a04aa8204d27d0 #### Base snapshot of trade events for the [Uniswap V2 and V3 dashboard data markets](/category/uniswap-dashboard): ```python reference -https://github.com/Powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/trade_volume.py#L14-L44 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/trade_volume.py#L14-L44 ``` @@ -125,7 +125,7 @@ The order and dependencies of these compositions are specified according to the ### `SingleProject` aggregation type ```json reference -https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/aggregator.example.json#L1-L10 +https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/aggregator.example.json#L1-L10 ``` * This type specifies the generation of an aggregation snapshot for a single project across a span of epochs relative to the current `epochId`. @@ -137,13 +137,13 @@ The following implementation aggregates [trade volume snapshots](/build-with-pow ```python reference -https://github.com/Powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/aggregate/single_uniswap_trade_volume_24h.py#L110-L121 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/aggregate/single_uniswap_trade_volume_24h.py#L110-L121 ``` ### `MultiProject` aggregation type ```json reference -https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/aggregator.example.json#L25-L31 +https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/aggregator.example.json#L25-L31 ``` * `projects_to_wait_for` specifies the exact project IDs on which this higher-order aggregation will be generated. @@ -160,5 +160,5 @@ In the case of 'MultiProject` aggregations, their project IDs are generated with The following is the section where the relevant project IDs are generated according to their configuration type. ```python reference -https://github.com/Powerloom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/utils/aggregation_worker.py#L59-L92 +https://github.com/powerloom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/utils/aggregation_worker.py#L59-L92 ``` diff --git a/docs/Protocol/Specifications/state-v1.md b/docs/Protocol/Specifications/state-v1.md index 035a00f..e5c8374 100644 --- a/docs/Protocol/Specifications/state-v1.md +++ b/docs/Protocol/Specifications/state-v1.md @@ -31,15 +31,15 @@ Presently, the protocol state is implemented as a smart contract on an EVM-compa ### Chain and Contract Address -Depending on the node type, the network RPC and the deployed contract address can be found in the `env.example` file in either the [`deploy`](https://github.com/Powerloom/deploy/) (for Full Nodes) and [`snapshotter`](https://github.com/Powerloom/snapshotter-lite) repositories (for Lite Nodes). +Depending on the node type, the network RPC and the deployed contract address can be found in the `env.example` file in either the [`deploy`](https://github.com/powerloom/deploy/) (for Full Nodes) and [`snapshotter`](https://github.com/powerloom/snapshotter-lite) repositories (for Lite Nodes). -* [Snapshotter Lite Node: Testnet](https://github.com/PowerLoom/snapshotter-lite/blob/97f25850d0319b601638e46440b9bce2372551c4/env.example#L8) -* [Snapshotter Full Node: Incentivized testnet Phase 2](https://github.com/PowerLoom/deploy/blob/4af571508f832b120d18e93f48c54ddbfe6074a5/env.example#L6) +* [Snapshotter Lite Node: Testnet](https://github.com/powerloom/snapshotter-lite/blob/97f25850d0319b601638e46440b9bce2372551c4/env.example#L8) +* [Snapshotter Full Node: Incentivized testnet Phase 2](https://github.com/powerloom/deploy/blob/4af571508f832b120d18e93f48c54ddbfe6074a5/env.example#L6) ### ABI -The [Application Binary Interface (ABI) to interact with the smart contract](https://github.com/PowerLoom/pooler/blob/main/snapshotter/static/abis/ProtocolContract.json) can be found within the snapshotter repository. +The [Application Binary Interface (ABI) to interact with the smart contract](https://github.com/powerloom/pooler/blob/main/snapshotter/static/abis/ProtocolContract.json) can be found within the snapshotter repository. ### State Modification Functions diff --git a/docs/Protocol/data-sources.md b/docs/Protocol/data-sources.md index 4783336..16bd9e5 100644 --- a/docs/Protocol/data-sources.md +++ b/docs/Protocol/data-sources.md @@ -16,10 +16,10 @@ Our implementation of a data market serves datasets to render live Uniswap V2, U * [Building with Powerloom -- Uniswap V2 and V3 Dashboards](/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/) * [Building with Powerloom -- Aave V3 Dashboard](/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/) -Continuing with the Uniswap V2 example, you can explore the data sources list defined as part of the Uniswap v2 specific configuration in the [`snapshotter-configs`](https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json#L1-L11) repository. +Continuing with the Uniswap V2 example, you can explore the data sources list defined as part of the Uniswap v2 specific configuration in the [`snapshotter-configs`](https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json#L1-L11) repository. ```json reference -https://github.com/Powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json#L1-L11 +https://github.com/powerloom/snapshotter-configs/blob/fcf9b852bac9694258d7afcd8beeaa4cf961c65f/projects.example.json#L1-L11 ``` The `projects` field in the above configuration snippet represents the Uniswap v2 pair contract addresses that are tracked, and snapshots are generated for these contracts. To further understand how snapshots on these contracts are tracked, refer to the section on [Project Types and IDs](#project-types-and-ids). @@ -31,7 +31,7 @@ In situations where data sources are constantly changing or numerous, making it Instead, the snapshotter implementation operates in ['bulk mode'](/Protocol/Specifications/Snapshotter/snapshot-build#data-source-specification-bulk-mode). The data source configuration specifies computation modules that utilize general-purpose [preloaded](/Protocol/Specifications/Snapshotter/preloading) datasets to filter transactions, event logs, etc., on contract addresses of interest. This is where the signaling of data sources comes into play. ```json reference -https://github.com/Powerloom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json#L1-L28 +https://github.com/powerloom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json#L1-L28 ``` ### Data Source Signaling diff --git a/docs/build-with-powerloom/devnet/FAQs.md b/docs/build-with-powerloom/devnet/FAQs.md index cbf5470..87e9c3c 100644 --- a/docs/build-with-powerloom/devnet/FAQs.md +++ b/docs/build-with-powerloom/devnet/FAQs.md @@ -29,7 +29,7 @@ To configure the environment variables, follow these steps: 2. Fill in the required variables in .env `SOURCE_RPC_URL`, `SIGNER_ACCOUNT_ADDRESS`, `SIGNER_ACCOUNT_PRIVATE_KEY`, `SLOT_ID`, `PROTOCOL_STATE_CONTRACT`. ### Where can I find documentation on Snapshotter Configs and Computes? -You can find the documentation for Snapshotter Configs and Computes in the [docs](/build-with-powerloom/snapshotter-node/architecture/#configuration-files). The pooler computes and config files for the Uniswap V2 use case are found in [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/tree/eth_uniswapv2) and [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/tree/eth_uniswapv2). +You can find the documentation for Snapshotter Configs and Computes in the [docs](/build-with-powerloom/snapshotter-node/architecture/#configuration-files). The pooler computes and config files for the Uniswap V2 use case are found in [snapshotter-computes](https://github.com/powerloom/snapshotter-computes/tree/eth_uniswapv2) and [snapshotter-configs](https://github.com/powerloom/snapshotter-configs/tree/eth_uniswapv2). ### What is RPC? In blockchain, RPC (Remote Procedure Calls) nodes allow users and apps to interact with the blockchain without network details. The `RPCHelper` in `utils/rpc.py` simplifies snapshot generation & data extraction from the blockchain state. diff --git a/docs/build-with-powerloom/devnet/getting-started.md b/docs/build-with-powerloom/devnet/getting-started.md index 8c49532..4bf3124 100644 --- a/docs/build-with-powerloom/devnet/getting-started.md +++ b/docs/build-with-powerloom/devnet/getting-started.md @@ -134,7 +134,7 @@ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin Navigate to the directory where you want to install the node and clone the repository: ```bash - git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-devnet + git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-devnet ``` 2. **Navigate to the Directory**: @@ -278,7 +278,7 @@ For users running the node on personal hardware, the minimum specifications are: Navigate to the directory where you want to install the node and clone the repository: ```bash - git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-devnet + git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-devnet ``` 2. **Navigate to the Directory**: @@ -443,7 +443,7 @@ Once python3 is installed, we can go ahead and run the node:- 1. Clone this repository using the following command in the terminal: ```bash - git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-devnet + git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-devnet ``` This will clone the repository into a directory named `powerloom-devnet`. @@ -572,7 +572,7 @@ Setting up the snapshotter node on Windows requires a few additional steps. We s - Use the following command in WSL terminal to clone the Snapshotter repository: ```bash - git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-devnet + git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-devnet ``` 2. **Navigate to the Repository Directory:** @@ -726,6 +726,26 @@ For further details on using the snapshotter dashboard to monitor the running st ::: +--- + +## Alternative Setup: Snapshotter CLI + +For operators who prefer a streamlined setup process or need to manage multiple nodes, we offer the **Powerloom Snapshotter CLI**. This tool provides: + +- **Interactive configuration** with guided prompts +- **Simplified deployment** process +- **Unified management** of multiple instances +- **Built-in diagnostics** and troubleshooting + +:::tip When to Use the CLI +The CLI is particularly useful if you: +- Plan to run multiple snapshotter nodes +- Want a simpler configuration process +- Need to manage nodes across different environments (devnet/mainnet) +::: + +Learn more: [Snapshotter CLI Documentation](/build-with-powerloom/snapshotter-node/lite-node-v2/cli) + --- ## Troubleshooting diff --git a/docs/build-with-powerloom/snapshotter-node/architecture.md b/docs/build-with-powerloom/snapshotter-node/architecture.md index 0ba2b1b..56dbdfb 100644 --- a/docs/build-with-powerloom/snapshotter-node/architecture.md +++ b/docs/build-with-powerloom/snapshotter-node/architecture.md @@ -21,7 +21,7 @@ Use case-specific logic for generating snapshots and other configurations are av The architecture has been designed to facilitate the seamless interchange of configuration and modules. Adapting the system to different use cases is as straightforward as changing a Git branch. -You can observe the corresponding branches within [snapshotter-configs](https://github.com/Powerloom/snapshotter-configs/) and [snapshotter-computes](https://github.com/Powerloom/snapshotter-computes/) repos: +You can observe the corresponding branches within [snapshotter-configs](https://github.com/powerloom/snapshotter-configs/) and [snapshotter-computes](https://github.com/powerloom/snapshotter-computes/) repos: #### Snapshotter Computes * `eth_uniswapv2`: Pooler implementation for the Uniswap v2 dashboard @@ -41,14 +41,14 @@ You can observe the corresponding branches within [snapshotter-configs](https:// ### Configuration Files -Configuration files, located in the `/config` directory and linked to [snapshotter-configs](https://github.com/Powerloom/snapshotter-configs/) repo, play a pivotal role in defining project types, specifying paths for individual compute modules, and managing various project-related settings. +Configuration files, located in the `/config` directory and linked to [snapshotter-configs](https://github.com/powerloom/snapshotter-configs/) repo, play a pivotal role in defining project types, specifying paths for individual compute modules, and managing various project-related settings. ### Compute Modules -The heart of the system resides in the `snapshotter/modules` directory, linked to [snapshotter-computes](https://github.com/Powerloom/snapshotter-computes/), where the actual computation logic for each project type is defined. These modules drive the snapshot generation process for specific project types. +The heart of the system resides in the `snapshotter/modules` directory, linked to [snapshotter-computes](https://github.com/powerloom/snapshotter-computes/), where the actual computation logic for each project type is defined. These modules drive the snapshot generation process for specific project types. ## Building Your Own Use Case -Working on a new use case is as simple as writing a new compute module and adding a new configuration file. Instructions to get started in `build-dev.sh` mode are available in the [Deploy repo](https://github.com/Powerloom/deploy/tree/devnet). +Working on a new use case is as simple as writing a new compute module and adding a new configuration file. Instructions to get started in `build-dev.sh` mode are available in the [Deploy repo](https://github.com/powerloom/deploy/tree/devnet). # Useful Links diff --git a/docs/build-with-powerloom/snapshotter-node/data.md b/docs/build-with-powerloom/snapshotter-node/data.md index f8dd9d1..2d4a151 100644 --- a/docs/build-with-powerloom/snapshotter-node/data.md +++ b/docs/build-with-powerloom/snapshotter-node/data.md @@ -22,7 +22,7 @@ For instance, the Powerloom Uniswap V2 dashboard (https://uniswapv2.powerloom.io ### Verifying Data -To verify the data, you need the Prost chain RPC Url and the address of the deployed protocol state contract. Find these details in the `env.example` file in the Powerloom [deploy](https://github.com/Powerloom/deploy/) repository. +To verify the data, you need the Prost chain RPC Url and the address of the deployed protocol state contract. Find these details in the `env.example` file in the Powerloom [deploy](https://github.com/powerloom/deploy/) repository. #### Adding the Chain to the Wallet @@ -43,7 +43,7 @@ Note: The RPC URL and the contract address in the screenshot below are for refer To access the ABI of the protocol state contract from the Powerloom project, follow these steps: -1. Visit the Powerloom 'pooler' repository on GitHub at this URL: [Powerloom pooler repository - ProtocolContract.json](https://github.com/Powerloom/pooler/blob/main/snapshotter/static/abis/ProtocolContract.json). +1. Visit the Powerloom 'pooler' repository on GitHub at this URL: [Powerloom pooler repository - ProtocolContract.json](https://github.com/powerloom/pooler/blob/main/snapshotter/static/abis/ProtocolContract.json). 2. Locate the `ProtocolContract.json` file. 3. Copy the contents of the file. 4. Open the Remix IDE. @@ -67,7 +67,7 @@ The logic to retrieve the CID from the protocol state contract can be extracted Here's the code to find the last finalized epoch for a project ID: ```python reference -https://github.com/Powerloom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/core_api.py#L248-L339 +https://github.com/powerloom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/core_api.py#L248-L339 ``` Call the relevant functions in the loaded Protocol state contract ABI in the Remix IDE to fetch the data. diff --git a/docs/build-with-powerloom/snapshotter-node/full-node/getting-started.md b/docs/build-with-powerloom/snapshotter-node/full-node/getting-started.md index a0b2cb3..b235814 100644 --- a/docs/build-with-powerloom/snapshotter-node/full-node/getting-started.md +++ b/docs/build-with-powerloom/snapshotter-node/full-node/getting-started.md @@ -36,7 +36,7 @@ Whether you are developing your own application or extending our existing use ca 1. **Cloning the Deploy Repo** - Clone the repository against the respective branch (main by default). Open the terminal and run the below command to clone the deploy repo in a directory named `powerloom_deploy`. ```bash - git clone https://github.com/PowerLoom/deploy.git --single-branch powerloom_deploy --branch main && cd powerloom_deploy + git clone https://github.com/powerloom/deploy.git --single-branch powerloom_deploy --branch main && cd powerloom_deploy ``` 2. **Configuring The Node** - Copy `env.example` to `.env`. @@ -63,17 +63,17 @@ Whether you are developing your own application or extending our existing use ca Extending existing use cases or building your own use cases requires a slightly different setup. Follow the process outlined below to ensure a smooth setup: -1. **Forking the Computes and Config templates** - For an optimized development process, it's recommended to fork the templates for [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes) and [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/). Our system utilizes the Git submodule architecture to manage these components efficiently. For a deeper understanding of how these elements integrate and function within our larger system, please refer to our [architecture documentation](../architecture.md). This approach ensures a streamlined and cohesive development workflow. +1. **Forking the Computes and Config templates** - For an optimized development process, it's recommended to fork the templates for [snapshotter-computes](https://github.com/powerloom/snapshotter-computes) and [snapshotter-configs](https://github.com/powerloom/snapshotter-configs/). Our system utilizes the Git submodule architecture to manage these components efficiently. For a deeper understanding of how these elements integrate and function within our larger system, please refer to our [architecture documentation](../architecture.md). This approach ensures a streamlined and cohesive development workflow. - - Snapshotter Configs: https://github.com/PowerLoom/snapshotter-configs - - Snapshotter Computes: https://github.com/PowerLoom/snapshotter-computes + - Snapshotter Configs: https://github.com/powerloom/snapshotter-configs + - Snapshotter Computes: https://github.com/powerloom/snapshotter-computes Once the above branches are forked, you should have the above two repositories in your profile. 2. **Cloning the Deploy Repo** - Clone the repository against the testnet branch. Open the terminal and run the below command to clone the deploy repo in a directory named `powerloom_deploy`. ```bash - git clone https://github.com/PowerLoom/deploy.git --single-branch powerloom_deploy --branch better_dev_workflow && cd powerloom_deploy + git clone https://github.com/powerloom/deploy.git --single-branch powerloom_deploy --branch better_dev_workflow && cd powerloom_deploy ``` 3. **Configuring The Node** - When you are in the deploy repo's directory, create a new file `.env`. diff --git a/docs/build-with-powerloom/snapshotter-node/lite-node-v2/build-sh-advanced.md b/docs/build-with-powerloom/snapshotter-node/lite-node-v2/build-sh-advanced.md index 5e39acb..aa269f2 100644 --- a/docs/build-with-powerloom/snapshotter-node/lite-node-v2/build-sh-advanced.md +++ b/docs/build-with-powerloom/snapshotter-node/lite-node-v2/build-sh-advanced.md @@ -15,7 +15,7 @@ The `build.sh` CLI supports a number of advanced features to fine tune its behav This flag is used to skip the prompt that asks you whether you want to update the existing env file. -Refer: [2.5 Using the `build.sh` script to reconfigure the node](/build-with-powerloom/snapshotter-node/lite-node-v2/monitoring#25-using-the-buildsh-script-to-reconfigure-the-node) +Refer: [2.5 Using the `build.sh` script to reconfigure the node](/build-with-powerloom/snapshotter-node/lite-node-v2/monitoring#25-using-buildsh-to-reconfigure-the-node) ### **`--no-collector`** @@ -66,7 +66,7 @@ Assuming you have 2 or more slot IDs to run to participate in the Uniswap V2 dat cd ~ # clone the snapshotter node repo -git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-mainnet1 +git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-mainnet1 # run new screen session screen -R powerloom-mainnet1 @@ -92,7 +92,7 @@ cd powerloom-mainnet1 cd ~ # clone the snapshotter node repo -git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-mainnet2 +git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-mainnet2 # run new screen session screen -R powerloom-mainnet2 diff --git a/docs/build-with-powerloom/snapshotter-node/lite-node-v2/cli.md b/docs/build-with-powerloom/snapshotter-node/lite-node-v2/cli.md new file mode 100644 index 0000000..5fb00f1 --- /dev/null +++ b/docs/build-with-powerloom/snapshotter-node/lite-node-v2/cli.md @@ -0,0 +1,345 @@ +--- +sidebar_position: 2 +title: Snapshotter CLI +--- + +# Powerloom Snapshotter CLI + +The Powerloom Snapshotter CLI (`powerloom-snapshotter-cli`) is a command-line tool that simplifies the process of configuring, deploying, and managing multiple snapshotter instances across different chains and data markets. + +:::tip +The CLI is specifically designed to manage multiple snapshotter instances and replaces the legacy manual setup. It is also perfect for running single snapshotter instances and can safely replace the [single node setup](/build-with-powerloom/snapshotter-node/lite-node-v2/getting-started). +::: + +## 📖 Overview + +### ✨ Key Features + +- 🚀 **Easy Configuration**: Set up credentials and settings for different chain/market combinations +- 📦 **Multi-Instance Management**: Deploy and manage multiple snapshotter instances +- 🔍 **Instance Monitoring**: View status, logs, and diagnostics for running instances +- 🐚 **Interactive Shell**: Fast command execution with history support +- 🔐 **Secure Credential Storage**: Namespaced environment files for different configurations +- 🏗️ **Identity Management**: Generate and manage signer identities + +### 🔄 Command Aliases + +The CLI provides multiple command aliases for convenience: +- `powerloom-snapshotter-cli` - Full command name +- `snapshotter` - Short alias + +Both commands are equivalent and can be used interchangeably throughout this documentation. + +## 🛠️ Installation {#installation} + +### 📦 Using Pre-built Binaries (Recommended) + +1. Download the latest binary for your platform from the [releases page](https://github.com/powerloom/snapshotter-lite-multi-setup/releases): + - Linux x86_64: `powerloom-snapshotter-cli-linux-amd64` + - Linux ARM64: `powerloom-snapshotter-cli-linux-arm64` + - macOS ARM64 (Apple Silicon): `powerloom-snapshotter-cli-macos-arm64` + +2. Make the binary executable: + ```bash + chmod +x powerloom-snapshotter-cli-* + ``` + +3. Move to a directory in your PATH: + ```bash + # Use the full name + sudo mv powerloom-snapshotter-cli-* /usr/local/bin/powerloom-snapshotter-cli + + # Or use a shorter alias for convenience + sudo mv powerloom-snapshotter-cli-* /usr/local/bin/snapshotter + ``` + +### 📥 From PyPI Package + +```bash +# Install using uv (recommended) +uv tool install powerloom-snapshotter-cli + +# Or install using pipx +pipx install powerloom-snapshotter-cli +``` + +## 🚀 Quick Start + +### 🪄 Interactive Shell Mode (Recommended) + +The CLI has a startup time when running individual commands. We strongly recommend using the **interactive shell mode** for a much faster and smoother experience: + +```bash +# Start the interactive shell +powerloom-snapshotter-cli shell + +# Now run commands instantly without delays: +powerloom-snapshotter> list +powerloom-snapshotter> configure +powerloom-snapshotter> deploy +powerloom-snapshotter> status +``` + +:::tip Why Use Shell Mode? +- ⚡ **Instant command execution** - No startup delay between commands +- 📝 **Command history** - Use arrow keys to navigate previous commands +- 🔄 **Persistent session** - Maintains context between commands +- 🎯 **Better workflow** - Run multiple operations smoothly +::: + +### ⌨️ Alternative: Individual Commands + +If you prefer to run individual commands: + +```bash +# Configure credentials +powerloom-snapshotter-cli configure --env mainnet --market uniswapv2 + +# Deploy snapshotter instances +powerloom-snapshotter-cli deploy --env mainnet --market uniswapv2 + +# Check status +powerloom-snapshotter-cli status + +``` + +### 🔍 Example Workflow + +Here's a typical workflow for setting up a snapshotter node using the interactive shell: + +```bash +# Start the interactive shell +powerloom-snapshotter-cli shell + +# Show all available data markets and protocol chains +powerloom-snapshotter> list + +# In the shell, configure credentials (interactive prompts will guide you) +powerloom-snapshotter> configure + +# Deploy your snapshotter instances +powerloom-snapshotter> deploy + +# Show status of deployed snapshotter instances (screen sessions and Docker containers). Optionally filter by environment and/or data market. +powerloom-snapshotter> status + +# Exit when done +powerloom-snapshotter> exit +``` + +## 💻 Core Commands + +### 📋 Show all available data markets and protocol chains + +**Shell mode:** +```bash +powerloom-snapshotter> list +``` + +**Command line mode:** +```bash +powerloom-snapshotter-cli list +``` + +### ⚙️ Configure credentials for deployments + +Set up credentials and settings for a specific chain and data market combination. + +**Shell mode:** +```bash +# Interactive configuration (prompts for all values) +powerloom-snapshotter> configure +``` + +**Command line mode:** +```bash +# Interactive configuration +powerloom-snapshotter-cli configure + +# With all options specified +powerloom-snapshotter-cli configure --env mainnet --market uniswapv2 --wallet 0x123... +``` + +**Options:** +- `--env, -e`: Powerloom chain name (e.g., devnet, mainnet) +- `--market, -m`: Data market name (e.g., uniswapv2, aavev3) +- `--wallet, -w`: Wallet address holding the slots +- `--signer, -s`: Signer account address +- `--signer-key, -k`: Signer account private key +- `--source-rpc, -r`: Source chain RPC URL +- `--powerloom-rpc, -p`: Powerloom RPC URL + +### 🚀 Deploy snapshotter nodes + +Deploy snapshotter nodes for specified environment and data markets. + +**Shell mode:** +```bash +# Interactive deployment (prompts for environment and market) +powerloom-snapshotter> deploy +``` + +**Command line mode:** +```bash +# Deploy all slots for a market +powerloom-snapshotter-cli deploy --env mainnet --market uniswapv2 + +# Deploy specific slots +powerloom-snapshotter-cli deploy --env mainnet --market uniswapv2 --slot 123 --slot 456 + +# Deploy multiple markets +powerloom-snapshotter-cli deploy --env mainnet --market uniswapv2 --market aavev3 +``` + +### 📊 Show deployment status + +Show status of deployed snapshotter instances (screen sessions and Docker containers). Optionally filter by environment and/or data market. + +**Shell mode:** +```bash +powerloom-snapshotter> status +``` + +**Command line mode:** +```bash +powerloom-snapshotter-cli status +``` + +Output shows: +- Instance name +- Status (Active/Inactive) +- Docker container status +- Process details + + +### 🔧 Diagnose and cleanup + +Run diagnostics on the system and check requirements. + +**Shell mode:** +```bash +# Run diagnostics +powerloom-snapshotter> diagnose +``` + +**Command line mode:** +```bash +# Run diagnostics +powerloom-snapshotter-cli diagnose + +# Clean up existing deployments +powerloom-snapshotter-cli diagnose --clean --force +``` + +Checks include: +- Python version +- System resources (CPU, memory, disk) +- Docker installation and status +- Network connectivity +- Required port availability + +### 🔑 Manage credentials + +Manage signer identities and other credentials configured against a specific environment and data market. + +**Shell mode:** +```bash +# Show summary of all available identities +powerloom-snapshotter> identity list + +# Show details of a specific identity +powerloom-snapshotter> identity show --env mainnet --market uniswapv2 +``` + +**Command line mode:** +```bash +# Show summary of all available identities +powerloom-snapshotter-cli identity list + +# Show details of a specific identity +powerloom-snapshotter-cli identity show --env mainnet --market uniswapv2 +``` + +## 📝 Credentials Management: Internals + +### 📁 Configuration Files + +Configuration files are stored in `~/.powerloom-snapshotter-cli/envs/` with the naming convention: +``` +.env.{chain}.{market}.{source_chain} +``` + +Example: `.env.mainnet.uniswapv2.eth_mainnet` + +### 🔐 Environment Variables + +Each configuration file contains: +- `WALLET_HOLDER_ADDRESS`: Address holding slot NFTs +- `SIGNER_ACCOUNT_ADDRESS`: Address used for signing snapshots +- `SIGNER_ACCOUNT_PRIVATE_KEY`: Private key for signer account +- `SOURCE_RPC_URL`: RPC endpoint for source blockchain +- `POWERLOOM_RPC_URL`: Powerloom protocol RPC endpoint + + +### 📊 Comparison with Manual Setup + +| Feature | Manual Setup | CLI | +|---------|-------------|-----| +| Multiple slots | Requires multiple repositories | Single command deployment | +| Configuration | Manual `.env` file editing | Interactive configuration | +| Monitoring | Check each instance separately | Unified `list` command | +| Updates | Update each repository | Single CLI update | + +## ❓ Troubleshooting + +### 🐛 Common Issues + +#### "Docker daemon is not running" +Start Docker Desktop or the Docker service: +```bash +# Linux +sudo systemctl start docker +``` + +#### "No slots found for wallet" +Verify that: +- The wallet address owns slots on the specified chain +- You're using the correct chain (devnet vs mainnet) +- The RPC URL is accessible + +#### "Screen session already exists" +Clean up existing sessions: +```bash +powerloom-snapshotter-cli diagnose --clean --force +``` + +## 🔄 Migration from Manual Setup + +If you're currently running snapshotter nodes manually, here's how to migrate to the CLI: + +1. **Stop and cleanup existing nodes**: + ```bash + # In your existing setup + ./diagnose.sh -y + ``` + +2. **Install the CLI** (see [Installation section](#installation) above) + +3. **Configure using the CLI**: + ```bash + powerloom-snapshotter-cli configure --env mainnet --market uniswapv2 + ``` + +4. **Deploy using the CLI**: + ```bash + powerloom-snapshotter-cli deploy --env mainnet --market uniswapv2 + ``` + +The CLI will handle all your slots automatically and provide better management capabilities. + +## 🔗 Additional Resources + +- [GitHub Repository: Dive into the technical details, file issues, and contribute](https://github.com/powerloom/snapshotter-lite-multi-setup) +- [Legacy Getting Started Guide: For single node setup](https://docs.powerloom.io/build-with-powerloom/snapshotter-node/lite-node-v2/getting-started) +- [Monitoring and Troubleshooting](/build-with-powerloom/snapshotter-node/lite-node-v2/monitoring) +- [Discord Support: Join our community and get help](https://discord.gg/powerloom) \ No newline at end of file diff --git a/docs/build-with-powerloom/snapshotter-node/lite-node-v2/getting-started.md b/docs/build-with-powerloom/snapshotter-node/lite-node-v2/getting-started.md index 52a6de9..9e73656 100644 --- a/docs/build-with-powerloom/snapshotter-node/lite-node-v2/getting-started.md +++ b/docs/build-with-powerloom/snapshotter-node/lite-node-v2/getting-started.md @@ -22,7 +22,7 @@ Follow the step-by-step instructions in the relevant guide to set up your Snapsh :::info Important note 1. It is recommended that the node operates continuously, 24/7. -2. If you possess multiple slots, refer to the section on [setting up multiple nodes](#setting-up-multiple-nodes-on-a-linux-vps) for further details. +2. If you possess multiple slots, refer to the section on [alternative setup methods](#alternative-setup-methods) for further details. ::: @@ -134,7 +134,7 @@ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin Navigate to the directory where you want to install the node and clone the repository: ```bash - git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-mainnet + git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-mainnet ``` 2. **Navigate to the Directory**: @@ -278,7 +278,7 @@ For users running the node on personal hardware, the minimum specifications are: Navigate to the directory where you want to install the node and clone the repository: ```bash - git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-mainnet + git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-mainnet ``` 2. **Navigate to the Directory**: @@ -443,7 +443,7 @@ Once python3 is installed, we can go ahead and run the lite node:- 1. Clone this repository using the following command in the terminal: ```bash - git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-mainnet + git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-mainnet ``` This will clone the repository into a directory named `powerloom-mainnet`. @@ -568,7 +568,7 @@ Setting up the snapshotter node on Windows requires a few additional steps. We s - Use the following command in WSL terminal to clone the Snapshotter Lite Node v2 repository: ```bash - git clone https://github.com/PowerLoom/snapshotter-lite-v2.git powerloom-mainnet + git clone https://github.com/powerloom/snapshotter-lite-v2.git powerloom-mainnet ``` 2. **Navigate to the Repository Directory:** @@ -718,22 +718,33 @@ It usually takes 2 - 5 minutes between the simulation mode submissions and a sub ![Sample node logs](/images/RegularSubmission.png) :::info -For further details on using the snapshotter dashboard to monitor the running status of your node, check the [`Monitoring Node Activity with the Snapshotter Dashboard`](/build-with-powerloom/snapshotter-node/lite-node-v2/monitoring#11-monitoring-node-activity-with-the-snapshotter-dashboard) section. +For further details on using the snapshotter dashboard to monitor the running status of your node, check the [`Monitoring Node Activity`](/build-with-powerloom/snapshotter-node/lite-node-v2/monitoring#11-monitoring-node-activity) section. ::: --- -## Setting Up Multiple Nodes on a Linux VPS +## Alternative Setup Methods -We have a detailed, extensive documentation and setup guide on our mutli node setup on a Linux VPS. +### Snapshotter CLI (Recommended for Multiple Slots) -Head over here and follow the detailed instructions in the README: https://github.com/PowerLoom/snapshotter-lite-multi-setup +For operators managing multiple slots or those who prefer a streamlined setup process, we offer the **Powerloom Snapshotter CLI**. This tool simplifies configuration, deployment, and management of snapshotter nodes. -:::tip -We recommend using this for even our single node operators, as long as you can follow the detailed instructions. +:::tip Snapshotter CLI Benefits +- **Single command deployment** for all your slots +- **Interactive configuration** with guided prompts +- **Unified monitoring** of all instances +- **Easier updates** compared to manual setup ::: +Learn more: [Snapshotter CLI Documentation](/build-with-powerloom/snapshotter-node/lite-node-v2/cli) + +### Manual Multi-Node Setup + +For advanced users who prefer manual control, we also provide detailed documentation for setting up multiple nodes on a Linux VPS. + +Repository: https://github.com/powerloom/snapshotter-lite-multi-setup + ## Troubleshooting diff --git a/docs/build-with-powerloom/snapshotter-node/lite-node-v2/slot-monitoring-setup.md b/docs/build-with-powerloom/snapshotter-node/lite-node-v2/slot-monitoring-setup.md index 9341be5..68a8dfc 100644 --- a/docs/build-with-powerloom/snapshotter-node/lite-node-v2/slot-monitoring-setup.md +++ b/docs/build-with-powerloom/snapshotter-node/lite-node-v2/slot-monitoring-setup.md @@ -24,7 +24,7 @@ Before setting up the monitoring service, ensure you have: ### Step 1: Clone the Repository ```bash -git clone https://github.com/Powerloom/submissions-monitor-alerts.git +git clone https://github.com/powerloom/submissions-monitor-alerts.git cd submissions-monitor-alerts ``` diff --git a/docs/build-with-powerloom/use-cases/building-new-usecase/extending-uniswapv2-dashboard.md b/docs/build-with-powerloom/use-cases/building-new-usecase/extending-uniswapv2-dashboard.md index 73080bf..c595b96 100644 --- a/docs/build-with-powerloom/use-cases/building-new-usecase/extending-uniswapv2-dashboard.md +++ b/docs/build-with-powerloom/use-cases/building-new-usecase/extending-uniswapv2-dashboard.md @@ -45,7 +45,7 @@ We'll use the example of creating a new data point that aggregates only Swap eve ``` 1. **Create a New Data Model**: - Develop a new data model in [`utils/message_models.py`](https://github.com/Powerloom/snapshotter-computes/blob/eth_uniswapv2/utils/models/message_models.py). Use existing models like `UniswapTradesAggregateSnapshot` and `UniswapTradesSnapshot` as references. Your model should be tailored to capture and represent data specific to the 2-hour Swap event aggregation. + Develop a new data model in [`utils/message_models.py`](https://github.com/powerloom/snapshotter-computes/blob/eth_uniswapv2/utils/models/message_models.py). Use existing models like `UniswapTradesAggregateSnapshot` and `UniswapTradesSnapshot` as references. Your model should be tailored to capture and represent data specific to the 2-hour Swap event aggregation. 2. **Focus on 2-Hour Time Span and Swap Events**: Modify the data collection logic to concentrate on a 2-hour time span (`epochId`). Ensure that your implementation is set to extract only Swap event logs and their associated trade volumes. Refer to the existing 24-hour aggregation example for guidance on structuring your logic. diff --git a/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/data-points.md b/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/data-points.md index bd0b52a..999a552 100644 --- a/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/data-points.md +++ b/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/data-points.md @@ -11,7 +11,7 @@ toc_max_heading_level: 5 In Aave V3-Pooler, data points are specific, quantifiable elements derived from Aave V3 lending activities. Each snapshot inherits `SnapshotBase` which looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L9-L17 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L9-L17 ``` ## Base Snapshots @@ -24,14 +24,14 @@ The Asset Metrics refer to the total supplied and borrowed amounts, supply and b The asset snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message_models.py#L26-L42 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/message_models.py#L26-L42 ``` #### 1a. AaveSupplyData / AaveDebtData The nominal token amount and the token amount in terms of USD at the time of snapshotting: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/data_models.py#L65-L72 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/data_models.py#L65-L72 ``` #### 1b. AssetDetailsData @@ -40,7 +40,7 @@ Additional asset details include the maximum loan-to-value ratio, liquidation in For further information, please see the Aave V3 Protocol [Documentation](https://docs.aave.com/risk/asset-risk/risk-parameters). ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/data_models.py#L25-L34 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/data_models.py#L25-L34 ``` #### 1c. RateDetailsData @@ -50,7 +50,7 @@ Additional rate details describing the interest rate strategy for the asset. For further information, please see the Aave V3 Protocol [Documentation](https://docs.aave.com/risk/liquidity-risk/borrow-interest-rate). ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/data_models.py#L37-L45 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/data_models.py#L37-L45 ``` ### 2. Volume-by-Action Snapshot @@ -59,7 +59,7 @@ The asset's total volume-by-action for the Epoch, and the corresponding events f The volume-by-action snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message_models.py#L91-L98 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/message_models.py#L91-L98 ``` #### 2a. volumeData @@ -67,7 +67,7 @@ https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message The total nominal token volume and the volume in terms of USD at the time of snapshotting: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/data_models.py#L92-L94 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/data_models.py#L92-L94 ``` #### 2b. liquidationData @@ -75,7 +75,7 @@ https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/data_mo Additional details describing a liquidation action: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/data_models.py#L117-L122 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/data_models.py#L117-L122 ``` ## Aggregate Snapshots @@ -89,7 +89,7 @@ The following aggregate snapshots are generated by AaveV3-Pooler using base snap All assets are organized by the total market share on the protocol. The top assets by marketshare snapshot look something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message_models.py#L55-L69 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/message_models.py#L55-L69 ``` ### 2. 24 Hour Volume-by-Action Snapshot @@ -97,7 +97,7 @@ https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message 24-hour volume-by-action for each asset per Epoch. The 24-hour volume-by-action snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message_models.py#L101-L107 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/message_models.py#L101-L107 ``` ### 3. Top Assets by Volume Snapshot @@ -105,7 +105,7 @@ https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message All assets are organized by their 24-hour total volume-by-action. The top assets by volume snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message_models.py#L110-L128 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/message_models.py#L110-L128 ``` ### 4. 6 Hour Average Rates Snapshot @@ -113,7 +113,7 @@ https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message Average supply and borrow rates over the previous 6 hours for each asset per Epoch. The 6-hour average rate snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message_models.py#L82-L88 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/message_models.py#L82-L88 ``` ### 5. 24 Hour Total Market Stats Snapshot @@ -121,6 +121,6 @@ https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message 24-hour stats for the entire Aave lending market. The 24-hour stats snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message_models.py#L72-L79 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/message_models.py#L72-L79 ``` diff --git a/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/index.md b/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/index.md index d5b4d2e..3b7b66a 100644 --- a/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/index.md +++ b/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/index.md @@ -18,7 +18,7 @@ Data points are essential elements that serve as the foundational units upon whi ### Development and Extension -The Pooler framework builds upon the modular architecture of a Snapshotter Node and enables extensions and custom use case implementations. A developer can extend the pooler compute files found in [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/tree/aave) and config files found in [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/tree/aave) to build their own custom use case implementations. +The Pooler framework builds upon the modular architecture of a Snapshotter Node and enables extensions and custom use case implementations. A developer can extend the pooler compute files found in [snapshotter-computes](https://github.com/powerloom/snapshotter-computes/tree/aave) and config files found in [snapshotter-configs](https://github.com/powerloom/snapshotter-configs/tree/aave) to build their own custom use case implementations. The documentation has a dedicated section that discusses further implementation and use-case extensions. Check out our guide on [Setup and Extension](/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/setup-and-extension.md). diff --git a/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/setup-and-extension.md b/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/setup-and-extension.md index ac26a50..3d1d7a2 100644 --- a/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/setup-and-extension.md +++ b/docs/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/setup-and-extension.md @@ -14,7 +14,7 @@ Prerequisite: Ensure that the minimum requirements are met by the system on whic 1. **Cloning the Deploy Repo** - Clone the repository against the `aave` branch. Open the terminal and run the below command to clone the deploy repo in a directory named `powerloom_deploy`. ```bash - git clone https://github.com/PowerLoom/deploy.git --single-branch powerloom_deploy --branch aave && cd powerloom_deploy + git clone https://github.com/powerloom/deploy.git --single-branch powerloom_deploy --branch aave && cd powerloom_deploy ``` 2. **Configuring The Node** - Copy `env.example` to `.env`. @@ -43,8 +43,8 @@ Prerequisite: Ensure that the minimum requirements are met by the system on whic ``` 4. **Displaying the Dashboard** -- Once all of the services are up and running, the front-end can be accessed via [Pooler Frontend](https://github.com/PowerLoom/pooler-frontend/tree/aave) to see an Aave V3 summary data dashboard similar to the [PowerLoom Aave V3 Dasboard](https://aave-v3.powerloom.io/). - - The front-end does not come packaged as part of the Aave deploy repository and will need to be run separately. The Front-end deployment instructions can be found [here](https://github.com/PowerLoom/pooler-frontend/tree/aave?tab=readme-ov-file#powerloom-pooler-product). +- Once all of the services are up and running, the front-end can be accessed via [Pooler Frontend](https://github.com/powerloom/pooler-frontend/tree/aave) to see an Aave V3 summary data dashboard similar to the [PowerLoom Aave V3 Dasboard](https://aave-v3.powerloom.io/). + - The front-end does not come packaged as part of the Aave deploy repository and will need to be run separately. The Front-end deployment instructions can be found [here](https://github.com/powerloom/pooler-frontend/tree/aave?tab=readme-ov-file#powerloom-pooler-product). - The dashboard's `Synced by` section displays the time of the last snapshot taken, indicating if your snapshotting is falling behind. ## Extending the Aave V3 Implementation @@ -55,14 +55,14 @@ This section will utilize core concepts explained in the [Closer Look at Snapsho ### Development Node Setup -1. **Forking the Computes and Config templates** - For an optimized development process, it's recommended to fork the templates [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/tree/aave) and [snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/tree/aave). Our system utilizes the Git submodule architecture to manage these components efficiently. For a deeper understanding of how these elements integrate and function within our larger system, please refer to our [architecture documentation](/build-with-powerloom/snapshotter-node/architecture.md). This approach ensures a streamlined and cohesive development workflow. +1. **Forking the Computes and Config templates** - For an optimized development process, it's recommended to fork the templates [snapshotter-computes](https://github.com/powerloom/snapshotter-computes/tree/aave) and [snapshotter-configs](https://github.com/powerloom/snapshotter-configs/tree/aave). Our system utilizes the Git submodule architecture to manage these components efficiently. For a deeper understanding of how these elements integrate and function within our larger system, please refer to our [architecture documentation](/build-with-powerloom/snapshotter-node/architecture.md). This approach ensures a streamlined and cohesive development workflow. - - Aave V3 Snapshotter Configs: https://github.com/PowerLoom/snapshotter-computes/tree/aave - - Aave V3 Snapshotter Computes: https://github.com/PowerLoom/snapshotter-configs/tree/aave + - Aave V3 Snapshotter Configs: https://github.com/powerloom/snapshotter-computes/tree/aave + - Aave V3 Snapshotter Computes: https://github.com/powerloom/snapshotter-configs/tree/aave Once the above branches are forked, you should have the two repositories in your profile. -2. **Deploy the Development Node** - Detailed instructions for configuring the Aave V3 use case for development can be found in the [Powerloom Deploy](https://github.com/PowerLoom/deploy/tree/aave?tab=readme-ov-file#instructions-for-code-contributors) GitHub repository. +2. **Deploy the Development Node** - Detailed instructions for configuring the Aave V3 use case for development can be found in the [Powerloom Deploy](https://github.com/powerloom/deploy/tree/aave?tab=readme-ov-file#instructions-for-code-contributors) GitHub repository. - Ensure that the correct Config and Computes submodules are provided in the `SNAPSHOT_CONFIG_REPO` and `SNAPSHOTTER_COMPUTE_REPO` entries in the `.env` file. These must be changed if you have chosen to fork the submodule repositories as they are set to the Powerloom URLs by default. Additionally, ensure the correct branch names are provided in `SNAPSHOT_CONFIG_REPO_BRANCH` and `SNAPSHOTTER_COMPUTE_REPO_BRANCH` entries if they have been changed. @@ -79,21 +79,21 @@ There are two important preloaders to be aware of: 1. **Bulk Asset Data Preloader** ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/preloaders/asset_data/preloader.py#L10-L34 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/preloaders/asset_data/preloader.py#L10-L34 ``` 2. **Bulk Events Preloader** ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/preloaders/volume_events/preloader.py#L9-L33 +https://github.com/powerloom/snapshotter-computes/blob/aave/utils/preloaders/volume_events/preloader.py#L9-L33 ``` -Both of these preloaders retrieve on-chain data and then store it locally in Redis for [later use](https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/core.py#L93-L121) by the base snapshot processors' `compute` functions. The data models describing the data gathered by the `BulkAssetDataPreloader` can be found [here](https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/data_models.py#L9-L45). The list of event emissions gathered by the `BulkVolumeEventsPreloader` can be found [here](https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/data_models.py#L9-L45). +Both of these preloaders retrieve on-chain data and then store it locally in Redis for [later use](https://github.com/powerloom/snapshotter-computes/blob/aave/utils/core.py#L93-L121) by the base snapshot processors' `compute` functions. The data models describing the data gathered by the `BulkAssetDataPreloader` can be found [here](https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/data_models.py#L9-L45). The list of event emissions gathered by the `BulkVolumeEventsPreloader` can be found [here](https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/data_models.py#L9-L45). Visit the [Data Points](/build-with-powerloom/use-cases/existing-implementations/aavev3-dashboard/data-points.md) section for a complete list of currently available base snapshots. ### Adding an Aggregate Snapshot -For this example, we will add a 6-hour volume-by-action aggregate using the 24-hour volume aggregate as a reference. The data for the 24-hour volume aggregate is already retrieved by the `BulkVolumeEventsPreloader` and processed using the `AggregateSupplyVolumeProcessor` as seen in this [compute](https://github.com/PowerLoom/snapshotter-computes/blob/aave/aggregate/single_aave_volume_24h.py#L108). +For this example, we will add a 6-hour volume-by-action aggregate using the 24-hour volume aggregate as a reference. The data for the 24-hour volume aggregate is already retrieved by the `BulkVolumeEventsPreloader` and processed using the `AggregateSupplyVolumeProcessor` as seen in this [compute](https://github.com/powerloom/snapshotter-computes/blob/aave/aggregate/single_aave_volume_24h.py#L108). 1. **Create the Aggregation Worker Processor**: @@ -128,7 +128,7 @@ For this example, we will add a 6-hour volume-by-action aggregate using the 24-h 3. **Modify the Processor Logic**: - Modify the data collection logic to concentrate on a 6-hour time span (`epochId`). Refer to the existing 24-hour aggregation example for guidance on structuring your logic. The 24-hour aggregate currently collects data for all core Aave V3 "actions" that can be taken on-chain. You may continue to collect data for all action's events, or you may choose to focus on a single action. Keep in mind that you will need to create a new data model in the `utils/message_models.py` file in your `snapshotter-computes` fork if you choose to change the snapshot data. See the [`AaveVolumeAggregateSnapshot`](https://github.com/PowerLoom/snapshotter-computes/blob/aave/utils/models/message_models.py#L101-L107) for reference. + Modify the data collection logic to concentrate on a 6-hour time span (`epochId`). Refer to the existing 24-hour aggregation example for guidance on structuring your logic. The 24-hour aggregate currently collects data for all core Aave V3 "actions" that can be taken on-chain. You may continue to collect data for all action's events, or you may choose to focus on a single action. Keep in mind that you will need to create a new data model in the `utils/message_models.py` file in your `snapshotter-computes` fork if you choose to change the snapshot data. See the [`AaveVolumeAggregateSnapshot`](https://github.com/powerloom/snapshotter-computes/blob/aave/utils/models/message_models.py#L101-L107) for reference. 4. **Testing and Validation**: diff --git a/docs/build-with-powerloom/use-cases/existing-implementations/activity-tracking/index.md b/docs/build-with-powerloom/use-cases/existing-implementations/activity-tracking/index.md index 3b54a6c..6b7260e 100644 --- a/docs/build-with-powerloom/use-cases/existing-implementations/activity-tracking/index.md +++ b/docs/build-with-powerloom/use-cases/existing-implementations/activity-tracking/index.md @@ -23,12 +23,12 @@ For example, we at Powerloom use Quest completion tracking for our Testnet progr ### Snapshot Building We let Snapshotter Peers handle the heavy lifting of capturing on-chain activities. Developers can focus on building a Snapshot Processor that extracts and returns the relevant data from the list of transactions in a block. -For example, `BungeeBridgeProcessor` located in [`snapshotter/modules/computes`](https://github.com/PowerLoom/snapshotter-computes/blob/zkevm_quests/bungee_bridge.py) is one of the base Processor computes for ZkEVM Quest Tracking. This class uses the `GenericProcessorSnapshot` structure found in [`snapshotter/utils/callback_helpers.py`](https://github.com/Powerloom/pooler/blob/main/snapshotter/utils/callback_helpers.py). +For example, `BungeeBridgeProcessor` located in [`snapshotter/modules/computes`](https://github.com/powerloom/snapshotter-computes/blob/zkevm_quests/bungee_bridge.py) is one of the base Processor computes for ZkEVM Quest Tracking. This class uses the `GenericProcessorSnapshot` structure found in [`snapshotter/utils/callback_helpers.py`](https://github.com/powerloom/pooler/blob/main/snapshotter/utils/callback_helpers.py). Any compute for base snapshots basically needs to implement the `compute` function. ```python reference -https://github.com/Powerloom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/utils/callback_helpers.py#L190-L195 +https://github.com/powerloom/pooler/blob/d8b7be32ad329e8dcf0a7e5c1b27862894bc990a/snapshotter/utils/callback_helpers.py#L190-L195 ``` The `compute` function is the main part where we create and process snapshots. It uses these inputs: @@ -39,7 +39,7 @@ The `compute` function is the main part where we create and process snapshots. I `epoch` is `PowerloomSnapshotProcessMessage` object which contains the following information: ```python reference -https://github.com/PowerLoom/pooler/blob/main/snapshotter/utils/models/message_models.py#L46-L50 +https://github.com/powerloom/pooler/blob/main/snapshotter/utils/models/message_models.py#L46-L50 ``` The Infrastructure is scalable enough to handle high throughput chains like ZkEvm in close to real time even with `Epoch` size of 1 block. @@ -47,7 +47,7 @@ The Infrastructure is scalable enough to handle high throughput chains like ZkEv The `BungeeBridgeProcessor` then goes through all preloaded block transactions, filters out, and then generates relevant snapshots for wallet addresses that received funds from the Bungee Bridge refuel contract during that epoch. ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/29199feab449ad0361b5867efcaae9854992966f/bungee_bridge.py#L40-L92 +https://github.com/powerloom/snapshotter-computes/blob/29199feab449ad0361b5867efcaae9854992966f/bungee_bridge.py#L40-L92 ``` The format of the output data can vary based on what you need it for. However, the return type must always be a list of (`wallet_address`, [`pydantic`](https://pypi.org/project/pydantic/) snapshot object) (for bulk mode). @@ -56,7 +56,7 @@ Each `wallet_address` is mapped to a `pydantic` snapshot object and submitted to ### Configuration -A configuration file [`config/projects.json`](https://github.com/Powerloom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json) specifies the details of the snapshots to be generated. For instance, sample config for [`BungeeBridgeProcessor`](https://github.com/PowerLoom/snapshotter-computes/blob/zkevm_quests/bungee_bridge.py) for `zkevm:bungee_bridge` might look like this: +A configuration file [`config/projects.json`](https://github.com/powerloom/snapshotter-configs/blob/39e4713cdd96fff99d100f1dea7fb7332df9e491/projects.example.json) specifies the details of the snapshots to be generated. For instance, sample config for [`BungeeBridgeProcessor`](https://github.com/powerloom/snapshotter-computes/blob/zkevm_quests/bungee_bridge.py) for `zkevm:bungee_bridge` might look like this: ```json { @@ -73,7 +73,7 @@ A configuration file [`config/projects.json`](https://github.com/Powerloom/snaps } ``` -Critical dependencies, such as [`block_transactions`](https://github.com/Powerloom/pooler/blob/main/snapshotter/utils/preloaders/tx_receipts/preloader.py), are essential components defining the initial dataset processed by the snapshot builder. +Critical dependencies, such as [`block_transactions`](https://github.com/powerloom/pooler/blob/main/snapshotter/utils/preloaders/tx_receipts/preloader.py), are essential components defining the initial dataset processed by the snapshot builder. ## Expanding to Monitor Additional On-chain Activities diff --git a/docs/build-with-powerloom/use-cases/existing-implementations/eth_price_tracking.md b/docs/build-with-powerloom/use-cases/existing-implementations/eth_price_tracking.md index 03affa0..9d063d3 100644 --- a/docs/build-with-powerloom/use-cases/existing-implementations/eth_price_tracking.md +++ b/docs/build-with-powerloom/use-cases/existing-implementations/eth_price_tracking.md @@ -22,7 +22,7 @@ We aim to capture real-time changes in ETH's price conversion ratio. Watch the t Clone repository against the respective branch (main by default). Open the terminal and run the below command to clone the deploy repo in a directory named `powerloom_deploy`. ```bash -git clone https://github.com/PowerLoom/deploy.git --single-branch powerloom_deploy --branch devnet && cd powerloom_deploy +git clone https://github.com/powerloom/deploy.git --single-branch powerloom_deploy --branch devnet && cd powerloom_deploy ``` ### Configuring The Node - Copy env.example to .env @@ -39,7 +39,7 @@ git clone https://github.com/PowerLoom/deploy.git --single-branch powerloom_depl Unless it is a customized need or instructed by us, the below need not be changed or modified from the values already supplied in `env.example` ::: -- `SNAPSHOT_CONFIG_REPO`=[https://github.com/powerloom/snapshotter-configs](https://github.com/PowerLoom/snapshotter-configs/tree/eth_price_monitor) +- `SNAPSHOT_CONFIG_REPO`=[https://github.com/powerloom/snapshotter-configs](https://github.com/powerloom/snapshotter-configs/tree/eth_price_monitor) - `SNAPSHOT_CONFIG_REPO_BRANCH`=eth_price_monitor - `SNAPSHOTTER_COMPUTE_REPO`=[https://github.com/powerloom/snapshotter-computes](https://github.com/powerloom/snapshotter-computes/tree/eth_price_monitor) - `SNAPSHOTTER_COMPUTE_REPO_BRANCH`=eth_price_monitor @@ -66,17 +66,17 @@ It is recommended to look at the docs on [snapshot generation](/Protocol/Specifi Snapshotter node has a simple interface as part of `GenericProcessorSnapshot` class that business logic specific computes need to implement. The rest of the heavy lifting around reliable submissions and finalization is taken care of by the rest of the features of the node. -In this case, `EthPriceProcessor` located in [`snapshotter/modules/computes`](https://github.com/PowerLoom/snapshotter-computes/blob/eth_price_monitor/eth_price_tracking.py) is the sole compute class for ETH price tracking. This class implements the [`GenericProcessorSnapshot`](https://github.com/Powerloom/pooler/blob/main/snapshotter/utils/callback_helpers.py) interface, most notably the `compute()` callback. This callback is where we build the snapshots. +In this case, `EthPriceProcessor` located in [`snapshotter/modules/computes`](https://github.com/powerloom/snapshotter-computes/blob/eth_price_monitor/eth_price_tracking.py) is the sole compute class for ETH price tracking. This class implements the [`GenericProcessorSnapshot`](https://github.com/powerloom/pooler/blob/main/snapshotter/utils/callback_helpers.py) interface, most notably the `compute()` callback. This callback is where we build the snapshots. ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/eth_price_monitor/eth_price_tracking.py#L1-L45 +https://github.com/powerloom/snapshotter-computes/blob/eth_price_monitor/eth_price_tracking.py#L1-L45 ``` -`get_eth_price_usd()` is located in [`snapshotter/modules/computes`](https://github.com/PowerLoom/snapshotter-computes/blob/eth_price_monitor/eth_price_tracking.py) and is a utility function that calculates the price of ETH as a weighted average of its price conversion ratios against DAI, USDC and USDT in the corresponding Uniswap V2 pair contracts. The weights assigned are relative to the total liquidity locked against the W(rapped)ETH token across the three pair contracts. +`get_eth_price_usd()` is located in [`snapshotter/modules/computes`](https://github.com/powerloom/snapshotter-computes/blob/eth_price_monitor/eth_price_tracking.py) and is a utility function that calculates the price of ETH as a weighted average of its price conversion ratios against DAI, USDC and USDT in the corresponding Uniswap V2 pair contracts. The weights assigned are relative to the total liquidity locked against the W(rapped)ETH token across the three pair contracts. ```python reference -https://github.com/PowerLoom/pooler/blob/main/snapshotter/utils/snapshot_utils.py#L140-L154 +https://github.com/powerloom/pooler/blob/main/snapshotter/utils/snapshot_utils.py#L140-L154 ``` ### Set Up the Codebase @@ -114,7 +114,7 @@ The snapshotted data as generated by the compute module referred above is retrie - `project_id (str)`: refer to [project id generation](/Protocol/Specifications/Snapshotter/snapshot-build) to know more. In this example, it is the concatenation of the the first element in the tuple [returned from the snapshot compute module's implementation of `compute()`](#snapshot-building) along with the project type prefix as found in the [corresponding config for this snapshotter node](#configuring-the-node---copy-envexample-to-env). ```python reference -https://github.com/PowerLoom/snapshotter-configs/blob/8ed08b19272005f5c45b1af1ff9fd0ab5195bbc6/projects.example.json#L4 +https://github.com/powerloom/snapshotter-configs/blob/8ed08b19272005f5c45b1af1ff9fd0ab5195bbc6/projects.example.json#L4 ``` :::info @@ -139,9 +139,9 @@ As the `epoch_id` moves ahead to `100793`, we find the above endpoint reflects t 1. Define the correct project ID against which data has to be fetched ```python -project_type = "eth:price_tracking" # from https://github.com/PowerLoom/snapshotter-configs/blob/eth_price_monitor/projects.example.json#L4 -project_namespace = "DEVNET" # from https://github.com/PowerLoom/deploy/blob/95ceb83a97a16279816c406eef484245df483fb1/env.example#L25 -data_source = "example" # from https://github.com/PowerLoom/snapshotter-computes/blob/15059013c6c17327d1c0d413d3885c23a6383305/eth_price_tracking.py#L45 +project_type = "eth:price_tracking" # from https://github.com/powerloom/snapshotter-configs/blob/eth_price_monitor/projects.example.json#L4 +project_namespace = "DEVNET" # from https://github.com/powerloom/deploy/blob/95ceb83a97a16279816c406eef484245df483fb1/env.example#L25 +data_source = "example" # from https://github.com/powerloom/snapshotter-computes/blob/15059013c6c17327d1c0d413d3885c23a6383305/eth_price_tracking.py#L45 project_id = f'{project_type}:{data_source}:{project_namespace}' ``` 2. Fetch the last finalized epoch from the API endpoint `http://localhost:8002/last_finalized_epoch/{project_id}` diff --git a/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/closer-look-at-snapshots.md b/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/closer-look-at-snapshots.md index ebed7ba..1fcd54e 100644 --- a/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/closer-look-at-snapshots.md +++ b/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/closer-look-at-snapshots.md @@ -10,12 +10,12 @@ Let's take a closer look at how base snapshots and aggregates are generated by t Before you dive into this section, please make sure you take a look into the [Snapshot Generation Section](/Protocol/Specifications/Snapshotter/snapshot-build#base-snapshots). ::: -Snapshotter node has several interfaces defined to handle the heavy lifting so that you can focus on just writing computes modules. For example, `TradeVolumeProcessor`, located in the **Snapshotter-computes** [`snapshotter-computer/trade_volume.py`](https://github.com/Powerloom/snapshotter-computes/blob/eth_uniswapv2/trade_volume.py), is one of the base Processor computes for Pooler. This class uses the `GenericProcessorSnapshot` structure found in [`snapshotter/utils/callback_helpers.py`](https://github.com/Powerloom/pooler/blob/main/snapshotter/utils/callback_helpers.py). +Snapshotter node has several interfaces defined to handle the heavy lifting so that you can focus on just writing computes modules. For example, `TradeVolumeProcessor`, located in the **Snapshotter-computes** [`snapshotter-computer/trade_volume.py`](https://github.com/powerloom/snapshotter-computes/blob/eth_uniswapv2/trade_volume.py), is one of the base Processor computes for Pooler. This class uses the `GenericProcessorSnapshot` structure found in [`snapshotter/utils/callback_helpers.py`](https://github.com/powerloom/pooler/blob/main/snapshotter/utils/callback_helpers.py). Any processor for base snapshots needs to implement the `compute` function. ```python reference -https://github.com/Powerloom/snapshotter-computes/blob/74b2eaa452bfac8c0e4e0a7ed74a4d2748e9c224/trade_volume.py#L23-L28 +https://github.com/powerloom/snapshotter-computes/blob/74b2eaa452bfac8c0e4e0a7ed74a4d2748e9c224/trade_volume.py#L23-L28 ``` The `compute` function is the main part where we create and process snapshots. It uses these inputs: @@ -25,7 +25,7 @@ The `compute` function is the main part where we create and process snapshots. I `epoch` is `PowerloomSnapshotProcessMessage` object which contains the following information: ```python reference -https://github.com/PowerLoom/pooler/blob/main/snapshotter/utils/models/message_models.py#L46-L50 +https://github.com/powerloom/pooler/blob/main/snapshotter/utils/models/message_models.py#L46-L50 ``` The `TradeVolumeProcessor` collects and stores information about trades that happen within a specific range of blocks in the blockchain, known as the epoch. This range is defined by the lowest block number (`min_chain_height`) and the highest block number (`max_chain_height`) in that epoch. @@ -43,11 +43,11 @@ In the preceding section, we delved into the `TradeVolumeProcessor` logic, respo The epoch size, as detailed in the earlier section on [epoch generation](/Protocol/Specifications/Epoch), remains constant for this specific implementation of the Uniswap v2 use case on the Powerloom Protocol. This consistency extends to the time duration encompassed within each epoch. -To access the finalized state and data CID corresponding to each epoch, refer to the smart contract on the anchor chain holding the protocol state. The relevant helpers for this can be located in `get_project_epoch_snapshot()` within [`pooler/snapshotter/utils/data_utils.py`](https://github.com/Powerloom/pooler/blob/main/snapshotter/utils/data_utils.py). +To access the finalized state and data CID corresponding to each epoch, refer to the smart contract on the anchor chain holding the protocol state. The relevant helpers for this can be located in `get_project_epoch_snapshot()` within [`pooler/snapshotter/utils/data_utils.py`](https://github.com/powerloom/pooler/blob/main/snapshotter/utils/data_utils.py). ```python reference -https://github.com/Powerloom/pooler/blob/fc08cdd951166ab0cea669d233cd28d0639f628d/snapshotter/utils/data_utils.py#L273-L295 +https://github.com/powerloom/pooler/blob/fc08cdd951166ab0cea669d233cd28d0639f628d/snapshotter/utils/data_utils.py#L273-L295 ``` @@ -60,22 +60,22 @@ tail_epoch_id = current_epoch_id - int(time_in_seconds / (source_chain_epoch_siz ```python reference -https://github.com/Powerloom/pooler/blob/fc08cdd951166ab0cea669d233cd28d0639f628d/snapshotter/utils/data_utils.py#L507-L546 +https://github.com/powerloom/pooler/blob/fc08cdd951166ab0cea669d233cd28d0639f628d/snapshotter/utils/data_utils.py#L507-L546 ``` The worker class for such aggregation is defined in `config/aggregator.json` in the following manner: ```json reference -https://github.com/Powerloom/snapshotter-configs/blob/ae77941311155a9126205af08735c3dfa5d72ac2/aggregator.example.json#L3-L10 +https://github.com/powerloom/snapshotter-configs/blob/ae77941311155a9126205af08735c3dfa5d72ac2/aggregator.example.json#L3-L10 ``` -`AggregateTradeVolumeProcessor`, located in the **Snapshotter-computes** [`snapshotter-computer/aggregate/single_uniswap_trade_volume_24h.py`](https://github.com/Powerloom/snapshotter-computes/blob/eth_uniswapv2/aggregate/single_uniswap_trade_volume_24h.py) is one of the aggregate computes for Pooler. This class uses the `GenericProcessorAggregate` structure found in [`snapshotter/utils/callback_helpers.py`](https://github.com/Powerloom/pooler/blob/main/snapshotter/utils/callback_helpers.py). +`AggregateTradeVolumeProcessor`, located in the **Snapshotter-computes** [`snapshotter-computer/aggregate/single_uniswap_trade_volume_24h.py`](https://github.com/powerloom/snapshotter-computes/blob/eth_uniswapv2/aggregate/single_uniswap_trade_volume_24h.py) is one of the aggregate computes for Pooler. This class uses the `GenericProcessorAggregate` structure found in [`snapshotter/utils/callback_helpers.py`](https://github.com/powerloom/pooler/blob/main/snapshotter/utils/callback_helpers.py). Any processor for aggregate snapshots needs to implement the `compute` function. ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/74b2eaa452bfac8c0e4e0a7ed74a4d2748e9c224/aggregate/single_uniswap_trade_volume_24h.py#L110-L118 +https://github.com/powerloom/snapshotter-computes/blob/74b2eaa452bfac8c0e4e0a7ed74a4d2748e9c224/aggregate/single_uniswap_trade_volume_24h.py#L110-L118 ``` The `compute` function is the main part where we create and process snapshots. It uses these inputs: @@ -91,15 +91,15 @@ The `compute` function is the main part where we create and process snapshots. I PowerloomSnapshotSubmittedMessage contains the following information: ```python reference -https://github.com/PowerLoom/pooler/blob/main/snapshotter/utils/models/message_models.py#L46-L50 +https://github.com/powerloom/pooler/blob/main/snapshotter/utils/models/message_models.py#L46-L50 ``` PowerloomCalculateAggregateMessage contains the following information: ```python reference -https://github.com/PowerLoom/pooler/blob/main/snapshotter/utils/models/message_models.py#L90-L93 +https://github.com/powerloom/pooler/blob/main/snapshotter/utils/models/message_models.py#L90-L93 ``` -For a comprehensive understanding of Pooler computes, refer to the [snapshotter-computes](https://github.com/PowerLoom/snapshotter-computes/tree/eth_uniswapv2) repository. +For a comprehensive understanding of Pooler computes, refer to the [snapshotter-computes](https://github.com/powerloom/snapshotter-computes/tree/eth_uniswapv2) repository. ## Extending Pooler for Additional Datapoints diff --git a/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/data-points.md b/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/data-points.md index c77e8f4..427da35 100644 --- a/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/data-points.md +++ b/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/data-points.md @@ -12,7 +12,7 @@ In Pooler, data points are specific, quantifiable elements derived from Uniswap Each snapshot inherits `SnapshotBase` which looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L9-L17 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L9-L17 ``` ## Base Snapshots @@ -24,7 +24,7 @@ The following type of Base Snapshots are generated by Pooler for the Uniswap V2 Liquidity reserves of each pair per Epoch. The reserves snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L20-L32 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L20-L32 ``` Significant differences between the V2 and V3 implementations can be found in the generation of this base snapshot and are outlined below: @@ -34,16 +34,16 @@ Significant differences between the V2 and V3 implementations can be found in th The token reserve amounts for a UniswapV2 pair are retrieved from the [getReserves()](https://github.com/Uniswap/v2-core/blob/ee547b17853e71ed4e0101ccfd52e70d5acded58/contracts/UniswapV2Pair.sol#L38) function that is available in the `UniswapV2Pair` contract. These amounts are converted from their native integer format into a normalized decimal format and are then used to populate the snapshot's `token0Reserves` and `token1Reserves` fields. This calculation can be seen in the UniswapV2 compute module here: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/core.py#L103-L130 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/core.py#L103-L130 ``` The price of a token in a UniswapV2 pair can be derived using the ratio of the pair's token reserves, or by utilizing one of the pricing functions available in the `UniswapV2Library` contract such as [getAmountsOut()](https://github.com/Uniswap/v2-periphery/blob/0335e8f7e1bd1e8d8329fd300aea2ef2f36dd19f/contracts/libraries/UniswapV2Library.sol#L62-L70). See the [UniswapV2 Pricing](https://docs.uniswap.org/contracts/v2/concepts/advanced-topics/pricing) documentation for additional information. The snapshot's `token0Prices` and `token1Prices` fields are calculated by first deriving each token's price in terms of ETH (or the source chain's equivalent gas token), and then converting the resulting ETH price to USD terms. -For each token, the compute attempts to find a UniswapV2 pair consisting of the given token and wrapped ETH (WETH). If the token's WETH pair exists, its ETH price is derived using the ratio of the pair's token reserves. If the token's WETH pair does not exist, an additional pair consisting of the given token and a whitelisted token is used to derive an intermediary price before converting it back in terms of ETH. This calculation can be seen in the [Pricing Utils](https://github.com/PowerLoom/snapshotter-computes/blob/1abcf639d00a2fed87f01fcf0332cfb090257272/utils/pricing.py#L259-L318) of the UniswapV2 compute module. +For each token, the compute attempts to find a UniswapV2 pair consisting of the given token and wrapped ETH (WETH). If the token's WETH pair exists, its ETH price is derived using the ratio of the pair's token reserves. If the token's WETH pair does not exist, an additional pair consisting of the given token and a whitelisted token is used to derive an intermediary price before converting it back in terms of ETH. This calculation can be seen in the [Pricing Utils](https://github.com/powerloom/snapshotter-computes/blob/1abcf639d00a2fed87f01fcf0332cfb090257272/utils/pricing.py#L259-L318) of the UniswapV2 compute module. -The ETH price in terms of USD is calculated in Pooler's [Snapshot Utils](https://github.com/PowerLoom/pooler/blob/db969eb3956d77cbca36daaeb96fce70314a9b63/snapshotter/utils/snapshot_utils.py#L37-L184), and it is used to convert each token's ETH price to USD in the [Pricing Utils](https://github.com/PowerLoom/snapshotter-computes/blob/1abcf639d00a2fed87f01fcf0332cfb090257272/utils/pricing.py#L320-L331) of the UniswapV2 compute module. The results are used to populate the snapshot's `token0Prices` and `token1Prices` fields. +The ETH price in terms of USD is calculated in Pooler's [Snapshot Utils](https://github.com/powerloom/pooler/blob/db969eb3956d77cbca36daaeb96fce70314a9b63/snapshotter/utils/snapshot_utils.py#L37-L184), and it is used to convert each token's ETH price to USD in the [Pricing Utils](https://github.com/powerloom/snapshotter-computes/blob/1abcf639d00a2fed87f01fcf0332cfb090257272/utils/pricing.py#L320-L331) of the UniswapV2 compute module. The results are used to populate the snapshot's `token0Prices` and `token1Prices` fields. -The snapshot's `token0ReservesUSD` and `token1ReservesUSD` fields are calculated by applying the `token0Prices` and `token1Prices` to the `token0Reserves` and `token1Reserves` values respectively. This calculation can be seen in the [Core Utils](https://github.com/PowerLoom/snapshotter-computes/blob/1abcf639d00a2fed87f01fcf0332cfb090257272/utils/core.py#L132-L133) of the UniswapV2 compute module. +The snapshot's `token0ReservesUSD` and `token1ReservesUSD` fields are calculated by applying the `token0Prices` and `token1Prices` to the `token0Reserves` and `token1Reserves` values respectively. This calculation can be seen in the [Core Utils](https://github.com/powerloom/snapshotter-computes/blob/1abcf639d00a2fed87f01fcf0332cfb090257272/utils/core.py#L132-L133) of the UniswapV2 compute module. #### V3 Reserves Snapshot Caclulations @@ -53,8 +53,8 @@ The following section contains references to advanced concepts specific to the U Unlike the `UniswapV2Pair` contract, the `getReserves()` function is not available in the `UniswapV3Pool` contract, so the token reserve amounts cannot be retrieved directly from the chain and must be calculated manually. The calculation is computationally expensive, as it involves iterating over each available tick on the pool's price curve to sum the liquidity available at each tick. In order to reduce resource usage and limit the number of RPC requests, Powerloom's UniswapV3 dashboard implementation splits the computation of the `UniswapPairTotalReservesSnapshot` into two phases: -1. An initial manual calculation of the token reserves for each pool using the `calculate_reserves()` function found in the [Total Value Locked Utils](https://github.com/PowerLoom/snapshotter-computes/blob/9241e32155107949ccf4dbc4214ef29a91996b7f/total_value_locked.py#L158-L182) of the UniswapV3 compute module. -2. Incremental adjustments to the initially calculated reserve values based on `Mint` and `Burn` events for each token in the pool. This calculation can be seen in the [Core Utils](https://github.com/PowerLoom/snapshotter-computes/blob/9241e32155107949ccf4dbc4214ef29a91996b7f/utils/core.py#L131-L168) of the UniswapV3 compute module. +1. An initial manual calculation of the token reserves for each pool using the `calculate_reserves()` function found in the [Total Value Locked Utils](https://github.com/powerloom/snapshotter-computes/blob/9241e32155107949ccf4dbc4214ef29a91996b7f/total_value_locked.py#L158-L182) of the UniswapV3 compute module. +2. Incremental adjustments to the initially calculated reserve values based on `Mint` and `Burn` events for each token in the pool. This calculation can be seen in the [Core Utils](https://github.com/powerloom/snapshotter-computes/blob/9241e32155107949ccf4dbc4214ef29a91996b7f/utils/core.py#L131-L168) of the UniswapV3 compute module. The resulting reserve values are used to populate the snapshot's `token0Reserves` and `token1Reserves` fields. @@ -63,17 +63,17 @@ The price of a token in a UniswapV3 pool can be calculated from the `sqrtPriceX9 The snapshot's `token0Prices` and `token1Prices` are computed following a similar process to the UniswapV2 calculation, however, instead of the using the ratio of the the token reserves or a pricing function, the prices are derived from each pool's `sqrtPriceX96`. This calculation can be seen here: ```python reference -https://github.com/PowerLoom/pooler/blob/bf68b35eba32255e25d7b6701b0e5609fef86655/snapshotter/utils/snapshot_utils.py#L40-L49 +https://github.com/powerloom/pooler/blob/bf68b35eba32255e25d7b6701b0e5609fef86655/snapshotter/utils/snapshot_utils.py#L40-L49 ``` -The snapshot's `token0ReservesUSD` and `token1ReservesUSD` fields are calculated by applying the `token0Prices` and `token1Prices` to the `token0Reserves` and `token1Reserves` values respectively. This calculation can be seen in the [Core Utils](https://github.com/PowerLoom/snapshotter-computes/blob/9241e32155107949ccf4dbc4214ef29a91996b7f/utils/core.py#L173-L176) of the UniswapV3 compute module. +The snapshot's `token0ReservesUSD` and `token1ReservesUSD` fields are calculated by applying the `token0Prices` and `token1Prices` to the `token0Reserves` and `token1Reserves` values respectively. This calculation can be seen in the [Core Utils](https://github.com/powerloom/snapshotter-computes/blob/9241e32155107949ccf4dbc4214ef29a91996b7f/utils/core.py#L173-L176) of the UniswapV3 compute module. ### 2. Trade Volume Snapshot Trade volume of each pair per Epoch. The trade volume snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L40-L54 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L40-L54 ``` @@ -88,7 +88,7 @@ The following aggregate snapshots are generated by Pooler using base snapshots. 24 hour trade volume of each pair per Epoch. The 24 hour trade volume snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L57-L64 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L57-L64 ``` ### 2. 7 days Trade Volume Snapshot @@ -100,7 +100,7 @@ https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc8 Top pairs by 24 hour trade volume per Epoch. The 24 hours top pairs snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L83-L93 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L83-L93 ``` ### 4. 7 days Top Pairs Snapshot @@ -112,7 +112,7 @@ Top pairs by 7 days trade volume per Epoch. The 7 days top pairs snapshot is ver Top tokens by 24 hour trade volume per Epoch. The 24 hours top tokens snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L67-L80 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L67-L80 ``` ### 6. 24 hours Stats Snapshot @@ -120,7 +120,7 @@ https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc8 24 hour stats of each pair per Epoch. The 24 hours stats snapshot looks something like this: ```python reference -https://github.com/PowerLoom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L108-L115 +https://github.com/powerloom/snapshotter-computes/blob/6fb98b1bbc22be8b5aba8bdc860004d35786f4df/utils/models/message_models.py#L108-L115 ``` ## Implementation diff --git a/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/index.md b/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/index.md index 6e47bd5..0bca486 100644 --- a/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/index.md +++ b/docs/build-with-powerloom/use-cases/existing-implementations/uniswap-dashboard/index.md @@ -41,7 +41,7 @@ Data points are key metrics sourced from Uniswap V2 or V3 pair contracts, detail ### Development and Extension -Snapshotter Node's design enables extensions and custom use case implementations. A developer can extend the pooler UniswapV2 compute files found in [snapshotter-computes (V2)](https://github.com/PowerLoom/snapshotter-computes/tree/eth_uniswapv2) and config files found in [snapshotter-configs (V2)](https://github.com/PowerLoom/snapshotter-configs/tree/eth_uniswapv2), or the UniswapV3 compute files found in [snapshotter-computes (V3)](https://github.com/PowerLoom/snapshotter-computes/tree/uniswapv3-v1) and the associated config files in [snapshotter-configs (V3)](https://github.com/PowerLoom/snapshotter-configs/tree/eth_uniswapv3) to build their own custom use case implementations. +Snapshotter Node's design enables extensions and custom use case implementations. A developer can extend the pooler UniswapV2 compute files found in [snapshotter-computes (V2)](https://github.com/powerloom/snapshotter-computes/tree/eth_uniswapv2) and config files found in [snapshotter-configs (V2)](https://github.com/powerloom/snapshotter-configs/tree/eth_uniswapv2), or the UniswapV3 compute files found in [snapshotter-computes (V3)](https://github.com/powerloom/snapshotter-computes/tree/uniswapv3-v1) and the associated config files in [snapshotter-configs (V3)](https://github.com/powerloom/snapshotter-configs/tree/eth_uniswapv3) to build their own custom use case implementations. We have a dedicated section in the documentation which walkthrough the details on further implementation and usecases extensions. Check out our guide on [Extending Uniswap V2 dashboard usecase.](/build-with-powerloom/use-cases/building-new-usecase/extending-uniswapv2-dashboard) diff --git a/docs/chain-migration/snapshotters.md b/docs/chain-migration/snapshotters.md index 801062d..ee52739 100644 --- a/docs/chain-migration/snapshotters.md +++ b/docs/chain-migration/snapshotters.md @@ -58,7 +58,7 @@ python3 multi_clone.py ``` :::info -Learn more: [Complete setup steps](https://github.com/PowerLoom/snapshotter-lite-multi-setup/blob/master/README.md) for multi node setup +Learn more: [Complete setup steps](https://github.com/powerloom/snapshotter-lite-multi-setup/blob/master/README.md) for multi node setup ::: ## ✅ Verification logs diff --git a/docs/participate/testnet-phase1.md b/docs/participate/testnet-phase1.md index ed847ae..fe8b24d 100644 --- a/docs/participate/testnet-phase1.md +++ b/docs/participate/testnet-phase1.md @@ -35,7 +35,7 @@ The inaugural phase of our testnet concluded in October 2023. It was met with an To learn more and gain further insights, you can check out our announcement on Twitter regarding Powerloom Phase 1. https://x.com/Powerloom/status/1693655148931993647 ::: -> The source code for this phase is publicly accessible on GitHub: [Powerloom Testnet Phase 1 Source Code](https://github.com/PowerLoom/pooler/blob/testnet_5_pairs/). +> The source code for this phase is publicly accessible on GitHub: [Powerloom Testnet Phase 1 Source Code](https://github.com/powerloom/pooler/blob/testnet_5_pairs/). > In our comprehensive documentation, you will find a dedicated section that guides you through the process of extending the UniswapV2 dashboard. Explore this section to understand how you can build upon the UniswapV2 dashboard, adapting it to fit your specific requirements and ideas. For more information, please visit the [Extending UniswapV2 Dashboard section](../build-with-powerloom/use-cases/building-new-usecase/extending-uniswapv2-dashboard.md) in our documentation. diff --git a/docs/participate/testnet-phase2.md b/docs/participate/testnet-phase2.md index 9be8497..e915ebf 100644 --- a/docs/participate/testnet-phase2.md +++ b/docs/participate/testnet-phase2.md @@ -34,6 +34,6 @@ Currently, Phase 2 is actively in progress and can be accessed at [Mercle](https ## Code and Documentation -For developers interested in the technical aspects, the source code for Phase 2 is available on GitHub. For detailed insights into the mechanisms behind onchain activity tracking, please refer to the repository [here](https://github.com/PowerLoom/pooler/tree/zkevm_quests). +For developers interested in the technical aspects, the source code for Phase 2 is available on GitHub. For detailed insights into the mechanisms behind onchain activity tracking, please refer to the repository [here](https://github.com/powerloom/pooler/tree/zkevm_quests). > Additionally, you can explore our documentation on [onchain Activity Tracking](../build-with-powerloom/use-cases/existing-implementations/activity-tracking/) for further information on the technical implementation. \ No newline at end of file