Research ideas for better comparison of retrievability across different networks

[pyropy]

We need to develop new measurements and refine our existing metrics to create a more standardised retrievability scoring system that works across all networks, allowing for fair "apples-to-apples" comparisons.

[pyropy]

Draft proposal for the new leaderboard scoring system

Overview

A leaderboard has been developed to compare various Decentralized Physical Infrastructure Networks (DePIN). Initially, the concept was to use a scoring system similar to Lighthouse, evaluating networks across multiple dimensions to generate a single overall score.

Currently, the leaderboard only assesses networks based on their retrievability, which is an incomplete reflection of a network's capabilities.

Therefore, this proposal suggests implementing a new weighted scoring system and including an additional network storage share metric to calculate a more comprehensive network score.

Network Storage Share (NSM) Metric

The network storage share metric should represent the percentage of total DePIN storage that a network captures. To simplify comparisons between different networks, the metric should be based on utilized storage rather than network capacity. A separate metric based on network capacity could be introduced later.

Weighted Scoring System

To enable scoring system similar to Lighthouse we would need to introduce a weighted scoring system. A weighted scoring system enables cross-network comparisons based on multiple metrics. By incorporating the Network Storage Share (NSM) metric, the proposed formula for network score evaluation could be represented as:

( Retrievability Score * 0.8 ) + ( Network Storage Share * 0.2 )

Weights can always be adjusted given the number of metrics and their importance in the total score.

Score Example

Given the formula above and the current networks stats in the example table we calculate new networks score (with two different type of weights):

Network	RSR	Storage (EiB)	NSM	Weights 0.8/0.2	Weights 0.7/0.3
Filecoin (FIL+)	24.3	1.18	92.180	37.876	44.66
Arweave	70.3	0.1	7.812	57.802	51.55
Walrus	96.3	0.0001	0.008	77.042	67.41

EDIT: I have updated table to take in account size FIL+ deals and not while Filecoin network as we're only testing FIL+ deals with Spark.

[bajtos]

The network storage share metric should represent the percentage of total DePIN storage that a network captures.

Isn't this creating another apples-to-oranges comparison, on that favours storage networks designed to store large data?

• Based on my limited understanding of Arweave, it's designed to store medium-sized data (websites, 10-100 MiBs) for a very long time (200 years). (link)

• Filecoin is designed to store large data archives (GiBs to TiBs) for 180-540 days (6-18 months). (link)

• Walrus currently allows the storage of blobs up to a maximum size that may be determined through the walrus info CLI command. The maximum blob size is currently 13.3 GiB (link).

[juliangruber]

Very important point, @bajtos! This goes to show how complex requirements for these "fair" scores are. I believe again this incentivizes us to keep the scores as unopinionated as possible, and expose other data (like storage network size) so that the viewer can create their own picture

[juliangruber]

I agree not to base RSR on storage network size

[bajtos]

I have two more ideas:

• By using the "all-time total" number, we favour networks that have been around the longest.
• IMO, a more meaningful metric is the number for a recent time. E.g. data from yesterday, since we are using RSR from yesterday as well. In case of storage size, the metric can be "how much new data was onboarded yesterday".
• Instead of measuring the data size, we can measure the number of storage deals. Both options have different pros and cons, further exploration is required.

[pyropy]

Idea behind this weighted score system was to grade networks on multiple attributes, one of them being network size. Where some networks excel, others fail and with adding more scores into the equation and with adjusting weights we can make a more fair scoring system.

Isn't this creating another apples-to-oranges comparison

Thing is that we're comparing very different networks with a single metric from the very beginning.

... on that favours storage networks designed to store large data?

I think that score could possibly be adjusted with changing the weights, but it is true that due to it's sheer size Filecoin has improved score in the examples above. But, we should also be aware that it's much easier to serve few TiBs of data rather than few EiBs.

Based on my limited understanding of Arweave, it's designed to store medium-sized data (websites, 10-100 MiBs) for a very long time (200 years).

From my understanding of Arweave (which is also somewhat limited) I think that's correct. Altought, in practice while observing some transactions there is more often than not few of them with that are few TBs in size. For example this is some random "new" block I've stumbled on and none of the transactions are bellow 1GB in size.

Walrus currently allows the storage of blobs up to a maximum size that may be determined through the walrus info CLI command. The maximum blob size is currently 13.3 GiB.

Walrus seems very different from Arweave and Filecoin, I think it would be more fair to compare it against Akave and Storacha.

Very important point, @bajtos! This goes to show how complex requirements for these "fair" scores are. I believe again this incentivizes us to keep the scores as unopinionated as possible, and expose other data (like storage network size) so that the viewer can create their own picture

I think that would be the most fair comparison. We could make leaderboard more data-centric but I'm afraid we would sacrifice simplicity.