Design: Spark retrieval checks for random payload blocks

[bajtos]

The IPNI reverse index has been deployed to cid.contact. Let's enhance the Spark v1.x retrieval checking flow to not only check the retrievability of the root block, but also check whether a random payload block can be retrieved as well. This should flag SPs that are "cheating" the system by serving root-block retrievals only.

The goal of this issue is to propose a design and create an implementation plan. The design should include a discussion about potential attack vectors.

We want a backwards-compatible solution that will preserve RSR for honest SPs with no changes required on their side.

Resources:

• https://github.com/ipni/specs/blob/main/IPNI_MH_SAMPLING.md
• https://www.notion.so/spacemeridian/Spark-v2-design-migration-115cdd5cccdb804ca4d1e0694c613318 for details about Spark v2 breaking changes that we want to avoid.

[bajtos]

IPNI reverse index sampling is live on https://cid.contact. Only deals created on or after 2025-03-27 are indexed.

Example

Miner: f03312989
Index Provider ID: 12D3KooWPVLQMdrtkHbqGrUBRt5ZPvhyEGYPvwEGooY14cCQ2pQV

Deal (discovered from active_deals table in spark_deal_observer DB):

• activated at epoch: 4835759 (2025-03-30 23:59:30 UTC)
• PieceCID: baga6ea4seaqnzjtvqytajo4yvuwr3dbahzdwcyexcey3pqkxxgffkurdmftbuay
• PieceSize: 17179869184
• Context ID from PieceInfo: ghsAAAAEAAAAANgqWCgAAYHiA5IgINymdYYmBLuYrS0djCA%2BR2FglxExt8FXuYpVUiNhZhoD

Example multihash we found by spark-deal-observer/piece-indexer from IPNI advertisements

• bafk2bzaceaaxbe7yoo3yvfxhampdg7xdtahs4b27n374tawndtymsjlst3s4g

The miner constructs ContextID in the old format.
https://cid.contact/cid/bafk2bzaceaaxbe7yoo3yvfxhampdg7xdtahs4b27n374tawndtymsjlst3s4g

The following entry has Metadata matching the deal's PieceCID:

      {
          "ContextID": "AXESIAf56t+Yl5HUayIzc3CKP2tiCqhaj8xxXQe2fkCU+hTm",
          "Metadata": "kBKjaFBpZWNlQ0lE2CpYKAABgeIDkiAg3KZ1hiYEu5itLR2MID5HYWCXETG3wVe5ilVSI2FmGgNsVmVyaWZpZWREZWFs9W1GYXN0UmV0cmlldmFs9Q==",
          "Provider": {
            "ID": "12D3KooWPVLQMdrtkHbqGrUBRt5ZPvhyEGYPvwEGooY14cCQ2pQV",
            "Addrs": [
              "/ip4/116.92.243.3/tcp/17133"
            ]
          }
        },

A sample from the reverse index (notice how ContextID is encoded using encodeURIComponent):

https://cid.contact/ipni/v0/sample/12D3KooWPVLQMdrtkHbqGrUBRt5ZPvhyEGYPvwEGooY14cCQ2pQV/AXESIAf56t%2BYl5HUayIzc3CKP2tiCqhaj8xxXQe2fkCU%2BhTm

{
  "samples": [
    "2DrjgbKfYs8F9qmBPUpr9UYiFXQ1M9ru4byipJbTPTJQgABifm"
  ]
}

Raw-codec CIDv1 created from the multihash shown above:

bafk2bzacedso5s2r4s2lnovczv33cdbtcgjhz3sofe5lywcqjcuteqnc2vvey

Conversion code:

import { base58btc } from 'multiformats/bases/base58'
import { CID } from 'multiformats/cid'
import * as multihash from 'multiformats/hashes/digest'
import assert from 'node:assert'

const mh = process.argv[2]
assert.ok(mh, 'multihash is required')
const cid = CID.create(1, 0x55 /* raw */, multihash.decode(base58btc.decode('z' + mh))).toString()
console.log(cid)

[bajtos]

Constraints

1. Only deals created on or after 2025-03-27 are indexed. When we attempt to sample a deal created before 2025-03-27, IPNI will return an empty response. Spark must not report an unsuccessful retrieval in such a case because this is not something the SP can fix.
2. The reverse index sampling requires two arguments: Provider PeerID and ContextID. We cannot construct ContextID from PieceInfo, because most SPs don't follow the new convention yet. (Only Curio implements the new convention, Boost & Venus do not.)

Proposed design

Happy path in the checker node for a task (minerId, payloadCid):

1. Obtain miner's index provider peer ID. (Already implemented.)
2. Find the retrieval provider record matching the miner in the response of https://cid.contact/cid/{payloadCid}. (Already implemented.)
3. Extract the ContextID value from the record found in step 2.
4. Query the reverse index using this ContextID. Build a new CID from the returned multihash (sampledCid) using CID v1 and Raw codec.
5. Continue with the existing retrieval check, but retrieve sampledCid instead of payloadCid.
   • TODO: can we use the multiaddr + protocol from the retrieval provider record found in step 2, or do we need to query the CID contact again using the new CID?
6. Add two new fields to the measurement object submitted to spark-api:
   • ipniContextId set to ContextID found in step 3
   • cidChecked set to sampledCid found in step 4
     By preserving the current cid field used by tasking evaluation, we avoid the need to update spark-evaluate.

Edge case: The reverse index query did not return any sample.

• Set IPNI result to a new enum value, e.g. NOT_IN_REVERSE_INDEX. This will allow us to detect this situation in data collected from the network.
• Set sampledCid to payloadCid and continue with the retrieval check.

Notes

• When querying the reverse index, we must use the DRAND beacon from the time when the Spark round started as the random seed (beacon query parameter). See getRandomnessForSparkRound(). We should also set max=1 in the query.

• In spark-evaluate, we need to update the consensus check (source code) to include the two new fields.

• In spark-evaluate, we need to update getRetrievalResult() to treat NOT_IN_REVERSE_INDEX as a success, similarly to how we handle HTTP_NOT_ADVERTISED. (source code).

• In the implementation plan, we need to handle the gradual rollout to the checker network. After we ship the new checker version, it will take some time (possibly more than one round) until all checkers pick up the new Spark version. During that time, we will be collecting mixed measurements - some of them will report retrievability of the sampled CID, others will not.

  One option is to introduce a feature flag in the round definition.

  • This flag will be set to false until most of the network picks up the new checker version, so checkers running the new version keep executing the old algorithm.
  • It's important to store this flag value in the database (spark_rounds table, see here), so that the REST API returns the same flag value for historical rounds. (That's required to keep Spark data verifiable.)
  • Another benefit of this approach is that we will have a "kill switch" - in case we discover a major bug in the new algorithm, we can quickly revert to the current proven one.

Next steps

• Review the proposal above. Can you identify any problems, gaps or new attack vectors?

• It would be great to have a design document we can share with people in the Filecoin ecosystem. This document should target people who are not deeply familiar with Spark details and explain:

  1. What problem are we solving? What are the constraints?
  2. A high-level description of the solution.
  3. Impact on the RSR score and SPs

• Implementation plan, including well-scoped GitHub issues added to children of Spark retrieval checks for random payload blocks #260

[bajtos]

TODO: can we use the multiaddr + protocol from the retrieval provider record found in step 2, or do we need to query the CID contact again using the new CID?

We can use the multiaddr + protocol found in step 2.

Discussion on Slack: https://space-meridian.slack.com/archives/C079K4JT2UC/p1744786055557899