Skip to content

Simplify check for truncated CARs #113

@olizilla

Description

@olizilla

Since #100 landed we have a second worker process that reads the entire CAR from S3 and uses linkdex to determine if the CAR is complete as a signal that the CAR was probably not truncated during upload.

We could simplify this as we have a local kubo sidecar for every pickup worker with the entire DAG in it.

  • add a through stream to decode the CAR as we send it to S3.
  • count each block in the CAR as we send it.
  • if the upload completes successfully then do a ipfs dag stat on the root CID to find the expected block count for the DAG and compare it to our count of blocks in the CAR.

Both ipfs dag stat and ipfs dag export dedupe repeated blocks, so the count of blocks in an untruncated CAR and the count of blocks in the dag should always match. We should attempt to verify the hash of the last block in the CAR as well to check that the last block is complete.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions