-
Notifications
You must be signed in to change notification settings - Fork 1
Closed
Description
Since #100 landed we have a second worker process that reads the entire CAR from S3 and uses linkdex to determine if the CAR is complete as a signal that the CAR was probably not truncated during upload.
We could simplify this as we have a local kubo sidecar for every pickup worker with the entire DAG in it.
- add a through stream to decode the CAR as we send it to S3.
- count each block in the CAR as we send it.
- if the upload completes successfully then do a
ipfs dag staton the root CID to find the expected block count for the DAG and compare it to our count of blocks in the CAR.
Both ipfs dag stat and ipfs dag export dedupe repeated blocks, so the count of blocks in an untruncated CAR and the count of blocks in the dag should always match. We should attempt to verify the hash of the last block in the CAR as well to check that the last block is complete.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels