You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Since #100 landed we have a second worker process that reads the entire CAR from S3 and uses linkdex to determine if the CAR is complete as a signal that the CAR was probably not truncated during upload.
We could simplify this as we have a local kubo sidecar for every pickup worker with the entire DAG in it.
add a through stream to decode the CAR as we send it to S3.
count each block in the CAR as we send it.
if the upload completes successfully then do a ipfs dag stat on the root CID to find the expected block count for the DAG and compare it to our count of blocks in the CAR.
Both ipfs dag stat and ipfs dag export dedupe repeated blocks, so the count of blocks in an untruncated CAR and the count of blocks in the dag should always match. We should attempt to verify the hash of the last block in the CAR as well to check that the last block is complete.
The text was updated successfully, but these errors were encountered:
we can't dag stat after writing the car as we don't know when repo gc will occur. We have simplified this sufficiently by streaming the CAR back through lindex as part of the pickup containers job.
Since #100 landed we have a second worker process that reads the entire CAR from S3 and uses linkdex to determine if the CAR is complete as a signal that the CAR was probably not truncated during upload.
We could simplify this as we have a local kubo sidecar for every pickup worker with the entire DAG in it.
ipfs dag stat
on the root CID to find the expected block count for the DAG and compare it to our count of blocks in the CAR.Both
ipfs dag stat
andipfs dag export
dedupe repeated blocks, so the count of blocks in an untruncated CAR and the count of blocks in the dag should always match. We should attempt to verify the hash of the last block in the CAR as well to check that the last block is complete.The text was updated successfully, but these errors were encountered: