fix: don't allow go-merkledag to reorder loaded links #675
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ref: #673 (comment) Ideally we can remove go-merkledag entirely from here because most (all?) of the deal-making infrastructure uses go-codec-dagpb + go-ipld-prime traversals and post-decode link-reordering is a subtle difference that impacts CAR ordering for non-spec-compliant DAG-PB blocks.
rvagg
force-pushed
the
rvagg/fix-traversal-link-reordering
branch
from
141961e
to
df0aed4
Compare
|
Added test case for this which fails on |
rvagg
force-pushed
the
rvagg/fix-traversal-link-reordering
branch
from
df0aed4
to
9707212
Compare
masih
approved these changes
Aug 3, 2022
willscott
reviewed
Aug 3, 2022
| @@ -106,8 +106,13 @@ func (s *CarOffsetWriter) writeBlocks(ctx context.Context, w io.Writer, headerSi | |||
| return nil, fmt.Errorf("getting block %s: %w", c, err) | |||
| } | |||
|
|
|||
| // take a copy of the links array before nd.RawData() triggers a sort | |||
There was a problem hiding this comment.
the real solution is to get ipld/go-car#291 to replace this class, right?
There was a problem hiding this comment.
yes, that's the proper thing that needs to be done
willscott
approved these changes
Aug 3, 2022
rvagg
added a commit
to ipfs/go-merkledag
that referenced
this pull request
Aug 16, 2022
When decoding a badly serialised block with Links out of order, don't sort the list until we receive an explicit mutation operation. This ensures stable DAG traversal ordering based on the links as they appear in the serialised form and removes surprise-sorting when performing certain operations that wouldn't be expected to mutate. The pre-v0.4.0 behaviour was to always sort, but this behaviour wasn't baked in to the dag-pb spec and wasn't built into go-codec-dagpb which now forms the backend of ProtoNode, although remnants of sorting remain in some operations. Almost all CAR-from-DAG creation code in Go uses go-codec-dagpb and go-ipld-prime's traversal engine. However this can result in a different order when encountering badly encoded blocks (unsorted Links) where certain intermediate operations are performed on the ProtoNode prior to obtaining the Links() list (Links() itself doesn't sort, but e.g. RawData() does). The included "TestLinkSorting/decode" test is the only case that passes without this patch. Ref: ipld/ipld#233 Ref: filecoin-project/boost#673 Ref: filecoin-project/boost#675
rvagg
added a commit
to ipfs/go-merkledag
that referenced
this pull request
Aug 16, 2022
When decoding a badly serialised block with Links out of order, don't sort the list until we receive an explicit mutation operation. This ensures stable DAG traversal ordering based on the links as they appear in the serialised form and removes surprise-sorting when performing certain operations that wouldn't be expected to mutate. The pre-v0.4.0 behaviour was to always sort, but this behaviour wasn't baked in to the dag-pb spec and wasn't built into go-codec-dagpb which now forms the backend of ProtoNode, although remnants of sorting remain in some operations. Almost all CAR-from-DAG creation code in Go uses go-codec-dagpb and go-ipld-prime's traversal engine. However this can result in a different order when encountering badly encoded blocks (unsorted Links) where certain intermediate operations are performed on the ProtoNode prior to obtaining the Links() list (Links() itself doesn't sort, but e.g. RawData() does). The included "TestLinkSorting/decode" test is the only case that passes without this patch. Ref: ipld/ipld#233 Ref: filecoin-project/boost#673 Ref: filecoin-project/boost#675
rvagg
added a commit
to ipfs/go-merkledag
that referenced
this pull request
Aug 22, 2022
When decoding a badly serialised block with Links out of order, don't sort the list until we receive an explicit mutation operation. This ensures stable DAG traversal ordering based on the links as they appear in the serialised form and removes surprise-sorting when performing certain operations that wouldn't be expected to mutate. The pre-v0.4.0 behaviour was to always sort, but this behaviour wasn't baked in to the dag-pb spec and wasn't built into go-codec-dagpb which now forms the backend of ProtoNode, although remnants of sorting remain in some operations. Almost all CAR-from-DAG creation code in Go uses go-codec-dagpb and go-ipld-prime's traversal engine. However this can result in a different order when encountering badly encoded blocks (unsorted Links) where certain intermediate operations are performed on the ProtoNode prior to obtaining the Links() list (Links() itself doesn't sort, but e.g. RawData() does). The included "TestLinkSorting/decode" test is the only case that passes without this patch. Ref: ipld/ipld#233 Ref: filecoin-project/boost#673 Ref: filecoin-project/boost#675
rvagg
added a commit
to ipfs/go-merkledag
that referenced
this pull request
Aug 25, 2022
When decoding a badly serialised block with Links out of order, don't sort the list until we receive an explicit mutation operation. This ensures stable DAG traversal ordering based on the links as they appear in the serialised form and removes surprise-sorting when performing certain operations that wouldn't be expected to mutate. The pre-v0.4.0 behaviour was to always sort, but this behaviour wasn't baked in to the dag-pb spec and wasn't built into go-codec-dagpb which now forms the backend of ProtoNode, although remnants of sorting remain in some operations. Almost all CAR-from-DAG creation code in Go uses go-codec-dagpb and go-ipld-prime's traversal engine. However this can result in a different order when encountering badly encoded blocks (unsorted Links) where certain intermediate operations are performed on the ProtoNode prior to obtaining the Links() list (Links() itself doesn't sort, but e.g. RawData() does). The included "TestLinkSorting/decode" test is the only case that passes without this patch. Ref: ipld/ipld#233 Ref: filecoin-project/boost#673 Ref: filecoin-project/boost#675
Jorropo
pushed a commit
to ipfs/boxo
that referenced
this pull request
Mar 15, 2023
When decoding a badly serialised block with Links out of order, don't sort the list until we receive an explicit mutation operation. This ensures stable DAG traversal ordering based on the links as they appear in the serialised form and removes surprise-sorting when performing certain operations that wouldn't be expected to mutate. The pre-v0.4.0 behaviour was to always sort, but this behaviour wasn't baked in to the dag-pb spec and wasn't built into go-codec-dagpb which now forms the backend of ProtoNode, although remnants of sorting remain in some operations. Almost all CAR-from-DAG creation code in Go uses go-codec-dagpb and go-ipld-prime's traversal engine. However this can result in a different order when encountering badly encoded blocks (unsorted Links) where certain intermediate operations are performed on the ProtoNode prior to obtaining the Links() list (Links() itself doesn't sort, but e.g. RawData() does). The included "TestLinkSorting/decode" test is the only case that passes without this patch. Ref: ipld/ipld#233 Ref: filecoin-project/boost#673 Ref: filecoin-project/boost#675 This commit was moved from ipfs/go-merkledag@48c7202
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ref: #673 (comment)
Ideally we can remove go-merkledag entirely from here because most (all?) of the deal-making infrastructure uses go-codec-dagpb + go-ipld-prime traversals and post-decode link-reordering is a subtle difference that impacts CAR ordering for non-spec-compliant DAG-PB blocks.
But for now, this is a quick-fix that should preserve the links in the order as they are appear in the block, which is what our other traversals are working with.