diff --git a/CHANGELOG.md b/CHANGELOG.md index c1fac6a714f..5f4eb669243 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,28 @@ # go-ipfs changelog +## v0.10.0 TBD + +**IPLD Levels Up** + +The handling of data serialization as well as many aspects of DAG traversal and pathing have been migrated from older libraries, including [go-merkledag](https://github.com/ipfs/go-merkledag) and [go-ipld-format](https://github.com/ipfs/go-ipld-format) to the new **[go-ipld-prime](https://github.com/ipld/go-ipld-prime)** library and its components. This allows us to use many of the newer tools afforded by go-ipld-prime, stricter and more uniform codec implementations, support for additional (pluggable) codecs, and some minor performance improvements. + +This is significant refactor of a core component that touches many parts of IPFS, and does come with some **breaking changes**: + +* **IPLD plugins**: + * The `PluginIPLD` interface has been changed to utilize go-ipld-prime. There is a demonstration of the change in the [bundled git plugin](./plugin/plugins/git/). +* **The semantics of `dag put` and `dag get` change**: + * `dag get` now takes the `format` option which accepts a multicodec name used to encode the output. By default this is `dag-json`. Users may notice differences from the previously plain Go JSON output, particularly where bytes are concerned which are now encoded using a form similar to CIDs: `{"/":{"bytes":"unpadded-base64-bytes"}}` rather than the previously Go-specific plain padded base64 string. See the [dag-json specification](https://ipld.io/specs/codecs/dag-json/spec/) for an explanation of these forms. + * `dag get` no longer prints an additional new-line character at the end of the encoded block output. This means that the output as presented by `dag get` are the exact bytes of the requested node. A round-trip of such bytes back in through `dag put` using the same codec should result in the same CID. + * `dag put` uses the `input-enc` option to specify the multicodec name of the format data is being provided in, and the `format` option to specify the multicodec multicodec name of the format the data should be stored in. These formerly defaulted to `json` and `cbor` respectively. They now default to `dag-json` and `dag-cbor` respectively but may be changed to any supported codec (bundled or loaded via plugin) by its [multicodec name](https://github.com/multiformats/multicodec/blob/master/table.csv). + * The `json` and `cbor` multicodec names (as used by `input-enc` and `format` options) are now no longer aliases for `dag-json` and `dag-cbor` respectively. Instead, they now refer to their proper [multicodec](https://github.com/multiformats/multicodec/blob/master/table.csv) types. `cbor` refers to a plain CBOR format, which will not encode CIDs and does not have strict deterministic encoding rules. `json` is a plain JSON format, which also won't encode CIDs and will encode bytes in the Go-specific padded base64 string format rather than the dag-json method of byte encoding. See https://ipld.io/specs/codecs/ for more information on IPLD codecs. +* The **dag-pb codec**, which is used to encode UnixFS data for IPFS, is now represented in a form via the `dag` API that mirrors the protobuf schema used to define the binary format and unifies the implementations and specification of dag-pb across the IPLD and IPFS stacks. Previously, additional layers of code within IPFS between protobuf serialization and UnixFS handling for file and directory data, obscured the forms that are described by the protobuf representation. Much of this code has now been replaced and there are fewer layers of transformation. This means that interacting with dag-pb data via the `dag` API will use different forms: + * Previously, using `dag get` on a dag-pb block would present the block serialized as JSON as `{"data":"padded-base64-bytes","links":[{"Name":"foo","Size":100,"Cid":{"/":"Qm..."}},...]}`. + * Using the dag-pb data model specification and the new default dag-json codec for output, this will now be serialized as: `{"Data":{"/":{"bytes":"unpadded-base64-bytes"}},"Links":[{"Name":"foo","Tsize":100,"Hash":{"/":"Qm..."}},...]}`. Aside from the change in byte formatting, most field names have changed: `data` → `Data`, `links` → `Links`, `Size` → `Tsize`, `Cid` → `Hash`. Note that this output can be changed now using the `--format` option to specify an alternative codec. + * Using `dag put` and a `format` option of `dag-pb` now requires that the input conform to this dag-pb specified form. Previously, input using `{"data":"...","links":[...]}` was accepted, now it must be `{"Data":"...","Links":[...]}`. + * Previously it was not possible to use paths to navigate to any of these properties of a dag-pb node, the only possible paths were named links, e.g. `dag get QmFoo/NamedLink` where `NamedLink` was one of the links whose name was `NamedLink`. This functionality remains the same, but by prefixing the path with `/ipld/` we enter data model pathing semantics and can `dag get /ipld/QmFoo/Links/0/Hash` to navigate to links or `/ipld/QmFoo/Data` to simply retrieve the data section of the node, for example. + * See the [dag-pb specification](https://ipld.io/specs/codecs/dag-pb/) for details on the codec and its data model representation. + * See this [detailed write-up](https://github.com/ipld/ipld/blob/master/design/tricky-choices/dag-pb-forms-impl-and-use.md) for further background on these changes. + ## v0.9.1 2021-07-20 This is a small bug fix release resolving the following issues: diff --git a/core/commands/dag/dag.go b/core/commands/dag/dag.go index cdfa8ce9661..8cd9864a6b6 100644 --- a/core/commands/dag/dag.go +++ b/core/commands/dag/dag.go @@ -77,10 +77,10 @@ into an object of the specified format. cmds.FileArg("object data", true, true, "The object to put").EnableStdin(), }, Options: []cmds.Option{ - cmds.StringOption("format", "f", "Format that the object will be added as.").WithDefault("cbor"), - cmds.StringOption("input-enc", "Format that the input object will be.").WithDefault("json"), + cmds.StringOption("format", "f", "Format that the object will be added as.").WithDefault("dag-cbor"), + cmds.StringOption("input-enc", "Format that the input object will be.").WithDefault("dag-json"), cmds.BoolOption("pin", "Pin this object when adding."), - cmds.StringOption("hash", "Hash function to use").WithDefault(""), + cmds.StringOption("hash", "Hash function to use").WithDefault("sha2-256"), }, Run: dagPut, Type: OutputObject{}, @@ -108,6 +108,9 @@ format. Arguments: []cmds.Argument{ cmds.StringArg("ref", true, false, "The object to get").EnableStdin(), }, + Options: []cmds.Option{ + cmds.StringOption("format", "f", "Format that the object will be serialized as.").WithDefault("dag-json"), + }, Run: dagGet, } diff --git a/core/commands/dag/get.go b/core/commands/dag/get.go index a5f92273ae3..bcc7c601709 100644 --- a/core/commands/dag/get.go +++ b/core/commands/dag/get.go @@ -1,11 +1,18 @@ package dagcmd import ( - "strings" + "fmt" + "io" "github.com/ipfs/go-ipfs/core/commands/cmdenv" + ipldlegacy "github.com/ipfs/go-ipld-legacy" "github.com/ipfs/interface-go-ipfs-core/path" + "github.com/ipld/go-ipld-prime" + "github.com/ipld/go-ipld-prime/multicodec" + "github.com/ipld/go-ipld-prime/traversal" + mc "github.com/multiformats/go-multicodec" + cmds "github.com/ipfs/go-ipfs-cmds" ) @@ -15,6 +22,12 @@ func dagGet(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e return err } + format, _ := req.Options["format"].(string) + var fCodec mc.Code + if err := fCodec.Set(format); err != nil { + return err + } + rp, err := api.ResolvePath(req.Context, path.New(req.Arguments[0])) if err != nil { return err @@ -25,14 +38,34 @@ func dagGet(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e return err } - var out interface{} = obj + universal, ok := obj.(ipldlegacy.UniversalNode) + if !ok { + return fmt.Errorf("%T is not a valid IPLD node", obj) + } + + finalNode := universal.(ipld.Node) + if len(rp.Remainder()) > 0 { - rem := strings.Split(rp.Remainder(), "/") - final, _, err := obj.Resolve(rem) + remainderPath := ipld.ParsePath(rp.Remainder()) + + finalNode, err = traversal.Get(finalNode, remainderPath) if err != nil { return err } - out = final } - return cmds.EmitOnce(res, &out) + + encoder, err := multicodec.LookupEncoder(uint64(fCodec)) + if err != nil { + return fmt.Errorf("invalid encoding: %s - %s", format, err) + } + + r, w := io.Pipe() + go func() { + defer w.Close() + if err := encoder(finalNode, w); err != nil { + _ = w.CloseWithError(err) + } + }() + + return res.Emit(r) } diff --git a/core/commands/dag/put.go b/core/commands/dag/put.go index 7f6e744c872..f98ab48e464 100644 --- a/core/commands/dag/put.go +++ b/core/commands/dag/put.go @@ -1,16 +1,28 @@ package dagcmd import ( + "bytes" "fmt" - "math" + blocks "github.com/ipfs/go-block-format" + "github.com/ipfs/go-cid" "github.com/ipfs/go-ipfs/core/commands/cmdenv" - "github.com/ipfs/go-ipfs/core/coredag" + ipldlegacy "github.com/ipfs/go-ipld-legacy" + "github.com/ipld/go-ipld-prime/multicodec" + basicnode "github.com/ipld/go-ipld-prime/node/basic" cmds "github.com/ipfs/go-ipfs-cmds" files "github.com/ipfs/go-ipfs-files" ipld "github.com/ipfs/go-ipld-format" - mh "github.com/multiformats/go-multihash" + mc "github.com/multiformats/go-multicodec" + + // Expected minimal set of available format/ienc codecs. + _ "github.com/ipld/go-codec-dagpb" + _ "github.com/ipld/go-ipld-prime/codec/cbor" + _ "github.com/ipld/go-ipld-prime/codec/dagcbor" + _ "github.com/ipld/go-ipld-prime/codec/dagjson" + _ "github.com/ipld/go-ipld-prime/codec/json" + _ "github.com/ipld/go-ipld-prime/codec/raw" ) func dagPut(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) error { @@ -24,16 +36,33 @@ func dagPut(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e hash, _ := req.Options["hash"].(string) dopin, _ := req.Options["pin"].(bool) - // mhType tells inputParser which hash should be used. MaxUint64 means 'use - // default hash' (sha256 for cbor, sha1 for git..) - mhType := uint64(math.MaxUint64) + var icodec mc.Code + if err := icodec.Set(ienc); err != nil { + return err + } + var fcodec mc.Code + if err := fcodec.Set(format); err != nil { + return err + } + var mhType mc.Code + if err := mhType.Set(hash); err != nil { + return err + } - if hash != "" { - var ok bool - mhType, ok = mh.Names[hash] - if !ok { - return fmt.Errorf("%s in not a valid multihash name", hash) - } + cidPrefix := cid.Prefix{ + Version: 1, + Codec: uint64(fcodec), + MhType: uint64(mhType), + MhLength: -1, + } + + decoder, err := multicodec.LookupDecoder(uint64(icodec)) + if err != nil { + return err + } + encoder, err := multicodec.LookupEncoder(uint64(fcodec)) + if err != nil { + return err } var adder ipld.NodeAdder = api.Dag() @@ -48,22 +77,36 @@ func dagPut(req *cmds.Request, res cmds.ResponseEmitter, env cmds.Environment) e if file == nil { return fmt.Errorf("expected a regular file") } - nds, err := coredag.ParseInputs(ienc, format, file, mhType, -1) + + node := basicnode.Prototype.Any.NewBuilder() + if err := decoder(node, file); err != nil { + return err + } + n := node.Build() + + bd := bytes.NewBuffer([]byte{}) + if err := encoder(n, bd); err != nil { + return err + } + + blockCid, err := cidPrefix.Sum(bd.Bytes()) + if err != nil { + return err + } + blk, err := blocks.NewBlockWithCid(bd.Bytes(), blockCid) if err != nil { return err } - if len(nds) == 0 { - return fmt.Errorf("no node returned from ParseInputs") + ln := ipldlegacy.LegacyNode{ + Block: blk, + Node: n, } - for _, nd := range nds { - err := b.Add(req.Context, nd) - if err != nil { - return err - } + if err := b.Add(req.Context, &ln); err != nil { + return err } - cid := nds[0].Cid() + cid := ln.Cid() if err := res.Emit(&OutputObject{Cid: cid}); err != nil { return err } diff --git a/core/core.go b/core/core.go index 250af72988a..9bcba1329f7 100644 --- a/core/core.go +++ b/core/core.go @@ -17,6 +17,7 @@ import ( "github.com/ipfs/go-ipfs-pinner" bserv "github.com/ipfs/go-blockservice" + "github.com/ipfs/go-fetcher" "github.com/ipfs/go-graphsync" bstore "github.com/ipfs/go-ipfs-blockstore" exchange "github.com/ipfs/go-ipfs-exchange-interface" @@ -24,7 +25,6 @@ import ( ipld "github.com/ipfs/go-ipld-format" logging "github.com/ipfs/go-log" mfs "github.com/ipfs/go-mfs" - resolver "github.com/ipfs/go-path/resolver" goprocess "github.com/jbenet/goprocess" connmgr "github.com/libp2p/go-libp2p-core/connmgr" ic "github.com/libp2p/go-libp2p-core/crypto" @@ -70,18 +70,19 @@ type IpfsNode struct { PNetFingerprint libp2p.PNetFingerprint `optional:"true"` // fingerprint of private network // Services - Peerstore pstore.Peerstore `optional:"true"` // storage for other Peer instances - Blockstore bstore.GCBlockstore // the block store (lower level) - Filestore *filestore.Filestore `optional:"true"` // the filestore blockstore - BaseBlocks node.BaseBlocks // the raw blockstore, no filestore wrapping - GCLocker bstore.GCLocker // the locker used to protect the blockstore during gc - Blocks bserv.BlockService // the block service, get/add blocks. - DAG ipld.DAGService // the merkle dag service, get/add objects. - Resolver *resolver.Resolver // the path resolution system - Reporter *metrics.BandwidthCounter `optional:"true"` - Discovery discovery.Service `optional:"true"` - FilesRoot *mfs.Root - RecordValidator record.Validator + Peerstore pstore.Peerstore `optional:"true"` // storage for other Peer instances + Blockstore bstore.GCBlockstore // the block store (lower level) + Filestore *filestore.Filestore `optional:"true"` // the filestore blockstore + BaseBlocks node.BaseBlocks // the raw blockstore, no filestore wrapping + GCLocker bstore.GCLocker // the locker used to protect the blockstore during gc + Blocks bserv.BlockService // the block service, get/add blocks. + DAG ipld.DAGService // the merkle dag service, get/add objects. + IPLDFetcherFactory fetcher.Factory `name:"ipldFetcher"` // fetcher that paths over the IPLD data model + UnixFSFetcherFactory fetcher.Factory `name:"unixfsFetcher"` // fetcher that interprets UnixFS data + Reporter *metrics.BandwidthCounter `optional:"true"` + Discovery discovery.Service `optional:"true"` + FilesRoot *mfs.Root + RecordValidator record.Validator // Online PeerHost p2phost.Host `optional:"true"` // the network host (server+client) diff --git a/core/coreapi/coreapi.go b/core/coreapi/coreapi.go index f8c165a0cd3..3d31abaabc2 100644 --- a/core/coreapi/coreapi.go +++ b/core/coreapi/coreapi.go @@ -19,11 +19,12 @@ import ( "fmt" bserv "github.com/ipfs/go-blockservice" - "github.com/ipfs/go-ipfs-blockstore" - "github.com/ipfs/go-ipfs-exchange-interface" + "github.com/ipfs/go-fetcher" + blockstore "github.com/ipfs/go-ipfs-blockstore" + exchange "github.com/ipfs/go-ipfs-exchange-interface" offlinexch "github.com/ipfs/go-ipfs-exchange-offline" - "github.com/ipfs/go-ipfs-pinner" - "github.com/ipfs/go-ipfs-provider" + pin "github.com/ipfs/go-ipfs-pinner" + provider "github.com/ipfs/go-ipfs-provider" offlineroute "github.com/ipfs/go-ipfs-routing/offline" ipld "github.com/ipfs/go-ipld-format" dag "github.com/ipfs/go-merkledag" @@ -55,13 +56,14 @@ type CoreAPI struct { baseBlocks blockstore.Blockstore pinning pin.Pinner - blocks bserv.BlockService - dag ipld.DAGService - - peerstore pstore.Peerstore - peerHost p2phost.Host - recordValidator record.Validator - exchange exchange.Interface + blocks bserv.BlockService + dag ipld.DAGService + ipldFetcherFactory fetcher.Factory + unixFSFetcherFactory fetcher.Factory + peerstore pstore.Peerstore + peerHost p2phost.Host + recordValidator record.Validator + exchange exchange.Interface namesys namesys.NameSystem routing routing.Routing @@ -167,8 +169,10 @@ func (api *CoreAPI) WithOptions(opts ...options.ApiOption) (coreiface.CoreAPI, e baseBlocks: n.BaseBlocks, pinning: n.Pinning, - blocks: n.Blocks, - dag: n.DAG, + blocks: n.Blocks, + dag: n.DAG, + ipldFetcherFactory: n.IPLDFetcherFactory, + unixFSFetcherFactory: n.UnixFSFetcherFactory, peerstore: n.Peerstore, peerHost: n.PeerHost, diff --git a/core/coreapi/path.go b/core/coreapi/path.go index 598f5d11581..b9bf83e0df6 100644 --- a/core/coreapi/path.go +++ b/core/coreapi/path.go @@ -8,10 +8,10 @@ import ( "github.com/ipfs/go-namesys/resolve" "github.com/ipfs/go-cid" + "github.com/ipfs/go-fetcher" ipld "github.com/ipfs/go-ipld-format" ipfspath "github.com/ipfs/go-path" - "github.com/ipfs/go-path/resolver" - uio "github.com/ipfs/go-unixfs/io" + ipfspathresolver "github.com/ipfs/go-path/resolver" coreiface "github.com/ipfs/interface-go-ipfs-core" path "github.com/ipfs/interface-go-ipfs-core/path" ) @@ -49,23 +49,19 @@ func (api *CoreAPI) ResolvePath(ctx context.Context, p path.Path) (path.Resolved return nil, err } - var resolveOnce resolver.ResolveOnce - - switch ipath.Segments()[0] { - case "ipfs": - resolveOnce = uio.ResolveUnixfsOnce - case "ipld": - resolveOnce = resolver.ResolveSingle - default: + if ipath.Segments()[0] != "ipfs" && ipath.Segments()[0] != "ipld" { return nil, fmt.Errorf("unsupported path namespace: %s", p.Namespace()) } - r := &resolver.Resolver{ - DAG: api.dag, - ResolveOnce: resolveOnce, + var dataFetcher fetcher.Factory + if ipath.Segments()[0] == "ipld" { + dataFetcher = api.ipldFetcherFactory + } else { + dataFetcher = api.unixFSFetcherFactory } + resolver := ipfspathresolver.NewBasicResolver(dataFetcher) - node, rest, err := r.ResolveToLastNode(ctx, ipath) + node, rest, err := resolver.ResolveToLastNode(ctx, ipath) if err != nil { return nil, err } diff --git a/core/node/core.go b/core/node/core.go index 48e37d600f6..9e7d75dcc20 100644 --- a/core/node/core.go +++ b/core/node/core.go @@ -3,12 +3,13 @@ package node import ( "context" "fmt" - "github.com/ipfs/go-bitswap" "github.com/ipfs/go-bitswap/network" "github.com/ipfs/go-blockservice" "github.com/ipfs/go-cid" "github.com/ipfs/go-datastore" + "github.com/ipfs/go-fetcher" + bsfetcher "github.com/ipfs/go-fetcher/impl/blockservice" "github.com/ipfs/go-filestore" blockstore "github.com/ipfs/go-ipfs-blockstore" exchange "github.com/ipfs/go-ipfs-exchange-interface" @@ -18,6 +19,11 @@ import ( "github.com/ipfs/go-merkledag" "github.com/ipfs/go-mfs" "github.com/ipfs/go-unixfs" + "github.com/ipfs/go-unixfsnode" + dagpb "github.com/ipld/go-codec-dagpb" + "github.com/ipld/go-ipld-prime" + basicnode "github.com/ipld/go-ipld-prime/node/basic" + "github.com/ipld/go-ipld-prime/schema" "github.com/libp2p/go-libp2p-core/host" "github.com/libp2p/go-libp2p-core/routing" "go.uber.org/fx" @@ -80,6 +86,26 @@ func (s *syncDagService) Session(ctx context.Context) format.NodeGetter { return merkledag.NewSession(ctx, s.DAGService) } +type fetchersOut struct { + fx.Out + IPLDFetcher fetcher.Factory `name:"ipldFetcher"` + UnixfsFetcher fetcher.Factory `name:"unixfsFetcher"` +} + +// FetcherConfig returns a fetcher config that can build new fetcher instances +func FetcherConfig(bs blockservice.BlockService) fetchersOut { + ipldFetcher := bsfetcher.NewFetcherConfig(bs) + ipldFetcher.PrototypeChooser = dagpb.AddSupportToChooser(func(lnk ipld.Link, lnkCtx ipld.LinkContext) (ipld.NodePrototype, error) { + if tlnkNd, ok := lnkCtx.LinkNode.(schema.TypedLinkNode); ok { + return tlnkNd.LinkTargetNodePrototype(), nil + } + return basicnode.Prototype.Any, nil + }) + + unixFSFetcher := ipldFetcher.WithReifier(unixfsnode.Reify) + return fetchersOut{IPLDFetcher: ipldFetcher, UnixfsFetcher: unixFSFetcher} +} + // Dag creates new DAGService func Dag(bs blockservice.BlockService) format.DAGService { return merkledag.NewDAGService(bs) diff --git a/core/node/groups.go b/core/node/groups.go index 8d37f84ce13..0cb0f1ab03a 100644 --- a/core/node/groups.go +++ b/core/node/groups.go @@ -18,7 +18,6 @@ import ( offline "github.com/ipfs/go-ipfs-exchange-offline" offroute "github.com/ipfs/go-ipfs-routing/offline" - "github.com/ipfs/go-path/resolver" uio "github.com/ipfs/go-unixfs/io" "go.uber.org/fx" ) @@ -294,7 +293,7 @@ func Offline(cfg *config.Config) fx.Option { var Core = fx.Options( fx.Provide(BlockService), fx.Provide(Dag), - fx.Provide(resolver.NewBasicResolver), + fx.Provide(FetcherConfig), fx.Provide(Pinning), fx.Provide(Files), ) diff --git a/core/node/provider.go b/core/node/provider.go index e865d2b5fd9..3dd7f9b728a 100644 --- a/core/node/provider.go +++ b/core/node/provider.go @@ -5,12 +5,12 @@ import ( "fmt" "time" + "github.com/ipfs/go-fetcher" "github.com/ipfs/go-ipfs-pinner" "github.com/ipfs/go-ipfs-provider" "github.com/ipfs/go-ipfs-provider/batched" q "github.com/ipfs/go-ipfs-provider/queue" "github.com/ipfs/go-ipfs-provider/simple" - ipld "github.com/ipfs/go-ipld-format" "github.com/libp2p/go-libp2p-core/routing" "github.com/multiformats/go-multihash" "go.uber.org/fx" @@ -172,7 +172,12 @@ func SimpleProviders(reprovideStrategy string, reprovideInterval string) fx.Opti } func pinnedProviderStrategy(onlyRoots bool) interface{} { - return func(pinner pin.Pinner, dag ipld.DAGService) simple.KeyChanFunc { - return simple.NewPinnedProvider(onlyRoots, pinner, dag) + type input struct { + fx.In + Pinner pin.Pinner + IPLDFetcher fetcher.Factory `name:"ipldFetcher"` + } + return func(in input) simple.KeyChanFunc { + return simple.NewPinnedProvider(onlyRoots, in.Pinner, in.IPLDFetcher) } } diff --git a/fuse/readonly/readonly_unix.go b/fuse/readonly/readonly_unix.go index 866cdca1a08..dbdee5190bf 100644 --- a/fuse/readonly/readonly_unix.go +++ b/fuse/readonly/readonly_unix.go @@ -10,16 +10,18 @@ import ( "os" "syscall" + fuse "bazil.org/fuse" + fs "bazil.org/fuse/fs" + "github.com/ipfs/go-cid" core "github.com/ipfs/go-ipfs/core" + ipld "github.com/ipfs/go-ipld-format" + logging "github.com/ipfs/go-log" mdag "github.com/ipfs/go-merkledag" path "github.com/ipfs/go-path" + "github.com/ipfs/go-path/resolver" ft "github.com/ipfs/go-unixfs" uio "github.com/ipfs/go-unixfs/io" - - fuse "bazil.org/fuse" - fs "bazil.org/fuse/fs" - ipld "github.com/ipfs/go-ipld-format" - logging "github.com/ipfs/go-log" + cidlink "github.com/ipld/go-ipld-prime/linking/cid" ) var log = logging.Logger("fuse/ipfs") @@ -65,20 +67,41 @@ func (s *Root) Lookup(ctx context.Context, name string) (fs.Node, error) { return nil, fuse.ENOENT } - nd, err := s.Ipfs.Resolver.ResolvePath(ctx, p) + nd, ndLnk, err := resolver.NewBasicResolver(s.Ipfs.UnixFSFetcherFactory).ResolvePath(ctx, p) if err != nil { // todo: make this error more versatile. return nil, fuse.ENOENT } - switch nd := nd.(type) { - case *mdag.ProtoNode, *mdag.RawNode: - return &Node{Ipfs: s.Ipfs, Nd: nd}, nil + cidLnk, ok := ndLnk.(cidlink.Link) + if !ok { + log.Debugf("non-cidlink returned from ResolvePath: %v", ndLnk) + return nil, fuse.ENOENT + } + + // convert ipld-prime node to universal node + blk, err := s.Ipfs.Blockstore.Get(cidLnk.Cid) + if err != nil { + log.Debugf("fuse failed to retrieve block: %v: %s", cidLnk, err) + return nil, fuse.ENOENT + } + + var fnd ipld.Node + switch cidLnk.Cid.Prefix().Codec { + case cid.DagProtobuf: + fnd, err = mdag.ProtoNodeConverter(blk, nd) + case cid.Raw: + fnd, err = mdag.RawNodeConverter(blk, nd) default: - log.Error("fuse node was not a protobuf node") + log.Error("fuse node was not a supported type") return nil, fuse.ENOTSUP } + if err != nil { + log.Error("could not convert protobuf or raw node") + return nil, fuse.ENOENT + } + return &Node{Ipfs: s.Ipfs, Nd: fnd}, nil } // ReadDirAll reads a particular directory. Disallowed for root. diff --git a/go.mod b/go.mod index a1e8721ef62..617c5825139 100644 --- a/go.mod +++ b/go.mod @@ -14,7 +14,7 @@ require ( github.com/hashicorp/go-multierror v1.1.1 github.com/ipfs/go-bitswap v0.3.4 github.com/ipfs/go-block-format v0.0.3 - github.com/ipfs/go-blockservice v0.1.4 + github.com/ipfs/go-blockservice v0.1.7 github.com/ipfs/go-cid v0.0.7 github.com/ipfs/go-cidutil v0.0.2 github.com/ipfs/go-datastore v0.4.5 @@ -23,6 +23,7 @@ require ( github.com/ipfs/go-ds-flatfs v0.4.5 github.com/ipfs/go-ds-leveldb v0.4.2 github.com/ipfs/go-ds-measure v0.1.0 + github.com/ipfs/go-fetcher v1.5.0 github.com/ipfs/go-filestore v0.0.3 github.com/ipfs/go-fs-lock v0.0.7 github.com/ipfs/go-graphsync v0.8.0 @@ -36,26 +37,30 @@ require ( github.com/ipfs/go-ipfs-keystore v0.0.2 github.com/ipfs/go-ipfs-pinner v0.1.2 github.com/ipfs/go-ipfs-posinfo v0.0.1 - github.com/ipfs/go-ipfs-provider v0.5.1 + github.com/ipfs/go-ipfs-provider v0.6.1 github.com/ipfs/go-ipfs-routing v0.1.0 github.com/ipfs/go-ipfs-util v0.0.2 github.com/ipfs/go-ipld-cbor v0.0.5 github.com/ipfs/go-ipld-format v0.2.0 - github.com/ipfs/go-ipld-git v0.0.4 + github.com/ipfs/go-ipld-git v0.1.0 + github.com/ipfs/go-ipld-legacy v0.1.0 github.com/ipfs/go-ipns v0.1.0 github.com/ipfs/go-log v1.0.5 - github.com/ipfs/go-merkledag v0.3.2 + github.com/ipfs/go-merkledag v0.4.0 github.com/ipfs/go-metrics-interface v0.0.1 github.com/ipfs/go-metrics-prometheus v0.0.2 github.com/ipfs/go-mfs v0.1.2 github.com/ipfs/go-namesys v0.3.0 - github.com/ipfs/go-path v0.0.9 + github.com/ipfs/go-path v0.1.1 github.com/ipfs/go-pinning-service-http-client v0.1.0 github.com/ipfs/go-unixfs v0.2.5 + github.com/ipfs/go-unixfsnode v1.1.2 github.com/ipfs/go-verifcid v0.0.1 - github.com/ipfs/interface-go-ipfs-core v0.4.0 + github.com/ipfs/interface-go-ipfs-core v0.5.1 github.com/ipfs/tar-utils v0.0.1 github.com/ipld/go-car v0.3.1 + github.com/ipld/go-codec-dagpb v1.3.0 + github.com/ipld/go-ipld-prime v0.11.0 github.com/jbenet/go-random v0.0.0-20190219211222-123a90aedc0c github.com/jbenet/go-temp-err-catcher v0.1.0 github.com/jbenet/goprocess v0.1.4 @@ -90,6 +95,7 @@ require ( github.com/multiformats/go-multiaddr v0.3.3 github.com/multiformats/go-multiaddr-dns v0.3.1 github.com/multiformats/go-multibase v0.0.3 + github.com/multiformats/go-multicodec v0.3.0 github.com/multiformats/go-multihash v0.0.15 github.com/opentracing/opentracing-go v1.2.0 github.com/pkg/errors v0.9.1 @@ -101,10 +107,10 @@ require ( go.opencensus.io v0.23.0 go.uber.org/fx v1.13.1 go.uber.org/zap v1.19.0 - golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf + golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a golang.org/x/lint v0.0.0-20201208152925-83fdc39ff7b5 // indirect golang.org/x/sync v0.0.0-20210220032951-036812b2e83c - golang.org/x/sys v0.0.0-20210511113859-b0526f3d8744 + golang.org/x/sys v0.0.0-20210514084401-e8d321eab015 ) go 1.15 diff --git a/go.sum b/go.sum index a17da770743..9340b0773d9 100644 --- a/go.sum +++ b/go.sum @@ -356,10 +356,10 @@ github.com/ipfs/go-block-format v0.0.3/go.mod h1:4LmD4ZUw0mhO+JSKdpWwrzATiEfM7WW github.com/ipfs/go-blockservice v0.0.7/go.mod h1:EOfb9k/Y878ZTRY/CH0x5+ATtaipfbRhbvNSdgc/7So= github.com/ipfs/go-blockservice v0.1.0/go.mod h1:hzmMScl1kXHg3M2BjTymbVPjv627N7sYcvYaKbop39M= github.com/ipfs/go-blockservice v0.1.1/go.mod h1:t+411r7psEUhLueM8C7aPA7cxCclv4O3VsUVxt9kz2I= -github.com/ipfs/go-blockservice v0.1.2/go.mod h1:t+411r7psEUhLueM8C7aPA7cxCclv4O3VsUVxt9kz2I= github.com/ipfs/go-blockservice v0.1.3/go.mod h1:OTZhFpkgY48kNzbgyvcexW9cHrpjBYIjSR0KoDOFOLU= -github.com/ipfs/go-blockservice v0.1.4 h1:Vq+MlsH8000KbbUciRyYMEw/NNP8UAGmcqKi4uWmFGA= github.com/ipfs/go-blockservice v0.1.4/go.mod h1:OTZhFpkgY48kNzbgyvcexW9cHrpjBYIjSR0KoDOFOLU= +github.com/ipfs/go-blockservice v0.1.7 h1:yVe9te0M7ow8i+PPkx03YFSpxqzXx594d6h+34D6qMg= +github.com/ipfs/go-blockservice v0.1.7/go.mod h1:GmS+BAt4hrwBKkzE11AFDQUrnvqjwFatGS2MY7wOjEM= github.com/ipfs/go-cid v0.0.1/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-cid v0.0.2/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= github.com/ipfs/go-cid v0.0.3/go.mod h1:GHWU/WuQdMPmIosc4Yn1bcCT7dSeX4lBafM7iqUPQvM= @@ -400,6 +400,8 @@ github.com/ipfs/go-ds-leveldb v0.4.2 h1:QmQoAJ9WkPMUfBLnu1sBVy0xWWlJPg0m4kRAiJL9 github.com/ipfs/go-ds-leveldb v0.4.2/go.mod h1:jpbku/YqBSsBc1qgME8BkWS4AxzF2cEu1Ii2r79Hh9s= github.com/ipfs/go-ds-measure v0.1.0 h1:vE4TyY4aeLeVgnnPBC5QzKIjKrqzha0NCujTfgvVbVQ= github.com/ipfs/go-ds-measure v0.1.0/go.mod h1:1nDiFrhLlwArTME1Ees2XaBOl49OoCgd2A3f8EchMSY= +github.com/ipfs/go-fetcher v1.5.0 h1:oreKTKBzja3S09rSmoZlA3KGVlRiUbJ1pQjtB4K6y3w= +github.com/ipfs/go-fetcher v1.5.0/go.mod h1:5pDZ0393oRF/fHiLmtFZtpMNBQfHOYNPtryWedVuSWE= github.com/ipfs/go-filestore v0.0.3 h1:MhZ1jT5K3NewZwim6rS/akcJLm1xM+r6nz6foeB9EwE= github.com/ipfs/go-filestore v0.0.3/go.mod h1:dvXRykFzyyXN2CdNlRGzDAkXMDPyI+D7JE066SiKLSE= github.com/ipfs/go-fs-lock v0.0.7 h1:6BR3dajORFrFTkb5EpCUFIAypsoxpGpDSVUdFwzgL9U= @@ -442,8 +444,8 @@ github.com/ipfs/go-ipfs-posinfo v0.0.1/go.mod h1:SwyeVP+jCwiDu0C313l/8jg6ZxM0qqt github.com/ipfs/go-ipfs-pq v0.0.1/go.mod h1:LWIqQpqfRG3fNc5XsnIhz/wQ2XXGyugQwls7BgUmUfY= github.com/ipfs/go-ipfs-pq v0.0.2 h1:e1vOOW6MuOwG2lqxcLA+wEn93i/9laCY8sXAw76jFOY= github.com/ipfs/go-ipfs-pq v0.0.2/go.mod h1:LWIqQpqfRG3fNc5XsnIhz/wQ2XXGyugQwls7BgUmUfY= -github.com/ipfs/go-ipfs-provider v0.5.1 h1:kZj72jzWLtGcorlwnMvBL6y6KJk6klO2Kb8QSeqEB0o= -github.com/ipfs/go-ipfs-provider v0.5.1/go.mod h1:fem6HKSru7n35Ljap6kowWdJrUzvcWJW01uhAkqNnzo= +github.com/ipfs/go-ipfs-provider v0.6.1 h1:4VenAH1J5XH+/mgM2Y7p+QN3wlk7CInMDG8rsT2CGW4= +github.com/ipfs/go-ipfs-provider v0.6.1/go.mod h1:I4Cig3InhftbRJohph76Qy/P2uKEZILNGiKvDJmmC28= github.com/ipfs/go-ipfs-routing v0.0.1/go.mod h1:k76lf20iKFxQTjcJokbPM9iBXVXVZhcOwc360N4nuKs= github.com/ipfs/go-ipfs-routing v0.1.0 h1:gAJTT1cEeeLj6/DlLX6t+NxD9fQe2ymTO6qWRDI/HQQ= github.com/ipfs/go-ipfs-routing v0.1.0/go.mod h1:hYoUkJLyAUKhF58tysKpids8RNDPO42BVMgK5dNsoqY= @@ -459,8 +461,10 @@ github.com/ipfs/go-ipld-format v0.0.1/go.mod h1:kyJtbkDALmFHv3QR6et67i35QzO3S0dC github.com/ipfs/go-ipld-format v0.0.2/go.mod h1:4B6+FM2u9OJ9zCV+kSbgFAZlOrv1Hqbf0INGQgiKf9k= github.com/ipfs/go-ipld-format v0.2.0 h1:xGlJKkArkmBvowr+GMCX0FEZtkro71K1AwiKnL37mwA= github.com/ipfs/go-ipld-format v0.2.0/go.mod h1:3l3C1uKoadTPbeNfrDi+xMInYKlx2Cvg1BuydPSdzQs= -github.com/ipfs/go-ipld-git v0.0.4 h1:fQv2Alq72g6mH+heDWQ9Awu5FQYc3hcCUVtzuWj/Mno= -github.com/ipfs/go-ipld-git v0.0.4/go.mod h1:RuvMXa9qtJpDbqngyICCU/d+cmLFXxLsbIclmD0Lcr0= +github.com/ipfs/go-ipld-git v0.1.0 h1:+HUPKbvOPicyF5J4gdRS3nvQRBgvYVGyvuAigiISkzc= +github.com/ipfs/go-ipld-git v0.1.0/go.mod h1:pfIxVWClpLmyDHDWI7qTva4/cdOQyVzC+0jzl5/n1Sk= +github.com/ipfs/go-ipld-legacy v0.1.0 h1:wxkkc4k8cnvIGIjPO0waJCe7SHEyFgl+yQdafdjGrpA= +github.com/ipfs/go-ipld-legacy v0.1.0/go.mod h1:86f5P/srAmh9GcIcWQR9lfFLZPrIyyXQeVlOWeeWEuI= github.com/ipfs/go-ipns v0.0.2/go.mod h1:WChil4e0/m9cIINWLxZe1Jtf77oz5L05rO2ei/uKJ5U= github.com/ipfs/go-ipns v0.1.0 h1:jk03sneWwh+/bSHrFjRfE38xHDnTzqCsx2wsJ8ipukM= github.com/ipfs/go-ipns v0.1.0/go.mod h1:3IbsuPkR6eAGcnx+E7j6HpOSbSQJPZ6zlRj+NK3jPxQ= @@ -484,8 +488,9 @@ github.com/ipfs/go-merkledag v0.1.0/go.mod h1:SQiXrtSts3KGNmgOzMICy5c0POOpUNQLvB github.com/ipfs/go-merkledag v0.2.3/go.mod h1:SQiXrtSts3KGNmgOzMICy5c0POOpUNQLvB3ClKnBAlk= github.com/ipfs/go-merkledag v0.3.0/go.mod h1:4pymaZLhSLNVuiCITYrpViD6vmfZ/Ws4n/L9tfNv3S4= github.com/ipfs/go-merkledag v0.3.1/go.mod h1:fvkZNNZixVW6cKSZ/JfLlON5OlgTXNdRLz0p6QG/I2M= -github.com/ipfs/go-merkledag v0.3.2 h1:MRqj40QkrWkvPswXs4EfSslhZ4RVPRbxwX11js0t1xY= github.com/ipfs/go-merkledag v0.3.2/go.mod h1:fvkZNNZixVW6cKSZ/JfLlON5OlgTXNdRLz0p6QG/I2M= +github.com/ipfs/go-merkledag v0.4.0 h1:ixNu/5MJSaT/Qs073T0/HsWKwnOoBgqSq1g+GaJIen0= +github.com/ipfs/go-merkledag v0.4.0/go.mod h1:XshXBkhyeS63YNGisLL1uDSfuTyrQIxVUOg3ojR5MOE= github.com/ipfs/go-metrics-interface v0.0.1 h1:j+cpbjYvu4R8zbleSs36gvB7jR+wsL2fGD6n0jO4kdg= github.com/ipfs/go-metrics-interface v0.0.1/go.mod h1:6s6euYU4zowdslK0GKHmqaIZ3j/b/tL7HTWtJ4VPgWY= github.com/ipfs/go-metrics-prometheus v0.0.2 h1:9i2iljLg12S78OhC6UAiXi176xvQGiZaGVF1CUVdE+s= @@ -495,8 +500,9 @@ github.com/ipfs/go-mfs v0.1.2/go.mod h1:T1QBiZPEpkPLzDqEJLNnbK55BVKVlNi2a+gVm4di github.com/ipfs/go-namesys v0.3.0 h1:6lytKWj1rG0Ot6J0nTHvFw+06q1a6n7DLA2CbSGmZco= github.com/ipfs/go-namesys v0.3.0/go.mod h1:/BL4xk8LP5Lq82AmaRKyxZv/eYRlumNiU9SZUe1Hlps= github.com/ipfs/go-path v0.0.7/go.mod h1:6KTKmeRnBXgqrTvzFrPV3CamxcgvXX/4z79tfAd2Sno= -github.com/ipfs/go-path v0.0.9 h1:BIi831cNED8YnIlIKo9y1SI3u+E+FwQQD+rIIw8PwFA= github.com/ipfs/go-path v0.0.9/go.mod h1:VpDkSBKQ9EFQOUgi54Tq/O/tGi8n1RfYNks13M3DEs8= +github.com/ipfs/go-path v0.1.1 h1:0rfiI0IoNTYUyQN0ifz2zQBR6mZhOKv7qW5Jjx/4fG8= +github.com/ipfs/go-path v0.1.1/go.mod h1:vC8q4AKOtrjJz2NnllIrmr2ZbGlF5fW2OKKyhV9ggb0= github.com/ipfs/go-peertaskqueue v0.0.4/go.mod h1:03H8fhyeMfKNFWqzYEVyMbcPUeYrqP1MX6Kd+aN+rMQ= github.com/ipfs/go-peertaskqueue v0.1.0/go.mod h1:Jmk3IyCcfl1W3jTW3YpghSwSEC6IJ3Vzz/jUmWw8Z0U= github.com/ipfs/go-peertaskqueue v0.1.1/go.mod h1:Jmk3IyCcfl1W3jTW3YpghSwSEC6IJ3Vzz/jUmWw8Z0U= @@ -508,19 +514,24 @@ github.com/ipfs/go-unixfs v0.1.0/go.mod h1:lysk5ELhOso8+Fed9U1QTGey2ocsfaZ18h0NC github.com/ipfs/go-unixfs v0.2.4/go.mod h1:SUdisfUjNoSDzzhGVxvCL9QO/nKdwXdr+gbMUdqcbYw= github.com/ipfs/go-unixfs v0.2.5 h1:irj/WzIcgTBay48mSMUYDbKlIzIocXWcuUUsi5qOMOE= github.com/ipfs/go-unixfs v0.2.5/go.mod h1:SUdisfUjNoSDzzhGVxvCL9QO/nKdwXdr+gbMUdqcbYw= +github.com/ipfs/go-unixfsnode v1.1.2 h1:aTsCdhwU0F4dMShMwYGroAj4v4EzSONLdoENebvTRb0= +github.com/ipfs/go-unixfsnode v1.1.2/go.mod h1:5dcE2x03pyjHk4JjamXmunTMzz+VUtqvPwZjIEkfV6s= github.com/ipfs/go-verifcid v0.0.1 h1:m2HI7zIuR5TFyQ1b79Da5N9dnnCP1vcu2QqawmWlK2E= github.com/ipfs/go-verifcid v0.0.1/go.mod h1:5Hrva5KBeIog4A+UpqlaIU+DEstipcJYQQZc0g37pY0= -github.com/ipfs/interface-go-ipfs-core v0.4.0 h1:+mUiamyHIwedqP8ZgbCIwpy40oX7QcXUbo4CZOeJVJg= github.com/ipfs/interface-go-ipfs-core v0.4.0/go.mod h1:UJBcU6iNennuI05amq3FQ7g0JHUkibHFAfhfUIy927o= +github.com/ipfs/interface-go-ipfs-core v0.5.1 h1:1KMM7RkjUD8W5fSoRsa9xR6ZMzeL8fLHOUM1UEW9Y4M= +github.com/ipfs/interface-go-ipfs-core v0.5.1/go.mod h1:lNBJrdXHtWS46evMPBdWtDQMDsrKcGbxCOGoKLkztOE= github.com/ipfs/tar-utils v0.0.1 h1:8Na0KBD6GddGyXwU4rXNtVTE24iuZws8mENJQPLG7W4= github.com/ipfs/tar-utils v0.0.1/go.mod h1:ACflm9wXvV9w0eMJt6yYXxS2zuIV+yXGNwbuq1bhLeE= github.com/ipld/go-car v0.3.1 h1:WT+3cdmXlvmWOlGxk9webhj4auGO5QvgqC2vCCkFRXs= github.com/ipld/go-car v0.3.1/go.mod h1:dPkEWeAK8KaVvH5TahaCs6Mncpd4lDMpkbs0/SPzuVs= -github.com/ipld/go-codec-dagpb v1.2.0 h1:2umV7ud8HBMkRuJgd8gXw95cLhwmcYrihS3cQEy9zpI= github.com/ipld/go-codec-dagpb v1.2.0/go.mod h1:6nBN7X7h8EOsEejZGqC7tej5drsdBAXbMHyBT+Fne5s= +github.com/ipld/go-codec-dagpb v1.3.0 h1:czTcaoAuNNyIYWs6Qe01DJ+sEX7B+1Z0LcXjSatMGe8= +github.com/ipld/go-codec-dagpb v1.3.0/go.mod h1:ga4JTU3abYApDC3pZ00BC2RSvC3qfBb9MSJkMLSwnhA= github.com/ipld/go-ipld-prime v0.9.0/go.mod h1:KvBLMr4PX1gWptgkzRjVZCrLmSGcZCb/jioOQwCqZN8= -github.com/ipld/go-ipld-prime v0.9.1-0.20210324083106-dc342a9917db h1:kFwGn8rXa/Z31ev1OFNQsYeNKNCdifnTPl/NvPy5L38= github.com/ipld/go-ipld-prime v0.9.1-0.20210324083106-dc342a9917db/go.mod h1:KvBLMr4PX1gWptgkzRjVZCrLmSGcZCb/jioOQwCqZN8= +github.com/ipld/go-ipld-prime v0.11.0 h1:jD/b/22R7CSL+F9xNffcexs+wO0Ji/TfwXO/TWck+70= +github.com/ipld/go-ipld-prime v0.11.0/go.mod h1:+WIAkokurHmZ/KwzDOMUuoeJgaRQktHtEaLglS3ZeV8= github.com/jackpal/gateway v1.0.5/go.mod h1:lTpwd4ACLXmpyiCTRtfiNyVnUmqT9RivzCDQetPfnjA= github.com/jackpal/go-nat-pmp v1.0.1/go.mod h1:QPH045xvCAeXUZOxsnwmrtiCoxIr9eob+4orBN1SBKc= github.com/jackpal/go-nat-pmp v1.0.2 h1:KzKSgb7qkJvOUTqYl9/Hg/me3pWgBmERKrTGD7BdWus= @@ -565,8 +576,9 @@ github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+o github.com/kkdai/bstream v0.0.0-20161212061736-f391b8402d23/go.mod h1:J+Gs4SYgM6CZQHDETBtE9HaSEkGmuNXF86RwHhHUvq4= github.com/klauspost/compress v1.11.7 h1:0hzRabrMN4tSTvMfnL3SCv1ZGeAP23ynzodBgaHeMeg= github.com/klauspost/compress v1.11.7/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdYsUV+/s2qKfXs= -github.com/klauspost/cpuid/v2 v2.0.4 h1:g0I61F2K2DjRHz1cnxlkNSBIaePVoJIjjnHui8QHbiw= github.com/klauspost/cpuid/v2 v2.0.4/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= +github.com/klauspost/cpuid/v2 v2.0.6 h1:dQ5ueTiftKxp0gyjKSx5+8BtPWkyQbd95m8Gys/RarI= +github.com/klauspost/cpuid/v2 v2.0.6/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg= github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= github.com/konsorten/go-windows-terminal-sequences v1.0.3 h1:CE8S1cTafDpPvMhIxNJKvHsGVBgn1xWYf1NbHQhywc8= github.com/konsorten/go-windows-terminal-sequences v1.0.3/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ= @@ -595,8 +607,6 @@ github.com/libp2p/go-conn-security-multistream v0.0.2/go.mod h1:nc9vud7inQ+d6SO0 github.com/libp2p/go-conn-security-multistream v0.1.0/go.mod h1:aw6eD7LOsHEX7+2hJkDxw1MteijaVcI+/eP2/x3J1xc= github.com/libp2p/go-conn-security-multistream v0.2.0/go.mod h1:hZN4MjlNetKD3Rq5Jb/P5ohUnFLNzEAR4DLSzpn2QLU= github.com/libp2p/go-conn-security-multistream v0.2.1 h1:ft6/POSK7F+vl/2qzegnHDaXFU0iWB4yVTYrioC6Zy0= -github.com/libp2p/go-conn-security-multistream v0.2.1 h1:ft6/POSK7F+vl/2qzegnHDaXFU0iWB4yVTYrioC6Zy0= -github.com/libp2p/go-conn-security-multistream v0.2.1/go.mod h1:cR1d8gA0Hr59Fj6NhaTpFhJZrjSYuNmhpT2r25zYR70= github.com/libp2p/go-conn-security-multistream v0.2.1/go.mod h1:cR1d8gA0Hr59Fj6NhaTpFhJZrjSYuNmhpT2r25zYR70= github.com/libp2p/go-doh-resolver v0.3.1 h1:1wbVGkB4Tdj4WEvjAuYknOPyt4vSSDn9thnj13pKPaY= github.com/libp2p/go-doh-resolver v0.3.1/go.mod h1:y5go1ZppAq9N2eppbX0xON01CyPBeUg2yS6BTssssog= @@ -616,6 +626,7 @@ github.com/libp2p/go-libp2p v0.7.4/go.mod h1:oXsBlTLF1q7pxr+9w6lqzS1ILpyHsaBPniV github.com/libp2p/go-libp2p v0.8.1/go.mod h1:QRNH9pwdbEBpx5DTJYg+qxcVaDMAz3Ee/qDKwXujH5o= github.com/libp2p/go-libp2p v0.12.0/go.mod h1:FpHZrfC1q7nA8jitvdjKBDF31hguaC676g/nT9PgQM0= github.com/libp2p/go-libp2p v0.13.0/go.mod h1:pM0beYdACRfHO1WcJlp65WXyG2A6NqYM+t2DTVAJxMo= +github.com/libp2p/go-libp2p v0.14.0/go.mod h1:dsQrWLAoIn+GkHPN/U+yypizkHiB9tnv79Os+kSgQ4Q= github.com/libp2p/go-libp2p v0.14.4 h1:QCJE+jGyqxWdrSPuS4jByXCzosgaIg4SJTLCRplJ53w= github.com/libp2p/go-libp2p v0.14.4/go.mod h1:EIRU0Of4J5S8rkockZM7eJp2S0UrCyi55m2kJVru3rM= github.com/libp2p/go-libp2p-asn-util v0.0.0-20200825225859-85005c6cf052 h1:BM7aaOF7RpmNn9+9g6uTjGJ0cTzWr5j9i9IKeun2M8U= @@ -805,6 +816,7 @@ github.com/libp2p/go-libp2p-yamux v0.2.8/go.mod h1:/t6tDqeuZf0INZMTgd0WxIRbtK2Ez github.com/libp2p/go-libp2p-yamux v0.4.0/go.mod h1:+DWDjtFMzoAwYLVkNZftoucn7PelNoy5nm3tZ3/Zw30= github.com/libp2p/go-libp2p-yamux v0.5.0/go.mod h1:AyR8k5EzyM2QN9Bbdg6X1SkVVuqLwTGf0L4DFq9g6po= github.com/libp2p/go-libp2p-yamux v0.5.1/go.mod h1:dowuvDu8CRWmr0iqySMiSxK+W0iL5cMVO9S94Y6gkv4= +github.com/libp2p/go-libp2p-yamux v0.5.3/go.mod h1:Vy3TMonBAfTMXHWopsMc8iX/XGRYrRlpUaMzaeuHV/s= github.com/libp2p/go-libp2p-yamux v0.5.4 h1:/UOPtT/6DHPtr3TtKXBHa6g0Le0szYuI33Xc/Xpd7fQ= github.com/libp2p/go-libp2p-yamux v0.5.4/go.mod h1:tfrXbyaTqqSU654GTvK3ocnSZL3BuHoeTSqhcel1wsE= github.com/libp2p/go-maddr-filter v0.0.1/go.mod h1:6eT12kSQMA9x2pvFQa+xesMKUBlj9VImZbj3B9FBH/Q= @@ -886,6 +898,7 @@ github.com/libp2p/go-yamux v1.4.0/go.mod h1:fr7aVgmdNGJK+N1g+b6DW6VxzbRCjCOejR/h github.com/libp2p/go-yamux v1.4.1 h1:P1Fe9vF4th5JOxxgQvfbOHkrGqIZniTLf+ddhZp8YTI= github.com/libp2p/go-yamux v1.4.1/go.mod h1:fr7aVgmdNGJK+N1g+b6DW6VxzbRCjCOejR/hkmpooHE= github.com/libp2p/go-yamux/v2 v2.0.0/go.mod h1:NVWira5+sVUIU6tu1JWvaRn1dRnG+cawOJiflsAM+7U= +github.com/libp2p/go-yamux/v2 v2.1.1/go.mod h1:3So6P6TV6r75R9jiBpiIKgU/66lOarCZjqROGxzPpPQ= github.com/libp2p/go-yamux/v2 v2.2.0 h1:RwtpYZ2/wVviZ5+3pjC8qdQ4TKnrak0/E01N1UWoAFU= github.com/libp2p/go-yamux/v2 v2.2.0/go.mod h1:3So6P6TV6r75R9jiBpiIKgU/66lOarCZjqROGxzPpPQ= github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20190605223551-bc2310a04743/go.mod h1:qklhhLq1aX+mtWk9cPHPzaBjWImj5ULL6C7HFJtXQMM= @@ -991,8 +1004,6 @@ github.com/multiformats/go-multiaddr-dns v0.0.2/go.mod h1:9kWcqw/Pj6FwxAwW38n/94 github.com/multiformats/go-multiaddr-dns v0.2.0/go.mod h1:TJ5pr5bBO7Y1B18djPuRsVkduhQH2YqYSbxWJzYGdK0= github.com/multiformats/go-multiaddr-dns v0.3.0/go.mod h1:mNzQ4eTGDg0ll1N9jKPOUogZPoJ30W8a7zk66FQPpdQ= github.com/multiformats/go-multiaddr-dns v0.3.1 h1:QgQgR+LQVt3NPTjbrLLpsaT2ufAA2y0Mkk+QRVJbW3A= -github.com/multiformats/go-multiaddr-dns v0.3.1 h1:QgQgR+LQVt3NPTjbrLLpsaT2ufAA2y0Mkk+QRVJbW3A= -github.com/multiformats/go-multiaddr-dns v0.3.1/go.mod h1:G/245BRQ6FJGmryJCrOuTdB37AMA5AMOVuO6NY3JwTk= github.com/multiformats/go-multiaddr-dns v0.3.1/go.mod h1:G/245BRQ6FJGmryJCrOuTdB37AMA5AMOVuO6NY3JwTk= github.com/multiformats/go-multiaddr-fmt v0.0.1/go.mod h1:aBYjqL4T/7j4Qx+R73XSv/8JsgnRFlf0w2KGLCmXl3Q= github.com/multiformats/go-multiaddr-fmt v0.1.0 h1:WLEFClPycPkp4fnIzoFoV9FVd49/eQsuaL3/CWe167E= @@ -1009,8 +1020,9 @@ github.com/multiformats/go-multiaddr-net v0.2.0/go.mod h1:gGdH3UXny6U3cKKYCvpXI5 github.com/multiformats/go-multibase v0.0.1/go.mod h1:bja2MqRZ3ggyXtZSEDKpl0uO/gviWFaSteVbWT51qgs= github.com/multiformats/go-multibase v0.0.3 h1:l/B6bJDQjvQ5G52jw4QGSYeOTZoAwIO77RblWplfIqk= github.com/multiformats/go-multibase v0.0.3/go.mod h1:5+1R4eQrT3PkYZ24C3W2Ue2tPwIdYQD509ZjSb5y9Oc= -github.com/multiformats/go-multicodec v0.2.0 h1:MUzKZWxOFagwLLtlx96pub9zwDQAbMAf1k9fXOdc3so= github.com/multiformats/go-multicodec v0.2.0/go.mod h1:/y4YVwkfMyry5kFbMTbLJKErhycTIftytRV+llXdyS4= +github.com/multiformats/go-multicodec v0.3.0 h1:tstDwfIjiHbnIjeM5Lp+pMrSeN+LCMsEwOrkPmWm03A= +github.com/multiformats/go-multicodec v0.3.0/go.mod h1:qGGaQmioCDh+TeFOnxrbU0DaIPw8yFgAZgFG0V7p1qQ= github.com/multiformats/go-multihash v0.0.1/go.mod h1:w/5tugSrLEbWqlcgJabL3oHFKTwfvkofsjW2Qa1ct4U= github.com/multiformats/go-multihash v0.0.5/go.mod h1:lt/HCbqlQwlPBz7lv0sQCdtfcMtlJvakRUn/0Ual8po= github.com/multiformats/go-multihash v0.0.8/go.mod h1:YSLudS+Pi8NHE7o6tb3D8vrpKa63epEDmG8nTduyAew= @@ -1026,7 +1038,6 @@ github.com/multiformats/go-multistream v0.1.0/go.mod h1:fJTiDfXJVmItycydCnNx4+wS github.com/multiformats/go-multistream v0.1.1/go.mod h1:KmHZ40hzVxiaiwlj3MEbYgK9JFk2/9UktWZAF54Du38= github.com/multiformats/go-multistream v0.2.0/go.mod h1:5GZPQZbkWOLOn3J2y4Y99vVW7vOfsAflxARk3x14o6k= github.com/multiformats/go-multistream v0.2.1/go.mod h1:5GZPQZbkWOLOn3J2y4Y99vVW7vOfsAflxARk3x14o6k= -github.com/multiformats/go-multistream v0.2.1/go.mod h1:5GZPQZbkWOLOn3J2y4Y99vVW7vOfsAflxARk3x14o6k= github.com/multiformats/go-multistream v0.2.2 h1:TCYu1BHTDr1F/Qm75qwYISQdzGcRdC21nFgQW7l7GBo= github.com/multiformats/go-multistream v0.2.2/go.mod h1:UIcnm7Zuo8HKG+HkWgfQsGL+/MIEhyTqbODbIUwSXKs= github.com/multiformats/go-varint v0.0.1/go.mod h1:3Ls8CIEsrijN6+B7PbrXRPxHRPuXSrVKRY101jdMZYE= @@ -1299,8 +1310,9 @@ go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/ go.uber.org/multierr v1.3.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4= go.uber.org/multierr v1.4.0/go.mod h1:VgVr7evmIr6uPjLBxg28wmKNXyqE9akIJ5XnfpiKl+4= go.uber.org/multierr v1.5.0/go.mod h1:FeouvMocqHpRaaGuG9EjoKcStLC43Zu/fmqdUMPcKYU= -go.uber.org/multierr v1.6.0 h1:y6IPFStTAIT5Ytl7/XYmHvzXQ7S3g/IeZW9hyZ5thw4= go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU= +go.uber.org/multierr v1.7.0 h1:zaiO/rmgFjbmCXdSYJWQcdvOCsthmdaHfr3Gm2Kx4Ec= +go.uber.org/multierr v1.7.0/go.mod h1:7EAYxJLBy9rStEaz58O2t4Uvip6FSURkq8/ppBp95ak= go.uber.org/tools v0.0.0-20190618225709-2cfd321de3ee/go.mod h1:vJERXedbb3MVM5f9Ejo0C68/HhF8uaILCdgjnY+goOA= go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q= go.uber.org/zap v1.13.0/go.mod h1:zwrFLgMcdUuIBviXEYEH1YKNaOBnKXsx2IPda5bBwHM= @@ -1340,10 +1352,10 @@ golang.org/x/crypto v0.0.0-20200602180216-279210d13fed/go.mod h1:LzIPMQfyMNhhGPh golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20200820211705-5c72a883971a/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto= golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I= -golang.org/x/crypto v0.0.0-20210220033148-5ea612d1eb83/go.mod h1:jdWPYTVW3xRLrWPugEBEK3UY2ZEsg3UU495nc5E+M+I= golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4= -golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf h1:B2n+Zi5QeYRDAEodEu72OS36gmTWjgpXr2+cWcBW90o= golang.org/x/crypto v0.0.0-20210506145944-38f3c27a63bf/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= +golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a h1:kr2P4QFmQr29mSLA43kwrOcgcReGTfbE9N577tCTuBc= +golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a/go.mod h1:P+XmwS30IXTQdn5tA2iutPOUgjI07+tq3H3K9MVA1s8= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= @@ -1499,7 +1511,6 @@ golang.org/x/sys v0.0.0-20201214210602-f9fddec55a1e/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210303074136-134d130e1a04/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210309074719-68d13333faf2/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210315160823-c6e025ad8005/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210317225723-c4fcb01b228e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= @@ -1507,8 +1518,9 @@ golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210426080607-c94f62235c83/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20210511113859-b0526f3d8744 h1:yhBbb4IRs2HS9PPlAg6DMC6mUOKexJBNsLf4Z+6En1Q= golang.org/x/sys v0.0.0-20210511113859-b0526f3d8744/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210514084401-e8d321eab015 h1:hZR0X1kPW+nwyJ9xRxqZk1vx5RUObAPBdKVvXPDUH/E= +golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1 h1:v+OssWQX+hTHEmOBgwxdZxK4zHq3yOs8F9J7mk0PY8E= golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= @@ -1641,8 +1653,9 @@ google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpAD google.golang.org/protobuf v1.24.0/go.mod h1:r/3tXBNzIEhYS9I1OUVjXDlt8tc493IdKGjtUeSXeh4= google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c= google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw= -google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk= google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= +google.golang.org/protobuf v1.27.1 h1:SnqbnDw1V7RiZcXPx5MEeqPv2s79L9i7BJUlG/+RurQ= +google.golang.org/protobuf v1.27.1/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc= gopkg.in/alecthomas/kingpin.v2 v2.2.6 h1:jMFz6MfLP0/4fUyZle81rXUoxOBFi19VUFKVDOQfozc= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/plugin/ipld.go b/plugin/ipld.go index 5b45e9cd333..2f783a61238 100644 --- a/plugin/ipld.go +++ b/plugin/ipld.go @@ -1,16 +1,13 @@ package plugin import ( - "github.com/ipfs/go-ipfs/core/coredag" - - ipld "github.com/ipfs/go-ipld-format" + multicodec "github.com/ipld/go-ipld-prime/multicodec" ) // PluginIPLD is an interface that can be implemented to add handlers for -// for different IPLD formats +// for different IPLD codecs type PluginIPLD interface { Plugin - RegisterBlockDecoders(dec ipld.BlockDecoder) error - RegisterInputEncParsers(iec coredag.InputEncParsers) error + Register(multicodec.Registry) error } diff --git a/plugin/loader/loader.go b/plugin/loader/loader.go index 14b7c18f513..167e1b3090b 100644 --- a/plugin/loader/loader.go +++ b/plugin/loader/loader.go @@ -10,14 +10,13 @@ import ( config "github.com/ipfs/go-ipfs-config" cserialize "github.com/ipfs/go-ipfs-config/serialize" + "github.com/ipld/go-ipld-prime/multicodec" "github.com/ipfs/go-ipfs/core" "github.com/ipfs/go-ipfs/core/coreapi" - coredag "github.com/ipfs/go-ipfs/core/coredag" plugin "github.com/ipfs/go-ipfs/plugin" fsrepo "github.com/ipfs/go-ipfs/repo/fsrepo" - ipld "github.com/ipfs/go-ipld-format" logging "github.com/ipfs/go-log" opentracing "github.com/opentracing/opentracing-go" ) @@ -335,11 +334,7 @@ func injectDatastorePlugin(pl plugin.PluginDatastore) error { } func injectIPLDPlugin(pl plugin.PluginIPLD) error { - err := pl.RegisterBlockDecoders(ipld.DefaultBlockDecoder) - if err != nil { - return err - } - return pl.RegisterInputEncParsers(coredag.DefaultInputEncParsers) + return pl.Register(multicodec.DefaultRegistry) } func injectTracerPlugin(pl plugin.PluginTracer) error { diff --git a/plugin/plugins/git/git.go b/plugin/plugins/git/git.go index 23b79ad59a1..bc28084ffe5 100644 --- a/plugin/plugins/git/git.go +++ b/plugin/plugins/git/git.go @@ -2,17 +2,15 @@ package git import ( "compress/zlib" - "fmt" "io" - "math" - "github.com/ipfs/go-ipfs/core/coredag" "github.com/ipfs/go-ipfs/plugin" - "github.com/ipfs/go-cid" - "github.com/ipfs/go-ipld-format" + // Note that depending on this package registers it's multicodec encoder and decoder. git "github.com/ipfs/go-ipld-git" - mh "github.com/multiformats/go-multihash" + "github.com/ipld/go-ipld-prime" + "github.com/ipld/go-ipld-prime/multicodec" + mc "github.com/multiformats/go-multicodec" ) // Plugins is exported list of plugins that will be loaded @@ -36,40 +34,21 @@ func (*gitPlugin) Init(_ *plugin.Environment) error { return nil } -func (*gitPlugin) RegisterBlockDecoders(dec format.BlockDecoder) error { - dec.Register(cid.GitRaw, git.DecodeBlock) +func (*gitPlugin) Register(reg multicodec.Registry) error { + // register a custom identifier in the reserved range for import of "zlib-encoded git objects." + reg.RegisterDecoder(uint64(mc.ReservedStart+mc.GitRaw), decodeZlibGit) + reg.RegisterEncoder(uint64(mc.GitRaw), git.Encode) + reg.RegisterDecoder(uint64(mc.GitRaw), git.Decode) return nil } -func (*gitPlugin) RegisterInputEncParsers(iec coredag.InputEncParsers) error { - iec.AddParser("raw", "git", parseRawGit) - iec.AddParser("zlib", "git", parseZlibGit) - return nil -} - -func parseRawGit(r io.Reader, mhType uint64, mhLen int) ([]format.Node, error) { - if mhType != math.MaxUint64 && mhType != mh.SHA1 { - return nil, fmt.Errorf("unsupported mhType %d", mhType) - } - - if mhLen != -1 && mhLen != mh.DefaultLengths[mh.SHA1] { - return nil, fmt.Errorf("invalid mhLen %d", mhLen) - } - - nd, err := git.ParseObject(r) - if err != nil { - return nil, err - } - - return []format.Node{nd}, nil -} - -func parseZlibGit(r io.Reader, mhType uint64, mhLen int) ([]format.Node, error) { +func decodeZlibGit(na ipld.NodeAssembler, r io.Reader) error { rc, err := zlib.NewReader(r) if err != nil { - return nil, err + return err } defer rc.Close() - return parseRawGit(rc, mhType, mhLen) + + return git.Decode(na, rc) } diff --git a/test/sharness/t0053-dag.sh b/test/sharness/t0053-dag.sh index 2225f79ec4d..b8f8196c137 100755 --- a/test/sharness/t0053-dag.sh +++ b/test/sharness/t0053-dag.sh @@ -21,45 +21,158 @@ test_expect_success "make a few test files" ' HASH4=$(ipfs add --pin=false -q file4) ' -test_expect_success "make an ipld object in json" ' +test_expect_success "make an ipld object in dag-json" ' printf "{\"hello\":\"world\",\"cats\":[{\"/\":\"%s\"},{\"water\":{\"/\":\"%s\"}}],\"magic\":{\"/\":\"%s\"},\"sub\":{\"dict\":\"ionary\",\"beep\":[0,\"bop\"]}}" $HASH1 $HASH2 $HASH3 > ipld_object ' +# This data is in https://github.com/ipld/codec-fixtures/tree/master/fixtures/dagpb_Data_some +test_expect_success "make the same ipld object in dag-cbor, dag-json and dag-pb" ' + echo "omREYXRhRQABAgMEZUxpbmtzgA==" | base64 -d > ipld_object_dagcbor + echo "CgUAAQIDBA==" | base64 -d > ipld_object_dagpb + echo "{\"Data\":{\"/\":{\"bytes\":\"AAECAwQ\"}},\"Links\":[]}" > ipld_object_dagjson +' + test_dag_cmd() { - test_expect_success "can add an ipld object using protobuf" ' - IPLDHASH=$(cat ipld_object | ipfs dag put -f protobuf) + # Working with a plain IPLD hello-world object that's dag-json and dag-cbor compatible + + test_expect_success "can add an ipld object using defaults (dag-json to dag-cbor)" ' + IPLDHASH=$(cat ipld_object | ipfs dag put) ' - test_expect_success "output looks correct" ' - EXPHASH="QmdfTbBqBPQ7VNxZEYEj14VmRuZBkqFbiwReogJgS1zR1n" + test_expect_success "CID looks correct" ' + EXPHASH="bafyreiblwimnjbqcdoeafiobk6q27jcw64ew7n2fmmhdpldd63edmjecde" test $EXPHASH = $IPLDHASH ' - test_expect_success "can add an ipld object using protobuf and --cid=base=base32" ' - IPLDHASHb32=$(cat ipld_object | ipfs dag put -f protobuf --cid-base=base32) + test_expect_success "can add an ipld object using dag-json to dag-json" ' + IPLDHASH=$(cat ipld_object | ipfs dag put --input-enc=dag-json -f dag-json) ' - test_expect_success "output looks correct (does not upgrade to CIDv1)" ' - test $EXPHASH = $IPLDHASHb32 + test_expect_success "CID looks correct" ' + EXPHASH="baguqeera6gviseelmbzn2ugoddo5vulxlshqs3kw5ymgsb6w4cabnoh4ldpa" + test $EXPHASH = $IPLDHASH ' - test_expect_success "can add an ipld object" ' - IPLDHASH=$(cat ipld_object | ipfs dag put) + test_expect_success "can add an ipld object using dag-json to dag-cbor" ' + IPLDHASH=$(cat ipld_object | ipfs dag put --input-enc=dag-json -f dag-cbor) ' - test_expect_success "output looks correct" ' - EXPHASH="bafyreidjtjfmavdk7epvztob2m5vlm3pxp3gjmpyewro4qlbw5n4f4iz64" + test_expect_success "CID looks correct" ' + EXPHASH="bafyreiblwimnjbqcdoeafiobk6q27jcw64ew7n2fmmhdpldd63edmjecde" test $EXPHASH = $IPLDHASH ' - test_expect_success "can add an ipld object using --cid-base=base32" ' - IPLDHASHb32=$(cat ipld_object | ipfs dag put --cid-base=base32) + test_expect_success "can add an ipld object using cid-base=base58btc" ' + IPLDb58HASH=$(cat ipld_object | ipfs dag put -cid-base=base58btc) ' - test_expect_success "output looks correct" ' - test $(ipfs cid base32 $EXPHASH) = $IPLDHASHb32 + test_expect_success "CID looks correct" ' + EXPHASH="zdpuAoN1XJ3GsrxEzMuCbRKZzRUVJekJUCbPVgCgE4D9yYqVi" + test $EXPHASH = $IPLDb58HASH + ' + + # Same object, different forms + # (1) dag-cbor input + + test_expect_success "can add a dag-cbor input block stored as dag-cbor" ' + IPLDCBORHASH=$(cat ipld_object_dagcbor | ipfs dag put --input-enc=dag-cbor -f dag-cbor) + ' + + test_expect_success "dag-cbor CID looks correct" ' + EXPHASH="bafyreieculsmrexh3ty5jentbvuku452o27mst4h2tq2rb2zntqhgcstji" + test $EXPHASH = $IPLDCBORHASH + ' + + test_expect_success "can add a dag-cbor input block stored as dag-pb" ' + IPLDPBHASH=$(cat ipld_object_dagcbor | ipfs dag put --input-enc=dag-cbor -f dag-pb) + ' + + test_expect_success "dag-pb CID looks correct" ' + EXPHASH="bafybeibazl2z4vqp2tmwcfag6wirmtpnomxknqcgrauj7m2yisrz3qjbom" + test $EXPHASH = $IPLDPBHASH + ' + + test_expect_success "can add a dag-cbor input block stored as dag-json" ' + IPLDJSONHASH=$(cat ipld_object_dagcbor | ipfs dag put --input-enc=dag-cbor -f dag-json) + ' + + test_expect_success "dag-json CID looks correct" ' + EXPHASH="baguqeerajwksxu3lxpomdwxvosl542zl3xknhjgxtq3277gafrhl6vdw5tcq" + test $EXPHASH = $IPLDJSONHASH + ' + + # (2) dag-json input + + test_expect_success "can add a dag-json input block stored as dag-cbor" ' + IPLDCBORHASH=$(cat ipld_object_dagjson | ipfs dag put --input-enc=dag-json -f dag-cbor) + ' + + test_expect_success "dag-cbor CID looks correct" ' + EXPHASH="bafyreieculsmrexh3ty5jentbvuku452o27mst4h2tq2rb2zntqhgcstji" + test $EXPHASH = $IPLDCBORHASH + ' + + test_expect_success "can add a dag-json input block stored as dag-pb" ' + IPLDPBHASH=$(cat ipld_object_dagjson | ipfs dag put --input-enc=dag-json -f dag-pb) + ' + + test_expect_success "dag-pb CID looks correct" ' + EXPHASH="bafybeibazl2z4vqp2tmwcfag6wirmtpnomxknqcgrauj7m2yisrz3qjbom" + test $EXPHASH = $IPLDPBHASH + ' + + test_expect_success "can add a dag-json input block stored as dag-json" ' + IPLDJSONHASH=$(cat ipld_object_dagjson | ipfs dag put --input-enc=dag-json -f dag-json) ' + test_expect_success "dag-json CID looks correct" ' + EXPHASH="baguqeerajwksxu3lxpomdwxvosl542zl3xknhjgxtq3277gafrhl6vdw5tcq" + test $EXPHASH = $IPLDJSONHASH + ' + + # (3) dag-pb input + + test_expect_success "can add a dag-pb input block stored as dag-cbor" ' + IPLDCBORHASH=$(cat ipld_object_dagpb | ipfs dag put --input-enc=dag-pb -f dag-cbor) + ' + + test_expect_success "dag-cbor CID looks correct" ' + EXPHASH="bafyreieculsmrexh3ty5jentbvuku452o27mst4h2tq2rb2zntqhgcstji" + test $EXPHASH = $IPLDCBORHASH + ' + + test_expect_success "can add a dag-pb input block stored as dag-pb" ' + IPLDPBHASH=$(cat ipld_object_dagpb | ipfs dag put --input-enc=dag-pb -f dag-pb) + ' + + test_expect_success "dag-pb CID looks correct" ' + EXPHASH="bafybeibazl2z4vqp2tmwcfag6wirmtpnomxknqcgrauj7m2yisrz3qjbom" + test $EXPHASH = $IPLDPBHASH + ' + + test_expect_success "can add a dag-pb input block stored as dag-json" ' + IPLDJSONHASH=$(cat ipld_object_dagpb | ipfs dag put --input-enc=dag-pb -f dag-json) + ' + + test_expect_success "dag-json CID looks correct" ' + EXPHASH="baguqeerajwksxu3lxpomdwxvosl542zl3xknhjgxtq3277gafrhl6vdw5tcq" + test $EXPHASH = $IPLDJSONHASH + ' + + test_expect_success "can get dag-cbor, dag-json, dag-pb blocks as dag-json" ' + ipfs dag get $IPLDCBORHASH >& dag-get-cbor && + ipfs dag get $IPLDJSONHASH >& dag-get-json && + ipfs dag get $IPLDPBHASH >& dag-get-pb + ' + + # this doesn't tell us if they are correct, we test that better below + test_expect_success "outputs are the same" ' + test_cmp dag-get-cbor dag-get-json && + test_cmp dag-get-cbor dag-get-pb + ' + + # Traversals using the original hello-world object + test_expect_success "various path traversals work" ' ipfs cat $IPLDHASH/cats/0 > out1 && ipfs cat $IPLDHASH/cats/1/water > out2 && @@ -81,25 +194,46 @@ test_dag_cmd() { ' test_expect_success "sub-objects look right" ' - echo "\"world\"" > sub1_exp && + echo -n "\"world\"" > sub1_exp && test_cmp sub1_exp sub1 && - echo "{\"beep\":[0,\"bop\"],\"dict\":\"ionary\"}" > sub2_exp && + echo -n "{\"beep\":[0,\"bop\"],\"dict\":\"ionary\"}" > sub2_exp && test_cmp sub2_exp sub2 && - echo "[0,\"bop\"]" > sub3_exp && + echo -n "[0,\"bop\"]" > sub3_exp && test_cmp sub3_exp sub3 && - echo "0" > sub4_exp && + echo -n "0" > sub4_exp && test_cmp sub4_exp sub4 && - echo "\"bop\"" > sub5_exp && + echo -n "\"bop\"" > sub5_exp && test_cmp sub5_exp sub5 ' - test_expect_success "can pin cbor object" ' - ipfs pin add $EXPHASH + test_expect_success "traversals using /ipld/ work" ' + ipfs dag get /ipld/$IPLDPBHASH/Data > ipld_path_Data_actual + ' + + test_expect_success "retrieved node looks right" ' + echo -n "{\"/\":{\"bytes\":\"AAECAwQ\"}}" > ipld_path_Data_expected + test_cmp ipld_path_Data_actual ipld_path_Data_expected + ' + + test_expect_success "can pin ipld object" ' + ipfs pin add $IPLDHASH + ' + + test_expect_success "can pin dag-pb object" ' + ipfs pin add $IPLDPBHASH + ' + + test_expect_success "can pin dag-cbor object" ' + ipfs pin add $IPLDCBORHASH + ' + + test_expect_success "can pin dag-json object" ' + ipfs pin add $IPLDJSONHASH ' test_expect_success "after gc, objects still accessible" ' ipfs repo gc > /dev/null && - ipfs refs -r --timeout=2s $EXPHASH > /dev/null + ipfs refs -r --timeout=2s $IPLDJSONHASH > /dev/null ' test_expect_success "can get object" ' @@ -111,7 +245,7 @@ test_dag_cmd() { ' test_expect_success "retrieved object hashes back correctly" ' - IPLDHASH2=$(cat ipld_obj_out | ipfs dag put) && + IPLDHASH2=$(cat ipld_obj_out | ipfs dag put --input-enc=dag-json -f dag-cbor) && test "$IPLDHASH" = "$IPLDHASH2" ' @@ -124,7 +258,7 @@ test_dag_cmd() { ' test_expect_success "output looks correct" ' - echo "{\"data\":\"CAISB2Zvb2JhcgoYBw==\",\"links\":[]}" > dag_get_pb_exp && + echo -n "{\"Data\":{\"/\":{\"bytes\":\"CAISB2Zvb2JhcgoYBw\"}},\"Links\":[]}" > dag_get_pb_exp && test_cmp dag_get_pb_exp dag_get_pb_out ' @@ -133,24 +267,23 @@ test_dag_cmd() { ' test_expect_success "output looks correct" ' - echo "{\"data\":\"CAISBGZvbwoYBA==\",\"links\":[]}" > cat_exp && + echo -n "{\"Data\":{\"/\":{\"bytes\":\"CAISBGZvbwoYBA\"}},\"Links\":[]}" > cat_exp && test_cmp cat_exp cat_out ' - test_expect_success "non-canonical cbor input is normalized" ' - HASH=$(cat ../t0053-dag-data/non-canon.cbor | ipfs dag put --format=cbor --input-enc=raw) && + test_expect_success "non-canonical dag-cbor input is normalized" ' + HASH=$(cat ../t0053-dag-data/non-canon.cbor | ipfs dag put --format=dag-cbor --input-enc=dag-cbor) && test $HASH = "bafyreiawx7ona7oa2ptcoh6vwq4q6bmd7x2ibtkykld327bgb7t73ayrqm" || test_fsh echo $HASH ' - test_expect_success "non-canonical cbor input is normalized with input-enc cbor" ' - HASH=$(cat ../t0053-dag-data/non-canon.cbor | ipfs dag put --format=cbor --input-enc=cbor) && - test $HASH = "bafyreiawx7ona7oa2ptcoh6vwq4q6bmd7x2ibtkykld327bgb7t73ayrqm" || - test_fsh echo $HASH + test_expect_success "cbor input can be fetched" ' + EXPARR=$(ipfs dag get $HASH/arr) + test $EXPARR = "[]" ' test_expect_success "add an ipld with pin" ' - PINHASH=$(printf {\"foo\":\"bar\"} | ipfs dag put --pin=true) + PINHASH=$(printf {\"foo\":\"bar\"} | ipfs dag put --input-enc=dag-json --pin=true) ' test_expect_success "after gc, objects still accessible" ' @@ -158,23 +291,23 @@ test_dag_cmd() { ipfs refs -r --timeout=2s $PINHASH > /dev/null ' - test_expect_success "can add an ipld object with sha3 hash" ' - IPLDHASH=$(cat ipld_object | ipfs dag put --hash sha3) + test_expect_success "can add an ipld object with sha3-512 hash" ' + IPLDHASH=$(cat ipld_object | ipfs dag put --hash sha3-512) ' test_expect_success "output looks correct" ' - EXPHASH="bafyriqgae54zjl3bjebmbat2rjem4ewj6vni6jxohandmvk3bibfgv3sioyeidppsghvulryxats43br3b7afa6jy77x6gqzqaicer6ljicck" + EXPHASH="bafyriqforjma7y7akqz7nhuu73r6liggj5zhkbfiqgicywe3fgkna2ijlhod2af3ue7doj56tlzt5hh6iu5esafc4msr3sd53jol5m2o25ucy" test $EXPHASH = $IPLDHASH ' test_expect_success "prepare dag-pb object" ' echo foo > test_file && - HASH=$(ipfs add -wq test_file | tail -n1) + HASH=$(ipfs add -wq test_file | tail -n1 | ipfs cid base32) ' test_expect_success "dag put with json dag-pb works" ' ipfs dag get $HASH > pbjson && - cat pbjson | ipfs dag put --format=dag-pb --input-enc=json > dag_put_out + cat pbjson | ipfs dag put --format=dag-pb --input-enc=dag-json > dag_put_out ' test_expect_success "dag put with dag-pb works output looks good" ' @@ -184,7 +317,7 @@ test_dag_cmd() { test_expect_success "dag put with raw dag-pb works" ' ipfs block get $HASH > pbraw && - cat pbraw | ipfs dag put --format=dag-pb --input-enc=raw > dag_put_out + cat pbraw | ipfs dag put --format=dag-pb --input-enc=dag-pb > dag_put_out ' test_expect_success "dag put with dag-pb works output looks good" ' @@ -271,10 +404,10 @@ test_dag_cmd() { test_expect_success "dag stat of simple IPLD object" ' ipfs dag stat $NESTED_HASH > actual_stat_inner_ipld_obj && - echo "Size: 15, NumBlocks: 1" > exp_stat_inner_ipld_obj && + echo "Size: 8, NumBlocks: 1" > exp_stat_inner_ipld_obj && test_cmp exp_stat_inner_ipld_obj actual_stat_inner_ipld_obj && ipfs dag stat $HASH > actual_stat_ipld_obj && - echo "Size: 61, NumBlocks: 2" > exp_stat_ipld_obj && + echo "Size: 54, NumBlocks: 2" > exp_stat_ipld_obj && test_cmp exp_stat_ipld_obj actual_stat_ipld_obj ' diff --git a/test/sharness/t0055-dag-put-json-new-line.sh b/test/sharness/t0055-dag-put-json-new-line.sh index 92ae648ddae..d69806760cd 100755 --- a/test/sharness/t0055-dag-put-json-new-line.sh +++ b/test/sharness/t0055-dag-put-json-new-line.sh @@ -8,14 +8,14 @@ test_init_ipfs test_expect_success 'create test JSON files' ' WANT_JSON="{\"data\":1234}" - WANT_HASH="bafyreidcqah6v3py3ujdc3ris22rjlfntaw7ajrus2f2477kpnizulaoea" + WANT_HASH="bafyreidbm2zncsc3j25zn7lofgd4woeh6eygdy73thfosuni2rwr3bhcvu" printf "${WANT_JSON}\n" > with_newline.json && printf "${WANT_JSON}" > without_newline.json ' test_expect_success 'puts as CBOR work' ' - GOT_HASH_WITHOUT_NEWLINE="$(cat without_newline.json | ipfs dag put -f cbor)" - GOT_HASH_WITH_NEWLINE="$(cat with_newline.json | ipfs dag put -f cbor)" + GOT_HASH_WITHOUT_NEWLINE="$(cat without_newline.json | ipfs dag put -f dag-cbor)" + GOT_HASH_WITH_NEWLINE="$(cat with_newline.json | ipfs dag put -f dag-cbor)" ' test_expect_success 'put hashes with or without newline are equal' ' @@ -27,12 +27,7 @@ test_expect_success 'hashes are of expected value' ' test "${WANT_HASH}" = "${GOT_HASH_WITHOUT_NEWLINE}" ' -# Retrieval must not contain a new-line regardless of input JSON, because -# objects are put using the stable CBOR format. -# despite this, dag retrieval returns JSON with new-line. -# Expect failure until fixed, as per: -# - https://github.com/ipfs/go-ipfs/issues/3503#issuecomment-877295280 -test_expect_failure "retrieval by hash does not have new line" ' +test_expect_success "retrieval by hash does not have new line" ' ipfs dag get "${WANT_HASH}" > got.json test_cmp without_newline.json got.json ' diff --git a/test/sharness/t0280-plugin-git.sh b/test/sharness/t0280-plugin-git.sh index e7bc0642742..4e035254924 100755 --- a/test/sharness/t0280-plugin-git.sh +++ b/test/sharness/t0280-plugin-git.sh @@ -17,25 +17,40 @@ test_expect_success "prepare test data" ' test_dag_git() { test_expect_success "add objects via dag put" ' - find objects -type f -exec ipfs dag put --format=git --input-enc=zlib {} \; -exec echo \; > hashes + find objects -type f -exec ipfs dag put --format=git-raw --input-enc=0x300078 --hash=sha1 {} \; -exec echo -n \; > hashes ' test_expect_success "successfully get added objects" ' cat hashes | xargs -I {} ipfs dag get -- {} > /dev/null ' + test_expect_success "dag get works" ' + echo -n "{\"message\":\"Some version\n\",\"object\":{\"/\":\"baf4bcfeq6c2mspupcvftgevza56h7rmozose6wi\"},\"tag\":\"v1\",\"tagger\":{\"date\":\"1497302532\",\"email\":\"johndoe@example.com\",\"name\":\"John Doe\",\"timezone\":\"+0200\"},\"type\":\"commit\"}" > tag_expected && + ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi > tag_actual + ' + + test_expect_success "outputs look correct" ' + test_cmp tag_expected tag_actual + ' + test_expect_success "path traversals work" ' - echo \"YmxvYiA3ACcsLnB5Zgo=\" > file1 && - ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi/object/parents/0/tree/dir2/hash/f3/hash > out1 + echo -n "{\"date\":\"1497302532\",\"email\":\"johndoe@example.com\",\"name\":\"John Doe\",\"timezone\":\"+0200\"}" > author_expected && + echo -n "{\"/\":{\"bytes\":\"YmxvYiAxMgBIZWxsbyB3b3JsZAo\"}}" > file1_expected && + echo -n "{\"/\":{\"bytes\":\"YmxvYiA3ACcsLnB5Zgo\"}}" > file2_expected && + ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi/object/author > author_actual && + ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi/object/tree/file/hash > file1_actual && + ipfs dag get baf4bcfhzi72pcj5cc4ocz7igcduubuu7aa3cddi/object/parents/0/tree/dir2/hash/f3/hash > file2_actual ' test_expect_success "outputs look correct" ' - test_cmp file1 out1 + test_cmp author_expected author_actual && + test_cmp file1_expected file1_actual && + test_cmp file2_expected file2_actual ' } # should work offline -#test_dag_git +test_dag_git # should work online test_launch_ipfs_daemon