refactor: BlockPutSettings.CidPrefix #80

schomatis · 2021-11-18T14:40:35Z

Cleanup related to ipfs/kubo#8471, required by ipfs/kubo#8568

TLDR

CIDv1 returned by default

Part of https://github.com/ipfs/ipfs/issues/337.
The raw codec is used by default, unless custom one is set in BlockPutOptions via CidCodec func:

result, err := api.Block().Put(req.Context, file, options.Block.CidCodec(cidCodec))

CID prefix is now part of BlockPutSettings

caopts "github.com/ipfs/interface-go-ipfs-core/options"

- settings, pref, err := caopts.BlockPutOptions(opts...)
+ settings, err := caopts.BlockPutOptions(opts...)
...
-	calculatedCid, err := pref.Sum(data)
+	calculatedCid, err := settings.CidPrefix.Sum(data)

BREAKING CHANGES

implicit BlockPutSettings now produces CIDv1 with raw codec by default now
- CidCodec makes block put return CID with alternative codec
  - this impacts only the returned CID, it does not trigger any validation or data transformation
  - codec names are validated against tables from https://github.com/multiformats/go-multicodec
- Format is deprecated. It used incorrect codec names and should be avoided for new deployments. Use it only if you need the old, invalid behavior, namely things described in fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ kubo#8568:
  - ipfs block put --format=v0 will produce CIDv0 (implicit dag-pb)
  - ipfs block put --format=cbor will produce CIDv1 with dag-cbor (!)
  - ipfs block put --format=protobuf will produce CIDv1 with dag-pb (!)

Closes #51 (we default to raw and not dag-pb anymore)

lidel · 2022-03-30T19:23:46Z

options/block.go

+		// FIXME(BLOCKING): Do we actually want to consult the CID codecs table
+		//  even with the old --format options? Or do we always want to check
+		//  the multicodec one?


Manually maintained CID mappings are being removed in ipfs/go-cid#137, we want to only use go-multicodec from now. Will rebase + push refactor shortly.

lidel · 2022-04-12T22:41:49Z

Pushed a6eb015:

Removes duplicated fields and replaces them with cid.Prefix

Codec, MhType and MhLength were  already in prefix, and we already return
prefix. A lot of duplicated values and code responsible for syncing them
did not really need to exist.

Will now take it for a spin in ipfs/kubo#8568 and adjust if necessary

Removes duplicated fields and replaces them with cid.Prefix Codec, MhType and MhLength were already in prefix, and we already return prefix. A lot of duplicated values and code responsible for syncing them did not really need to exist.

- switches to ipfs/interface-go-ipfs-core#80 (comment) - updates helptext of block commands to reflect the fact we now use CIDs everywhere and document the defaults - add tests for new --cid-codec and old --format behavior

lidel

Seems to be ready, this PR is being used in ipfs/kubo#8568.
See comments below for key changes.

lidel · 2022-04-13T02:15:25Z

options/block.go

 type BlockPutSettings struct {
-	Codec    string
-	MhType   uint64
-	MhLength int
-	Pin      bool
+	CidPrefix cid.Prefix
+	Pin       bool
 }


ℹ️ this is the core change in this PR – codec/mh were present twice, this folds them into cid.Prefix that is then used in go-ipfs in https://github.com/ipfs/go-ipfs/pull/8568/files#diff-0c55f05d877cead8d5b53c5dea1912d738456bc774332cd9406efef74e224afcR44

lidel · 2022-04-13T02:16:47Z

options/block.go

+		// Opt-in CIDv0 support for backward-compatibility
+		if format == "v0" {
+			settings.CidPrefix.Version = 0
+		}
+
+		// Fixup a legacy (invalid) names for dag-pb (0x70)
+		if format == "v0" || format == "protobuf" {
+			format = "dag-pb"
+		}
+
+		// Fixup invalid name for dag-cbor (0x71)
+		if format == "cbor" {
+			format = "dag-cbor"
+		}


ℹ️ this maintains backward-compatibility for existing users of ipfs block put --format – see sharness+CI tests in ipfs/kubo#8568

lidel · 2022-04-13T02:17:19Z

tests/block.go

+	t.Run("TestBlockPut (get raw CIDv1)", tp.TestBlockPut)
+	t.Run("TestBlockPutCidCodec: dag-pb", tp.TestBlockPutCidCodecDagPb)
+	t.Run("TestBlockPutCidCodec: dag-cbor", tp.TestBlockPutCidCodecDagCbor)
+	t.Run("TestBlockPutFormat (legacy): cbor → dag-cbor", tp.TestBlockPutFormatDagCbor)
+	t.Run("TestBlockPutFormat (legacy): protobuf → dag-pb", tp.TestBlockPutFormatDagPb)
+	t.Run("TestBlockPutFormat (legacy): v0 → CIDv0", tp.TestBlockPutFormatV0)


ℹ️ these tests run in ipfs/kubo#8568

github-actions · 2022-04-21T15:27:38Z

Suggested version: v0.7.0
Comparing to: v0.6.2 (diff)

Changes in go.mod file(s):

diff --git a/go.mod b/go.mod
index 5d9726e..b7fc215 100644
--- a/go.mod
+++ b/go.mod
@@ -8,14 +8,59 @@ require (
 	github.com/ipfs/go-merkledag v0.6.0
 	github.com/ipfs/go-path v0.1.1
 	github.com/ipfs/go-unixfs v0.2.4
-	github.com/klauspost/cpuid/v2 v2.0.6 // indirect
 	github.com/libp2p/go-libp2p-core v0.8.5
 	github.com/multiformats/go-multiaddr v0.3.3
 	github.com/multiformats/go-multibase v0.0.3
+	github.com/multiformats/go-multicodec v0.3.0
 	github.com/multiformats/go-multihash v0.0.15
+)
+
+require (
+	github.com/btcsuite/btcd v0.21.0-beta // indirect
+	github.com/crackcomm/go-gitignore v0.0.0-20170627025303-887ab5e44cc3 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/google/uuid v1.2.0 // indirect
+	github.com/hashicorp/golang-lru v0.5.4 // indirect
+	github.com/ipfs/bbloom v0.0.4 // indirect
+	github.com/ipfs/go-block-format v0.0.3 // indirect
+	github.com/ipfs/go-blockservice v0.3.0 // indirect
+	github.com/ipfs/go-datastore v0.5.0 // indirect
+	github.com/ipfs/go-ipfs-blockstore v1.2.0 // indirect
+	github.com/ipfs/go-ipfs-chunker v0.0.1 // indirect
+	github.com/ipfs/go-ipfs-ds-help v1.1.0 // indirect
+	github.com/ipfs/go-ipfs-exchange-interface v0.1.0 // indirect
+	github.com/ipfs/go-ipfs-posinfo v0.0.1 // indirect
+	github.com/ipfs/go-ipfs-util v0.0.2 // indirect
+	github.com/ipfs/go-ipld-legacy v0.1.0 // indirect
+	github.com/ipfs/go-log v1.0.5 // indirect
+	github.com/ipfs/go-log/v2 v2.3.0 // indirect
+	github.com/ipfs/go-metrics-interface v0.0.1 // indirect
+	github.com/ipfs/go-verifcid v0.0.1 // indirect
+	github.com/ipld/go-codec-dagpb v1.3.0 // indirect
+	github.com/ipld/go-ipld-prime v0.11.0 // indirect
+	github.com/jbenet/goprocess v0.1.4 // indirect
+	github.com/klauspost/cpuid/v2 v2.0.6 // indirect
+	github.com/libp2p/go-buffer-pool v0.0.2 // indirect
+	github.com/libp2p/go-openssl v0.0.7 // indirect
+	github.com/mattn/go-isatty v0.0.13 // indirect
+	github.com/minio/blake2b-simd v0.0.0-20160723061019-3f5f724cb5b1 // indirect
+	github.com/minio/sha256-simd v1.0.0 // indirect
+	github.com/mr-tron/base58 v1.2.0 // indirect
+	github.com/multiformats/go-base32 v0.0.3 // indirect
+	github.com/multiformats/go-base36 v0.1.0 // indirect
+	github.com/multiformats/go-varint v0.0.6 // indirect
+	github.com/opentracing/opentracing-go v1.2.0 // indirect
+	github.com/polydawn/refmt v0.0.0-20201211092308-30ac6d18308e // indirect
+	github.com/spacemonkeygo/spacelog v0.0.0-20180420211403-2296661a0572 // indirect
+	github.com/whyrusleeping/cbor-gen v0.0.0-20200123233031-1cdf64d27158 // indirect
+	github.com/whyrusleeping/chunker v0.0.0-20181014151217-fe64bd25879f // indirect
+	go.uber.org/atomic v1.7.0 // indirect
 	go.uber.org/multierr v1.7.0 // indirect
+	go.uber.org/zap v1.16.0 // indirect
 	golang.org/x/crypto v0.0.0-20210513164829-c07d793c2f9a // indirect
 	golang.org/x/sys v0.0.0-20210514084401-e8d321eab015 // indirect
+	golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 // indirect
+	google.golang.org/protobuf v1.27.1 // indirect
 )
 
-go 1.16
+go 1.17

gorelease says:

# github.com/ipfs/interface-go-ipfs-core/options
## incompatible changes
BlockPutOptions: changed from func(...BlockPutOption) (*BlockPutSettings, github.com/ipfs/go-cid.Prefix, error) to func(...BlockPutOption) (*BlockPutSettings, error)
BlockPutSettings.Codec: removed
BlockPutSettings.MhLength: removed
BlockPutSettings.MhType: removed
## compatible changes
BlockPutSettings.CidPrefix: added
blockOpts.CidCodec: added

# github.com/ipfs/interface-go-ipfs-core/tests
## incompatible changes
(*TestSuite).TestBlockPutFormat: removed
## compatible changes
(*TestSuite).TestBlockPutCidCodecDagCbor: added
(*TestSuite).TestBlockPutCidCodecDagPb: added
(*TestSuite).TestBlockPutFormatDagCbor: added
(*TestSuite).TestBlockPutFormatDagPb: added
(*TestSuite).TestBlockPutFormatV0: added

# summary
Suggested version: v0.7.0

gocompat says:

(empty)

lidel · 2022-04-21T15:39:00Z

Merging as per verbal from Adin to unblock ipfs/kubo#8568 for go-ipfs 0.13

schomatis requested a review from aschmahmann November 18, 2021 14:40

schomatis self-assigned this Nov 18, 2021

This was referenced Nov 19, 2021

Use standard IPLD codec names across the CLI/HTTP API ipfs/kubo#8471

Closed

fix(cmds): CIDv1 and correct multicodecs in 'block put' and 'cid codecs’ ipfs/kubo#8568

Merged

BigLep assigned aschmahmann Jan 5, 2022

BigLep added this to In Review in Go IPFS Roadmap via automation Jan 5, 2022

BigLep unassigned aschmahmann Mar 2, 2022

BigLep added this to the go-ipfs 0.14 milestone Mar 11, 2022

lidel assigned lidel and unassigned schomatis Mar 30, 2022

schomatis added 4 commits April 12, 2022 17:53

feat(block options): add store codec

a2c0c40

go mod tidy

385f0ca

add basic test

fe6da49

update cid cbor to multicodec dag-cbor

16bad39

lidel removed this from the go-ipfs 0.14 milestone Apr 12, 2022

lidel reviewed Apr 12, 2022

View reviewed changes

lidel force-pushed the schomatis/feat/block-option/add-store-codec branch from 1187b24 to 48b4b81 Compare April 12, 2022 22:40

lidel changed the title ~~feat(block options): add store codec~~ refactor: BlockPutSettings.CidPrefix Apr 12, 2022

refactor: BlockPutSettings.CidPrefix

a6eb015

Removes duplicated fields and replaces them with cid.Prefix Codec, MhType and MhLength were already in prefix, and we already return prefix. A lot of duplicated values and code responsible for syncing them did not really need to exist.

lidel force-pushed the schomatis/feat/block-option/add-store-codec branch from 48b4b81 to a6eb015 Compare April 12, 2022 23:37

test: CIDv1 raw and dag-pb cases

7e14e1e

lidel marked this pull request as ready for review April 13, 2022 02:12

lidel approved these changes Apr 13, 2022

View reviewed changes

chore: release 0.7.0

3981781

lidel merged commit a3374d9 into master Apr 21, 2022

lidel deleted the schomatis/feat/block-option/add-store-codec branch April 21, 2022 15:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

refactor: BlockPutSettings.CidPrefix #80

refactor: BlockPutSettings.CidPrefix #80

schomatis commented Nov 18, 2021 •

edited by lidel

lidel Mar 30, 2022 •

edited

lidel commented Apr 12, 2022 •

edited

lidel left a comment

lidel Apr 13, 2022

lidel Apr 13, 2022 •

edited

lidel Apr 13, 2022

github-actions bot commented Apr 21, 2022

lidel commented Apr 21, 2022

refactor: BlockPutSettings.CidPrefix #80

refactor: BlockPutSettings.CidPrefix #80

Conversation

schomatis commented Nov 18, 2021 • edited by lidel

TLDR

CIDv1 returned by default

CID prefix is now part of BlockPutSettings

BREAKING CHANGES

lidel Mar 30, 2022 • edited

Choose a reason for hiding this comment

lidel commented Apr 12, 2022 • edited

lidel left a comment

Choose a reason for hiding this comment

lidel Apr 13, 2022

Choose a reason for hiding this comment

lidel Apr 13, 2022 • edited

Choose a reason for hiding this comment

lidel Apr 13, 2022

Choose a reason for hiding this comment

github-actions bot commented Apr 21, 2022

lidel commented Apr 21, 2022

schomatis commented Nov 18, 2021 •

edited by lidel

lidel Mar 30, 2022 •

edited

lidel commented Apr 12, 2022 •

edited

lidel Apr 13, 2022 •

edited