Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release v0.10 #8176

Closed
62 tasks done
BigLep opened this issue Jun 4, 2021 · 19 comments · Fixed by #8459, #8472, #8474 or #8475
Closed
62 tasks done

Release v0.10 #8176

BigLep opened this issue Jun 4, 2021 · 19 comments · Fixed by #8459, #8472, #8474 or #8475
Assignees
@guseggert @aschmahmann
Labels
topic/release Topic release
Projects
Go IPFS Roadmap
Milestone
go-ipfs 0.10

Comments

@BigLep
Copy link
Contributor

BigLep commented Jun 4, 2021 •
edited by guseggert

go-ipfs 0.10.0 Release

We're happy to announce go-ipfs 0.10.0. This release brings some big changes to the IPLD internals of go-ipfs that make working with non-UnixFS DAGs easier than ever. There are also a variety of new commands and configuration options available.

As usual, this release includes important fixes, some of which may be critical for security. Unless the fix addresses a bug being exploited in the wild, the fix will not be called out in the release notes. Please make sure to update ASAP. See our release process for details.

🛠 TLDR: BREAKING CHANGES

  • ipfs dag get
    • default output changed to dag-json
    • dag-pb (e.g. unixfs) field names changed - impacts userland code that works with dag-pb objects returned by dag get
    • no longer emits an additional new-line character at the end of the data output
  • ipfs dag put
    • defaults changed to reduce ambiguity and surprises: input is now assumed to be dag-json, and data is serialized to dag-cbor at rest.
    • --format and --input-enc were removed and replaced with --store-codec and --input-codec
    • codec names now match the ones defined in the multicodec table
    • dag-pb (e.g. unixfs) field names changed - impacts userland code that works with dag-pb objects stored via dag put

Keep reading to learn more details.

🔦 Highlights

🌲 IPLD Levels Up

The handling of data serialization as well as many aspects of DAG traversal and pathing have been migrated from older libraries, including go-merkledag and go-ipld-format to the new go-ipld-prime library and its components. This allows us to use many of the newer tools afforded by go-ipld-prime, stricter and more uniform codec implementations, support for additional (pluggable) codecs, and some minor performance improvements.

This is significant refactor of a core component that touches many parts of IPFS, and does come with some breaking changes:

  • IPLD plugins:
    • The PluginIPLD interface has been changed to utilize go-ipld-prime. There is a demonstration of the change in the bundled git plugin.
  • The semantics of dag put and dag get change:
    • dag get now takes the output-codec option which accepts a multicodec name used to encode the output. By default this is dag-json, which is a strict and deterministic subset of JSON created by the IPLD team. Users may notice differences from the previously plain Go JSON output, particularly where bytes are concerned which are now encoded using a form similar to CIDs: {"/":{"bytes":"unpadded-base64-bytes"}} rather than the previously Go-specific plain padded base64 string. See the dag-json specification for an explanation of these forms.
    • dag get no longer prints an additional new-line character at the end of the encoded block output. This means that the output as presented by dag get are the exact bytes of the requested node. A round-trip of such bytes back in through dag put using the same codec should result in the same CID.
    • dag put uses the input-codec option to specify the multicodec name of the format data is being provided in, and the store-codec option to specify the multicodec name of the format the data should be stored in at rest. These formerly defaulted to json and cbor respectively. They now default to dag-json and dag-cbor respectively but may be changed to any supported codec (bundled or loaded via plugin) by its multicodec name.
    • The json and cbor multicodec names (as used by input-enc and format options) are now no longer aliases for dag-json and dag-cbor respectively. Instead, they now refer to their proper multicodec types. cbor refers to a plain CBOR format, which will not encode CIDs and does not have strict deterministic encoding rules. json is a plain JSON format, which also won't encode CIDs and will encode bytes in the Go-specific padded base64 string format rather than the dag-json method of byte encoding. See https://ipld.io/specs/codecs/ for more information on IPLD codecs.
    • protobuf is no longer used as the codec name for dag-pb
    • The codec name raw is used to mean Bytes in the IPLD Data Model
  • UnixFS refactor. The dag-pb codec, which is used to encode UnixFS data for IPFS, is now represented through the dag API in a form that mirrors the protobuf schema used to define the binary format. This unifies the implementations and specification of dag-pb across the IPLD and IPFS stacks. Previously, additional layers of code for file and directory handling within IPFS between protobuf serialization and UnixFS obscured the protobuf representation. Much of this code has now been replaced and there are fewer layers of transformation. This means that interacting with dag-pb data via the dag API will use different forms:
    • Previously, using dag get on a dag-pb block would present the block serialized as JSON as {"data":"padded-base64-bytes","links":[{"Name":"foo","Size":100,"Cid":{"/":"Qm..."}},...]}.
    • Now, the dag-pb data with dag-json codec for output will be serialized using the data model from the dag-pb specification: {"Data":{"/":{"bytes":"unpadded-base64-bytes"}},"Links":[{"Name":"foo","Tsize":100,"Hash":{"/":"Qm..."}},...]}. Aside from the change in byte formatting, most field names have changed: data → Data, links → Links, Size → Tsize, Cid → Hash. Note that this output can be changed now using the output-codec option to specify an alternative codec.
    • Similarly, using dag put and a store-codec option of dag-pb now requires that the input conform to this dag-pb specified form. Previously, input using {"data":"...","links":[...]} was accepted, now it must be {"Data":"...","Links":[...]}.
    • Previously it was not possible to use paths to navigate to any of these properties of a dag-pb node, the only possible paths were named links, e.g. dag get QmFoo/NamedLink where NamedLink was one of the links whose name was NamedLink. This functionality remains the same, but by prefixing the path with /ipld/ we enter data model pathing semantics and can dag get /ipld/QmFoo/Links/0/Hash to navigate to links or /ipld/QmFoo/Data to simply retrieve the data section of the node, for example.
    • ℹ See the dag-pb specification for details on the codec and its data model representation.
    • ℹ See this detailed write-up for further background on these changes.

Ⓜ Multibase Command

go-ipfs now provides utility commands for working with multibase:

$ echo -n hello | ipfs multibase encode -b base16 > file-mbase16
$ cat file-mbase16
f68656c6c6f

$ ipfs multibase decode file-mbase16
hello

$ cat file-mbase16 | ipfs multibase decode
hello

$ ipfs multibase transcode -b base2 file-mbase16
00110100001100101011011000110110001101111

See ipfs multibase --help for more examples.

🔨 Bitswap now supports greater configurability

This release adds an Internal section to the configuration file that is designed to help advanced users optimize their setups without needing a custom binary. The Internal section is not guaranteed to be the same from release to release and may not be covered by migrations. If you use the Internal section you should be making sure to check the config documentation between releases for any changes.

🐚 Programmatic shell completions command

ipfs commands completion bash will generate a bash completion script for go-ipfs commands

📜 Profile collection command

Performance profiles can now be collected using ipfs diag profile. If you need to do some debugging or have an issue to submit the collected profiles are very useful to have around.

🍎 Mac OS notarized binaries

The go-ipfs and related migration binaries (for both Intel and Apple Sillicon) are now signed and notarized to make Mac OS installation easier.

👨‍👩‍👦 Improved MDNS

There is a completed implementation of the revised libp2p MDNS spec. This should result in better MDNS discovery and better local/offline operation as a result.

🚗 CAR import statistics

dag import command now supports --stats option which will include the number of imported blocks and their total size in the output.

🕸 Peering command

This release adds swarm peering command for easy management of the peering subsystem. Peer in the peering subsystem is maintained to be connected at all times, and gets reconnected on disconnect with a back-off.

See ipfs swarm peering --help for more details.

✅ Release Checklist

For each RC published in each stage:

  • version string in version.go has been updated (in the release-vX.Y.Z branch).
  • tag commit with vX.Y.Z-rcN
  • upload to dist.ipfs.io
    1. Build: https://github.com/ipfs/distributions#usage.
    2. Pin the resulting release.
    3. Make a PR against ipfs/distributions with the updated versions, including the new hash in the PR comment.
    4. Ask the infra team to update the DNSLink record for dist.ipfs.io to point to the new distribution.
  • cut a pre-release on github and upload the result of the ipfs/distributions build in the previous step.
  • Announce the RC:

Checklist:

  • Stage 0 - Automated Testing
    • Fork a new branch (release-vX.Y.Z) from master and make any further release related changes to this branch. If any "non-trivial" changes (see the footnotes of docs/releases.md for a definition) get added to the release, uncheck all the checkboxes and return to this stage.
      • Follow the RC release process to cut the first RC.
      • Bump the version in version.go in the master branch to vX.(Y+1).0-dev.
    • Automated Testing (already tested in CI) - Ensure that all tests are passing, this includes:
  • Stage 1 - Internal Testing
    • CHANGELOG.md has been updated
    • Infrastructure Testing:
      • Deploy new version to a subset of Bootstrappers
      • Deploy new version to a subset of Gateways
      • Deploy new version to a subset of Preload nodes
      • Collect metrics every day. Work with the Infrastructure team to learn of any hiccup
    • IPFS Application Testing - Run the tests of the following applications:
  • Stage 2 - Community Dev Testing
    • Reach out to the IPFS early testers listed in docs/EARLY_TESTERS.md for testing this release (check when no more problems have been reported). If you'd like to be added to this list, please file a PR.
    • Reach out to on IRC for beta testers.
    • Run tests available in the following repos with the latest beta (check when all tests pass):
  • Stage 3 - Community Prod Testing
  • Stage 4 - Release
    • Final preparation
      • Verify that version string in version.go has been updated.
      • Merge release-vX.Y.Z into the release branch.
      • Tag this merge commit (on the release branch) with vX.Y.Z.
      • Release published
      • Cut a new ipfs-desktop release
    • Publish a Release Blog post (at minimum, a c&p of this release issue with all the highlights, API changes, link to changelog and thank yous)
    • Broadcasting (link to blog post)
  • Post-Release
    • Merge the release branch back into master, ignoring the changes to version.go (keep the -dev version from master).
    • Create an issue using this release issue template for the next release.
    • Make sure any last-minute changelog updates from the blog post make it back into the CHANGELOG.

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works and what you can do with it is at discuss.ipfs.io. We are also available at the #lobby:ipfs.io Matrix channel which is bridged with other chat platforms.
@BigLep BigLep added the topic/release Topic release label Jun 4, 2021
@BigLep BigLep added this to the go-ipfs 0.10 milestone Jun 4, 2021
@BigLep BigLep pinned this issue Jun 4, 2021
@BigLep BigLep changed the title Release 0.10 Release v0.10 Jun 9, 2021
@eminence
Copy link
Contributor

improvements that will enable us to release faster with more effort.

more effort, or less effort?

@BigLep
Copy link
Contributor Author

BigLep commented Jun 30, 2021

@eminence : doh - fixed - thanks!

This was referenced Jun 30, 2021
Closed
Closed
@BigLep BigLep added this to In Progress in Go IPFS Roadmap Jul 1, 2021
@BigLep BigLep moved this from In Progress to Backlog in Go IPFS Roadmap Jul 1, 2021
@BigLep BigLep mentioned this issue Aug 12, 2021
Merged
48 tasks
@aschmahmann
Copy link
Contributor

aschmahmann commented Aug 20, 2021 •
edited

Changelog

Full Changelog

❤️ Contributors

Contributor Commits Lines ± Files Changed
Daniel Martí 42 +8549/-6587 170
Eric Myhre 55 +5883/-6715 395
Marten Seemann 100 +1814/-2028 275
Steven Allen 80 +1573/-1998 127
hannahhoward 18 +1721/-671 53
Will 2 +1114/-1217 18
Andrew Gillis 2 +1220/-720 14
gammazero 3 +43/-1856 10
Masih H. Derkani 3 +960/-896 8
Adin Schmahmann 25 +1458/-313 44
vyzo 27 +986/-353 60
Will Scott 6 +852/-424 16
Rod Vagg 19 +983/-255 66
Petar Maymounkov 6 +463/-179 22
web3-bot 10 +211/-195 24
adlrocha 1 +330/-75 15
RubenKelevra 2 +128/-210 2
Ian Davis 3 +200/-109 17
Cory Schwartz 3 +231/-33 7
Keenan Nemetz 1 +184/-71 2
Randy Reddig 2 +187/-53 8
Takashi Matsuda 3 +201/-2 7
guseggert 4 +161/-20 9
Lucas Molas 5 +114/-47 27
nisdas 4 +115/-45 7
Michael Muré 6 +107/-33 24
Richard Ramos 2 +113/-9 3
Marcin Rataj 12 +88/-24 13
Ondrej Prazak 2 +104/-6 4
Michal Dobaczewski 2 +77/-28 3
Jorropo 3 +9/-75 4
Andey Robins 1 +70/-3 3
Gus Eggert 10 +34/-31 12
noot 1 +54/-9 5
Maxim Merzhanov 1 +29/-24 1
Adrian Lanzafame 1 +30/-13 2
Bogdan Stirbat 1 +22/-16 2
Shad Sterling 1 +28/-3 1
Jesse Bouwman 5 +30/-0 5
Pavel Karpy 1 +19/-7 2
lasiar 5 +14/-10 5
Dennis Trautwein 1 +20/-4 2
Louis Thibault 1 +22/-1 2
whyrusleeping 2 +21/-1 2
aarshkshah1992 3 +12/-8 3
Peter Rabbitson 2 +20/-0 2
bt90 2 +17/-2 2
Dominic Della Valle 1 +13/-1 2
Audrius Butkevicius 1 +12/-1 1
Brian Strauch 1 +9/-3 1
Aarsh Shah 2 +1/-11 2
Whyrusleeping 1 +11/-0 1
Max 1 +7/-3 1
vallder 1 +3/-5 1
Michael Burns 3 +2/-6 3
Lasse Johnsen 1 +4/-4 2
snyh 1 +5/-2 1
Hector Sanjuan 2 +3/-2 2
市川恭佑 (ebi) 1 +1/-3 1
godcong 2 +2/-1 2
Mathis Engelbart 1 +1/-2 1
folbrich 1 +1/-1 1
Med Mouine 1 +1/-1 1

Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started:

@ianopolous
Copy link
Member

My reading of this is that this doesn't include the pending move of the blockstore from cids to multihashes. Is that correct?

@aschmahmann
Copy link
Contributor

@ianopolous correct. That is currently planned to be a release all of its own (i.e. that will be the only change in that release in order to ease people through the migration). It's currently scheduled for v0.12.0

@BigLep BigLep moved this from Current Release Backlog to In Progress in Go IPFS Roadmap Aug 24, 2021
@b5
Copy link
Contributor

b5 commented Aug 25, 2021

Is anyone aware of anything like https://deno.land/benchmarks for go-ipfs? The switch to go-ipld-prime under the hood is very exciting (thanks @hannahhoward !). To help assess release candidates, I'd really appreciate the sanity checks an automated set of benchmarks run between releases would provide.

@aschmahmann
Copy link
Contributor

aschmahmann commented Aug 31, 2021 •
edited by lidel

Early testers ping for RC1 testing 😄

@Jorropo
Copy link
Contributor

Jorropo commented Aug 31, 2021

<List of items with PRs and/or Issues to be considered for this release>

#8177 pls ?

@obo20
Copy link

obo20 commented Aug 31, 2021

Early testers ping for RC testing 😄

Things are looking good on our end. We've been running this on a segment of our infrastructure and there are no issues to report as of yet.

@tabcat
Copy link
Contributor

tabcat commented Sep 2, 2021

tracking orbit-db support: orbitdb/orbitdb#909

@aschmahmann aschmahmann mentioned this issue Sep 2, 2021
Closed
4 tasks
@kallisti5
Copy link

Any thoughts on #8413 ? Seems like an easy add with a big usability improvement.

@BigLep
Copy link
Contributor Author

BigLep commented Sep 9, 2021

Per 2021-09-09 verbal:

  1. Clearer callout on the breaking changes TLDR (e.g., ipfs-dag break)
  2. Clearer documentation on how to migrate
  3. Need to call out determinstic CBOR encoding
  4. Need to add callouts for the various items Rod found
  5. We need to track down that the JS HTTP tests didn't run (related to OrbitDB also finding failures during their test runs). This may just need to be a tracking issue for cases where Go is ahead of JS.

@BigLep
Copy link
Contributor Author

BigLep commented Sep 9, 2021 •
edited

@Jorropo:

#8177 pls ?

This won't make it in this release

@kallisti5:

Any thoughts on #8413 ? Seems like an easy add with a big usability improvement.

This won't make it in this release

@BigLep BigLep linked a pull request Sep 27, 2021 that will close this issue
v0.10.0-rc2 #8459
Merged
5 tasks
@lidel lidel mentioned this issue Sep 28, 2021
Merged
@lidel
Copy link
Member

lidel commented Sep 28, 2021 •
edited

v0.10.0-rc2

Just published RC2 on github, dist.ipfs.io, NPM and dockerhub.

👉 We are planning to ship the final release around Thursday 30th (TBD), mind the window for testing is a bit short.

Changes since -rc1

  • improved how we introduce breaking changes in ipfs dag get and ipfs dag put
    • rc1 had them too, but we realized that some edge cases around codec name aliases were difficult to reason about and could lead to silent errors
    • rc2 makes the changes very clear and explicit (renamed params, strict codec names). this should make migration and refactoring of userland apps that rely on the old behavior of dag pub/get easier with no surface for ambiguity
    • details in the updated draft of release notes
  • new commands / parameters:
    • ipfs multibase transcode --help
    • ipfs swarm peering --help
    • ipfs dag import --stats

Call for feedback

Early testers – ping for v0.10.0-rc2 testing 😄

@MichaelMure
Copy link
Contributor

Would it be reasonable to include this merkledag fix ?

@tabcat
Copy link
Contributor

tabcat commented Sep 30, 2021

tests are passing locally using orbitdb/orbitdb#913 (will be merged and published very soon) + go-ipfs@0.10.0-rc2. 👍

@aschmahmann
Copy link
Contributor

@MichaelMure IIUC that fix doesn't effect really effect the go-ipfs binary, but might be useful in some edge cases with go-ipfs as a library.

If so then we can do a go-merkledag release. I'd prefer not to add much more into the release if we can to expedite getting it out.

If there's a more pressing need or problem that I'm missing lmk (here, on the go-merkeldag PR, or in a DM)

This was linked to pull requests Sep 30, 2021
chore: update changelog for v0.10.0 #8472
Merged
Release v0.10.0 #8474
Merged
Merge Release v0.10.0 #8475
Merged
@BigLep BigLep moved this from In Progress to In Review in Go IPFS Roadmap Oct 5, 2021
Go IPFS Roadmap automation moved this from In Review to Done Oct 7, 2021
@eminence
Copy link
Contributor

eminence commented Oct 7, 2021

perhaps https://docs.ipfs.io/install/command-line/#official-distributions could be updated to reference the 0.10 release

@lidel
Copy link
Member

lidel commented Oct 7, 2021

Good catch @eminence!
I've opened ipfs/ipfs-docs#904 but this is a good candidate for automation (or adding a special 'latest' dir to dist.ipfs.io releases).

@BigLep BigLep unpinned this issue Oct 8, 2021
@guseggert guseggert mentioned this issue Nov 23, 2021
Closed
80 tasks
@aschmahmann aschmahmann mentioned this issue Dec 9, 2021
Closed
59 tasks
@aschmahmann aschmahmann mentioned this issue May 4, 2022
Closed
65 tasks
@guseggert guseggert mentioned this issue Jun 9, 2022
Closed
68 tasks
@BigLep BigLep mentioned this issue Aug 17, 2022
Closed
72 tasks
@galargh galargh mentioned this issue Sep 19, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@guseggert guseggert

@aschmahmann aschmahmann

Labels
topic/release Topic release
Projects
IPFS Shipyard Team
Archived in project
Go IPFS Roadmap
  
Done
Milestone
go-ipfs 0.10