Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release v0.11 #8343

Closed
79 of 80 tasks
BigLep opened this issue Aug 14, 2021 · 7 comments · Fixed by #8571
Closed
79 of 80 tasks

Release v0.11 #8343

BigLep opened this issue Aug 14, 2021 · 7 comments · Fixed by #8571
Assignees
@guseggert
Labels
topic/release Topic release
Projects
Go IPFS Roadmap
Milestone
go-ipfs 0.11

Comments

@BigLep
Copy link
Contributor

BigLep commented Aug 14, 2021 •
edited by guseggert

go-ipfs 0.11.0 Release Notes

We're happy to announce go-ipfs 0.11.0. This release comes with improvements to the UnixFS Sharding and PubSub experiments as well as support for Circuit-Relay v2 which sets the network up for decentralized hole punching support.

As usual, this release includes important fixes, some of which may be critical for security. Unless the fix addresses a bug being exploited in the wild, the fix will not be called out in the release notes. Please make sure to update ASAP. See our release process for details.

🛠 BREAKING CHANGES

  • UnixFS sharding is now automatic and enabled by default
    • HAMT-based sharding is applied to large directories (i.e. those that would serialize into block larger than ~256KiB)s. This means importing data via commands like ipfs add -r <directory> may result in different CIDs due to the different DAG representations.
    • Support for Experimental.ShardingEnabled is removed.
  • go-ipfs can no longer act as a Circuit Relay v1
    • Node will refuse to start if Swarm.EnableRelayHop is set to true
    • If you depend on v1 relay service provider, see "Removal of v1 relay service" section for available migration options.
  • HTTP RPC wire format for experimental commands at /api/v0/pubsub changed.

Keep reading to learn more details.

🔦 Highlights

🗃️ Automatic UnixFS sharding

Truly big directories can have so many items, that the root block with all of their names is too big to be exchanged with other peers. This was partially solved by HAMT-sharding, which was introduced a while ago as opt-in. The main downside of the implementation was that it was a global flag that sharded all imported directories (big and small).

This release solves that inconvenience by making UnixFS sharding smarter and applies it only to larger directories (i.e. directories that would be at least ~256KiB). This is now the default behavior in ipfs add and ipfs files commands, where UnixFS sharding works out-of-the-box.

🔁 Circuit Relay v2

This release adds support for the circuit relay v2 protocol based on the reference implementation from go-libp2p 0.16.

This is the cornerstone for maximizing p2p connections between IPFS peers. Every publicly dialable peer can now act as a limited relay v2, which can be used for hole punching and other decentralized signaling protocols.

Limited relay v2 configuration options

go-ipfs can now be configured to act as a RelayClient that uses other peers for autorelay functionality when behind a NAT, or provide a limited RelayService to other peers on the network.

Starting with go-ipfs v0.11 every publicly dialable go-ipfs (based on AutoNAT determination) will start a limited RelayService. RelayClient remains disabled by default for now, as we want the network to update and get enough v2 service providers first.

Note: the limited Circuit Relay v2 provided with this release only allows low-bandwidth protocols (identify, ping, holepunch) over transient connections. If you want to relay things like bitswap sessions, you need to set up a v1 relay by some other means. See details below.

Removal of unlimited v1 relay service provider

Switching to v2 of the relay spec means removal or deprecation of configuration keys that were specific to v1.

  • Relay transport and client support circuit-relay v2:
    • Swarm.EnableAutoRelay was replaced by Swarm.RelayClient.Enable.
    • Swarm.DisableRelay is deprecated, relay transport can be now disabled globally (both client and service) by setting Swarm.Transports.Network.Relay to false
  • Relay v1 service provider was replaced by v2:
    • Swarm.EnableRelayHop no longer starts an unlimited v1 relay. If you have it set to true the node will refuse to start and display an error message.
    • Existing users who choose to continue running a v1 relay should migrate their setups to relay v1 based on js-ipfs running in node, or the standalone libp2p-relay-daemon configured with RelayV1.Enabled set to true. Be mindful that v1 relays are unlimited, and one may want to set up some ACL based either on PeerIDs or Subnets.

🕳 Decentralized Hole Punching (DCUtR protocol client)

We are working towards enabling hole punching for NAT traversal when port forwarding is not possible.

go-libp2p 0.16 provides an implementation of the DCUtR (decentralized hole punching) protocol. It is hidden behind the Swarm.EnableHolePunching configuration flag.

When enabled, go-ipfs will coordinate with the counterparty using a relayed v2 connection, to upgrade to a direct connection through a NAT/firewall whenever possible.

This feature is disabled by default in this release, but we hope to enable it by default as soon the network updates to go-ipfs v0.11 and gains a healthy set of limited v2 relays.

💬 Multibase in PubSub HTTP RPC API

This release fixed some edge cases that were reported by users of the PubSub experiment, getting it closer to becoming a stable feature of go-ipfs. Some PubSub users will notice that the plaintext limitation is lifted: one can now use line breaks in messages published to non-ascii topic names, or even publish arbitrary bytes to arbitrary topics. It required a change to the wire format used when pubsub commands are executed over the HTTP RPC API at /api/v0/pubsub/*, and also modified the behavior of the ipfs pubsub pub command, which now is publishing only a single pubsub message with data read from a file or stdin.

PubSub client migration tips

If you use the HTTP RPC API with the go-ipfs-http-client library, make sure to update to the latest version. The next version of js-ipfs-http-client will use the new wire format as well, so you don't need to do anything.

If you use /api/v0/pubsub/* directly or maintain your own client library, you must adjust your HTTP client code. Byte fields and URL args are now encoded in base64url Multibase. Encode/decode bytes using the ipfs multibase --help commands, or use the multiformats libraries (js-multiformats, go-multibase).

Low level changes:

  • topic passed as URL arg in requests to /api/v0/pubsub/* must be encoded in URL-safe multibase (base64url)
  • data, from, seqno and topicIDs returned in JSON responses are now encoded in multibase
  • Peer IDs returned in from now use the same default text representation from go-libp2p and peerid encoder/decoder from libp2p. This means the same text representation as in as in swarm peers, which makes it possible to compare them without decoding multibase.
  • /api/v0/pubsub/pub no longer accepts data to be passed as URL, it has to be sent as multipart/form-data. This removes size limitations based on URL length, and enables regular HTTP clients to publish data to PubSub topics. For example, to publish some-file to topic named test-topic using vanilla curl, one would execute: curl -X POST -v -F "stdin=@some-file" 'http://127.0.0.1:5001/api/v0/pubsub/pub?arg=$(echo -n test-topic | ipfs multibase encode -b base64url)'
  • ipfs pubsub pub on the command line no longer accepts variadic data arguments. Instead, it expects a single file input or stream of bytes from stdin. This ensures arbitrary stream of bytes can be published, removing limitation around messages that include \n or \r\n.

⚙️ New configuration flags

  • Addresses.AppendAnnounce is an array of multiaddrs, similar to Addresses.Announce, except it does not override inferred swarm addresses, but appends custom ones to the list.
  • Pubsub experiments can now be enabled via config, removing the need for CLI flag to be passed every time daemon starts:

🔐 Support for DAG-JOSE IPLD codec

JOSE is a standard for signing and encrypting JSON objects. DAG-JOSE is an IPLD codec based on JOSE and represented in CBOR. Upon encountering the dag-jose multicodec indicator, implementations can expect that the block contains dag-cbor encoded data which matches the IPLD schema from the DAG-JOSE spec.

This work was contributed by Ceramic and acts as a template for future IPFS improvements driven by the real world needs of the IPFS community.

🗺 What's left for release

🚢 Estimated shipping date

RC1: 2021-11-29
ECD: 2021-12-06

✅ Release Checklist

For each RC published in each stage:

  • version string in version.go has been updated (in the release-vX.Y.Z branch).
  • tag commit with vX.Y.Z-rcN
  • upload to dist.ipfs.io
    1. Build: https://github.com/ipfs/distributions#usage.
    2. Pin the resulting release.
    3. Make a PR against ipfs/distributions with the updated versions, including the new hash in the PR comment.
    4. Ask the infra team to update the DNSLink record for dist.ipfs.io to point to the new distribution.
  • cut a pre-release on github and upload the result of the ipfs/distributions build in the previous step.
  • Announce the RC:
    • On Matrix (both #ipfs and #ipfs-dev)
    • To the early testers listed in docs/EARLY_TESTERS.md. Do this by copy/pasting their GitHub usernames and checkboxes as a comment so they get a GitHub notification. (example)

Checklist:

  • Stage 0 - Automated Testing
    • Fork a new branch (release-vX.Y.Z) from master and make any further release related changes to this branch. If any "non-trivial" changes (see the footnotes of docs/releases.md for a definition) get added to the release, uncheck all the checkboxes and return to this stage.
      • Follow the RC release process to cut the first RC.
      • Bump the version in version.go in the master branch to vX.(Y+1).0-dev.
    • Automated Testing (already tested in CI) - Ensure that all tests are passing, this includes:
  • Stage 1 - Internal Testing
    • CHANGELOG.md has been updated
    • Infrastructure Testing:
      • Deploy new version to a subset of Bootstrappers
      • Deploy new version to a subset of Gateways
      • Deploy new version to a subset of Preload nodes
      • Collect metrics every day. Work with the Infrastructure team to learn of any hiccup
    • IPFS Application Testing - Run the tests of the following applications:
  • Stage 2 - Community Prod Testing
  • Stage 3 - Release
    • Final preparation
      • Verify that version string in version.go has been updated.
      • Merge release-vX.Y.Z into the release branch.
      • Tag this merge commit (on the release branch) with vX.Y.Z.
      • Release published
      • Cut a new ipfs-desktop release
    • Submit this form to publish a blog post, linking to the GitHub release notes
    • Broadcasting (link to blog post)
  • Post-Release
    • Merge the release branch back into master, ignoring the changes to version.go (keep the -dev version from master).
    • Create an issue using this release issue template for the next release.
    • Make sure any last-minute changelog updates from the blog post make it back into the CHANGELOG.
    • Mark PR draft created for IPFS Desktop as ready for review.
    • https://github.com/ipfs/go-ds-s3/pull/210/files
    • Deploy our final release to our infra

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works and what you can do with it is at discuss.ipfs.io. We are also available at the #lobby:ipfs.io Matrix channel which is bridged with other chat platforms.

Release improvements for next time

< Add any release improvements that were observed this cycle here so they can get incorporated into future releases. >
@BigLep BigLep added the topic/release Topic release label Aug 14, 2021
@BigLep BigLep added this to the go-ipfs 0.11 milestone Aug 14, 2021
@BigLep BigLep pinned this issue Aug 14, 2021
@BigLep BigLep added this to Current Release Backlog in Go IPFS Roadmap Oct 5, 2021
@BigLep BigLep mentioned this issue Oct 8, 2021
Open
@BigLep BigLep moved this from Current Release Backlog to In Progress in Go IPFS Roadmap Nov 30, 2021
@BigLep BigLep linked a pull request Nov 30, 2021 that will close this issue
chore: update deps #8571
Merged
Go IPFS Roadmap automation moved this from In Progress to Done Nov 30, 2021
@aschmahmann aschmahmann reopened this Nov 30, 2021
@aschmahmann aschmahmann moved this from Done to In Progress in Go IPFS Roadmap Nov 30, 2021
@aschmahmann
Copy link
Contributor

aschmahmann commented Dec 1, 2021 •
edited

Changelog

Full Changelog

❤️ Contributors

Contributor Commits Lines ± Files Changed
Will 13 +73226/-130481 43
Masih H. Derkani 99 +10549/-5799 489
hannahhoward 43 +5515/-3293 233
Daniel Martí 60 +5312/-2883 208
Marten Seemann 175 +4839/-3254 396
Eric Myhre 73 +3924/-3328 175
Jessica Schilling 52 +2709/-2386 75
Rod Vagg 30 +2719/-1703 79
vyzo 10 +3516/-177 87
Gus Eggert 64 +1677/-1416 147
Adin Schmahmann 23 +1708/-381 95
Lucas Molas 14 +1557/-365 48
Will Scott 7 +1846/-15 34
Steven Allen 32 +537/-897 56
Cory Schwartz 3 +614/-109 12
rht 3 +576/-4 7
Simon Zhu 9 +352/-51 16
Petar Maymounkov 7 +173/-167 23
RubenKelevra 1 +107/-188 1
jwh 2 +212/-80 7
longfeiW 1 +4/-249 10
guseggert 5 +230/-21 11
Kevin Neaton 8 +137/-80 13
Takashi Matsuda 1 +199/-0 5
Andrey Kostakov 1 +107/-49 2
Jesse Bouwman 1 +151/-0 7
web3-bot 39 +136/-3 52
Marcin Rataj 16 +62/-57 25
Marco Munizaga 1 +118/-0 2
Aaron Riekenberg 4 +64/-52 6
Ian Davis 4 +81/-32 7
Jorropo 2 +79/-19 6
Mohsin Zaidi 1 +89/-1 20
Andey Robins 1 +70/-3 3
gammazero 3 +40/-25 4
Steve Loeppky 2 +26/-27 3
Dimitris Apostolou 1 +25/-25 15
Sudarshan Reddy 1 +9/-40 1
Richard Littauer 2 +42/-1 3
pymq 1 +32/-8 2
Dirk McCormick 2 +23/-1 2
Nicholas Bollweg 1 +21/-0 1
anorth 1 +14/-6 2
Jack Loughran 1 +16/-0 2
whyrusleeping 2 +11/-2 2
bt90 1 +13/-0 1
Yi Cao 1 +10/-0 1
Max 1 +7/-3 1
Juan Batiz-Benet 2 +8/-2 2
Keenan Nemetz 1 +8/-0 1
muXxer 1 +3/-3 1
galargh 2 +3/-3 3
Didrik Nordström 1 +2/-4 1
Ben Lubar 1 +3/-3 1
arjunraghurama 1 +5/-0 1
Whyrusleeping 1 +3/-2 1
TUSF 1 +3/-2 3
mathew-cf 1 +3/-1 2
Stephen Whitmore 1 +2/-2 1
Song Zhu 1 +2/-2 1
Michael Muré 1 +4/-0 1
Alex Good 1 +4/-0 2
aarshkshah1992 1 +2/-1 1
susarlanikhilesh 1 +1/-1 1
falstack 1 +1/-1 1
Michael Vorburger ⛑️ 1 +1/-1 1
Ismail Khoffi 1 +1/-1 1
George Xie 1 +1/-1 1
Bryan Stenson 1 +1/-1 1
Lars Gierth 1 +1/-0 1

@lidel lidel mentioned this issue Dec 1, 2021
Merged
@aschmahmann
Copy link
Contributor

Reported/Known RC related bugs:

@lidel lidel mentioned this issue Dec 3, 2021
Merged
2 tasks
@guseggert
Copy link
Contributor

Tagged RC2, see #8582

@Xyncgas

This comment has been minimized.

Sign in to view
@guseggert

This comment has been minimized.

Sign in to view
@Luflosi

This comment has been minimized.

Sign in to view
@Arlodotexe Arlodotexe mentioned this issue Dec 15, 2021
Closed
3 tasks
@lidel
Copy link
Member

lidel commented Dec 15, 2021

👉 ipfs-http-client@55.0.0 compatible with dag and pubsub wire format changes shipped today

This was referenced Dec 16, 2021
Merged
Open
Go IPFS Roadmap automation moved this from In Progress to Done Jan 6, 2022
@TheDiscordian TheDiscordian mentioned this issue Jan 26, 2022
Closed
@guseggert guseggert unpinned this issue Feb 28, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@guseggert guseggert

Labels
topic/release Topic release
Projects
IPFS Shipyard Team
Archived in project
Go IPFS Roadmap
  
Done
Milestone
go-ipfs 0.11
Development

Successfully merging a pull request may close this issue.

chore: update deps ipfs/kubo
6 participants
@BigLep @lidel @guseggert @aschmahmann @Luflosi @Xyncgas