Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release v0.9.0 #8058

Closed
63 of 71 tasks
aschmahmann opened this issue Apr 7, 2021 · 23 comments
Closed
63 of 71 tasks

Release v0.9.0 #8058

aschmahmann opened this issue Apr 7, 2021 · 23 comments
Assignees
@aschmahmann
Projects
Maintenance Priorities - Go
Milestone
go-ipfs 0.9

Comments

@aschmahmann
Copy link
Contributor

aschmahmann commented Apr 7, 2021 •
edited

go-ipfs 0.9.0 Release

We're happy to announce go-ipfs 0.9.0. This release makes go-ipfs even more configurable with some fun experiments to boot. We're also deprecating or removing some uncommonly used features to make it easier for users to discover the easy ways to use go-ipfs safely and efficiently.

As usual, this release includes important fixes, some of which may be critical for security. Unless the fix addresses a bug being exploited in the wild, the fix will not be called out in the release notes. Please make sure to update ASAP. See our release process for details.

🗺 What's left for release

🚢 Estimated shipping date

  • Start release process 2021-04-14 2021-04-22 2021-05-04 2021-05-11
  • Early testing begins 2021-04-21 2021-04-28 2021-05-11 2021-05-19
  • Public/full release occurs 2021-04-28 2021-05-05 2021-05-18 2021-05-26 202106

🔦 Highlights

Highlights

📦 Exporting of DAGs via Gateways

Gateways now support downloading arbitrary IPLD graphs via the /api/v0/dag/export endpoint. This endpoint works in the same way as the ipfs dag export command.

One major thing this enables is ability to verify data downloaded from public gateways. If you go to https://somegateway.example.net/ipfs/bafyexample you are using the old school HTTP transport, and trusting that the gateway is being well behaved. However, if you download the graph as a DAG archive then it is possible to verify that the data you downloaded does in fact match bafyexample.

Additionally, it was previously quite painful to download things other than UnixFS (files + directories) using gateways. It is now possible to download arbitrary IPLD graphs from gateways, making them useful as a general-purpose alternative to p2p transports.

This opens exciting opportunities in areas like thin clients, mobile browsers and IoT devices, which now can delegate IPFS resolution to any public gateway, and have ability to verify that the data received matches the requested hash.

☁ Custom DNS Resolvers

Resolution of DNS records for DNSLink and DNSAddrs means that names are sent in cleartext between the operating system and the DNS server provided by an ISP. In the past, the only way to customize DNS resolution in IPFS stack was to set up own DNS proxy server.

There is now the ability to customize DNS resolution and override the default resolver from the OS with DNS over HTTPS (DoH) one. We made it really flexible: override can be applied globally, or per specific TLD/FQDN. Examples can be found in the documentation.

👪 Support for non-ICANN DNSLink names

Building off of the support for custom DNS resolvers it is now possible to create DNSLink names not handled by ICANN and choose how that domain name will be resolved. An example of this is how ENS is supported, despite .eth not being an ICANN TLD you can point .eth to any ENS resolver you want (including a local one).

While go-ipfs may have some DoH defaults for a few popular non-ICANN DNSLink names (e.g. ENS), you are free to use any protocol for a naming system and as long as it exposes a DNSLink record via a DNS endpoint you can make it work.

🖥️ Updated to the latest WebUI

Our web interface now includes experimental support for pinning services, and various updates to Files and Peers screens.

Remote pinning services added via the ipfs pin remote service add command are already detected, one can also add one from Settings screen, and it will appear in Set pinning interface on the Files screen.

Data presented on the Peers screen can now be copied by simply clicking on a specific cell, and a list of open streams gives better insight into how a local node interacts with a specific peer.

See release notes for ipfs-webui v2.12 for screenshots and more details.

🔑 IPNS keys can now be exported via the CLI without stopping the daemon

ipfs key export no longer requires interrupting ipfs daemon ✨

🕸 Experimental DHT Client and Provider System

An area of go-ipfs that has been historically tricky is how go-ipfs finds who has the data they are looking for. While the IPFS Public DHT is only one of the ways go-ipfs can find data it tends to be an important one. While since go-ipfs v0.5.0 the time to find content in the network has dropped significantly the time to put/get IPNS records or for a node to advertise the content it has still has much room for improvement.

We have been doing some experimenting and have an alternative DHT client that essentially trades off some resources and in return is much more performant. We have also included with the experimental DHT client a bulk provider system that takes advantage of the new client to more efficiently do many advertisements at a time

This work is quite new and still under development, however, the results so far have been promising especially for users with lots of data who have otherwise been having difficulty advertising their data into the IPFS Public DHT

As described in the experimental features documentation the experimental client can be enabled using the command below (or modifying the config file).

ipfs config --json Experimental.AcceleratedDHTClient true

A few things to take note of when AcceleratedDHTClient is enabled:

  • go-ipfs will likely use more resources then previously
  • DHT queries will not be usable (i.e. finding which peers have some data, finding where a particular peer is, etc.) for the first 5-10 minutes of operation depending on your network conditions
  • There is an ipfs stats provide command that will help you track your provide/reprovide usage, if you are providing lots of data you may want to consider how to reduce the amount you are providing (e.g. Reprovider Strategies and/or Strategic Providing)

See the documentation for more details.

🚶‍♀️ Migrations

Migrations are now individually packaged

While previously the go-ipfs repo migration binary was monolithic and contained all migrations from previous go-ipfs versions the binaries are now packaged individually. However, the fs-repo-migrations binary is still there to help those who manually upgrade their repos to download all the individual migrations.

This means faster download times for upgrades, a much easier time building migrations for those who make use of custom plugins, and an easier time developing new migrations going forward.

Configurable migration downloads enable downloading over IPFS

Previously the migration downloader built into go-ipfs downloaded the migrations from dist.ipfs.io. While users could use tools like ipfs-update to download the migrations over IPFS or manually download the migrations (over IPFS or otherwise) themselves, this is now automated and configurable. Users can choose to download the migrations over IPFS or from any specified IPFS Gateway.

The configurable migration options are described in the config file documentation, although most users should not need to change the default settings.

The main benefit here is that users behind restrictive firewalls, or in offline/private deployments, won't have to run migrations manually, which is especially important for desktop use cases where go-ipfs is running inside of IPFS Desktop and Brave.

🍎 Published builds for Apple M1 hardware

Go now supports building for Darwin ARM64 and we are now publishing those builds

👋 Deprecations and Feature Removals

The ipfs object commands are now deprecated

In the last couple years most of the Object API's commands have become fulfillable using alternative APIs.

The utility of Object API's is limited to data in UnixFS-v1 (dag-pb) format. If you are still using it, it is highly recommended that you switch to the DAG ipfs dag (supports modern data types like dag-cbor) or Files ipfs files (more intuitive for working with dag-pb) APIs.

While the Object API and commands are still usable they are now marked as deprecated and hidden from users on the command line to discourage further use. We also updated their --help text to point at the modern replacements.

X-Ipfs-Gateway-Prefix is now deprecated

IPFS community moved towards dedicated Origins (DNSLink and subdomain gateways) which are much easier to isolate and reason about.

Setting up Gateway.PathPrefixes and X-Ipfs-Gateway-Prefix is no longer necessary and support will be removed in near future.

Proquints support removed

A little known feature that was not well used or documented and was more well known for the error message Error: not a valid proquint string users received when trying to download invalid IPNS or DNSLink names (e.g. https://dweb.link/ipns/badname). We have removed support for proquints as they were out of place and largely unused, however proquints are valid multibases so if there is renewed interest in them there is a way forward.

SECIO support removed

SECIO was deprecated and turned off by default given the prevalence of TLS and Noise support, SECIO support is now removed entirely.

Changelog

Full Changelog

✅ Release Checklist

For each RC published in each stage:

  • version string in version.go has been updated (in the release-vX.Y.Z branch).
  • tag commit with vX.Y.Z-rcN
  • upload to dist.ipfs.io
    1. Build: https://github.com/ipfs/distributions#usage.
    2. Pin the resulting release.
    3. Make a PR against ipfs/distributions with the updated versions, including the new hash in the PR comment.
    4. Ask the infra team to update the DNSLink record for dist.ipfs.io to point to the new distribution.
  • cut a pre-release on github and upload the result of the ipfs/distributions build in the previous step.
  • Announce the RC:

Checklist:

  • Stage 0 - Automated Testing
    • Fork a new branch (release-vX.Y.Z) from master and make any further release related changes to this branch. If any "non-trivial" changes (see the footnotes of docs/releases.md for a definition) get added to the release, uncheck all the checkboxes and return to this stage.
      • Follow the RC release process to cut the first RC.
      • Bump the version in version.go in the master branch to vX.(Y+1).0-dev.
    • Automated Testing (already tested in CI) - Ensure that all tests are passing, this includes:
  • Stage 1 - Internal Testing
    • CHANGELOG.md has been updated
    • Network Testing:
      • test lab things - TBD
    • Infrastructure Testing:
      • Deploy new version to a subset of Bootstrappers
      • Deploy new version to a subset of Gateways
      • Deploy new version to a subset of Preload nodes
      • Collect metrics every day. Work with the Infrastructure team to learn of any hiccup
    • IPFS Application Testing - Run the tests of the following applications:
  • Stage 2 - Community Dev Testing
    • Reach out to the IPFS early testers listed in docs/EARLY_TESTERS.md for testing this release (check when no more problems have been reported). If you'd like to be added to this list, please file a PR.
    • Reach out to on IRC for beta testers.
    • Run tests available in the following repos with the latest beta (check when all tests pass):
  • Stage 3 - Community Prod Testing
  • Stage 4 - Release
    • Final preparation
      • Verify that version string in version.go has been updated.
      • Merge release-vX.Y.Z into the release branch.
      • Tag this merge commit (on the release branch) with vX.Y.Z.
      • Release published
      • Cut a new ipfs-desktop release
    • Publish a Release Blog post (at minimum, a c&p of this release issue with all the highlights, API changes, link to changelog and thank yous)
    • Broadcasting (link to blog post)
  • x] Post-Release
    • Merge the release branch back into master, ignoring the changes to version.go (keep the -dev version from master).
    • Create an issue using this release issue template for the next release.
    • Make sure any last-minute changelog updates from the blog post make it back into the CHANGELOG.

❤️ Contributors

Contributors

Contributor Commits Lines ± Files Changed
Marten Seemann 320 +16488/-11587 1188
Eric Myhre 82 +9672/-2459 328
Daniel Martí 18 +2733/-4377 313
Adin Schmahmann 61 +4690/-1483 136
hannahhoward 14 +1380/-3667 84
Steven Allen 76 +3060/-1572 152
Hector Sanjuan 12 +511/-3129 52
gammazero 28 +2462/-1129 87
vyzo 76 +2188/-940 116
Will Scott 12 +912/-593 37
Dirk McCormick 3 +1384/-63 14
Andrew Gillis 3 +1231/-39 19
web3-bot 34 +734/-256 44
Marcin Rataj 30 +479/-292 65
Aarsh Shah 13 +668/-86 30
Olivier Poitrey 1 +469/-182 15
whyrusleeping 5 +253/-32 11
Adrian Lanzafame 8 +212/-60 11
Cory Schwartz 9 +154/-103 32
aarshkshah1992 6 +93/-109 8
huoju 4 +127/-41 6
ipldbot 8 +119/-7 8
Jakub Sztandera 4 +70/-39 8
Rod Vagg 2 +73/-1 3
acruikshank 6 +32/-24 7
Toby 1 +41/-1 2
Naveen 1 +40/-0 1
Kévin Dunglas 1 +32/-2 2
Nicholas Bollweg 1 +22/-0 1
q191201771 2 +4/-11 2
Mathis Engelbart 1 +12/-2 1
requilence 1 +13/-0 1
divingpetrel 1 +7/-4 2
Oli Evans 2 +9/-2 3
dependabot-preview[bot] 1 +8/-0 1
RubenKelevra 3 +2/-6 3
Will 1 +1/-5 1
Jorropo 1 +4/-2 1
Lucas Molas 2 +2/-2 2
Ju Huo 1 +2/-2 1
Luflosi 1 +1/-1 1
Jonathan Rudenberg 1 +1/-1 1
David Pflug 1 +1/-1 1
Ari Mattila 1 +1/-1 1
Yingrong Zhao 1 +0/-1 1

Would you like to contribute to the IPFS project and don't know how? Well, there are a few places you can get started:

⁉️ Do you have questions?

The best place to ask your questions about IPFS, how it works and what you can do with it is at discuss.ipfs.io. We are also available at the #ipfs channel on Freenode, which is also accessible through our Matrix bridge.
@aschmahmann aschmahmann added kind/enhancement A net-new feature or improvement to an existing feature and removed kind/enhancement A net-new feature or improvement to an existing feature labels Apr 7, 2021
@aschmahmann aschmahmann added this to the go-ipfs 0.9 milestone Apr 7, 2021
@BigLep
Copy link
Contributor

BigLep commented Apr 7, 2021

Some other FAQs about this release:

Why aren't we proposing to start the release sooner?

We're waiting to start the release process until the "What's left for release" items land because:

  1. It helps this release have more substantive and user-benefiting functionality than the set of bug fixes it would otherwise have. The other major improvement in this release is around repo migrations. While this is important architecturally, most users will not see its effects. The remaining work to add "Experimental DHT client" and "Providing in bulk when using experimental DHT client" will enable large writers to the DHT to see orders of magnitude better performance. We believe it's advantageous to such users to start experimenting with this feature, and we can use their feedback to ensure it's worth investing in finishing off for general consumption.
  2. Releases do take manual time on the order of days. We want to change this in 2021 (goal TBD), but we are mindful about how frequently we do releases until we automate this more. To baseline for this goal, we will record the specific amount of manual time taken to do this specific release.

What are our options for releasing more quickly?

  1. We could forgo the items in "What's left for release" and start the release process now. This would likely pull in the release date by a week.
  2. We could also forgo the "early tester" phases, also saving a ~week.

@RubenKelevra
Copy link
Contributor

A question about the plans after this release. There's an extensive list of roadmap submissions and haven't heard anything about them. Is there still a plan to give some of them priorities and publish them?

//cc @vorburger @JonasKruckenberg @Gozala @realChainLife @bertrandfalguiere @RhettSampson @Danger89 @rklaehn @obo20 @MichaelMure @autonome @atopal @dchoi27 @NatoBoram @momack2

@bertrandfalguiere
Copy link

@RubenKelevra

I asked on Discourse and @momack2 wrote that a blog post was incoming. In her answer, it seems that they are multitasking and that they won't follow only one or two objectives IIUC.
See discourse: https://discuss.ipfs.io/t/2021-roadmap-for-ipfs-is-this-still-a-thing/10534/5

@RubenKelevra
Copy link
Contributor

RubenKelevra commented Apr 10, 2021 •
edited

Oh, didn't found this post.

Looks like they're just a bit overwhelmed with the amount of proposals, which is clearly a good thing 😬

Was just hoping to hop onto some smaller stuff and have a look for my first contributions. :)

@aschmahmann aschmahmann pinned this issue Apr 12, 2021
@BigLep BigLep unpinned this issue Apr 16, 2021
@BigLep BigLep pinned this issue Apr 16, 2021
@BigLep
Copy link
Contributor

BigLep commented Apr 16, 2021

@RubenKelevra

Wad just hoping to hop onto some smaller stuff and have a look for my first contributions. :)

Did you find something? This likely needs to get better written up somewhere, but some links for getting started on finding issues:

  1. https://github.com/ipfs/community/blob/master/CONTRIBUTING_GO.md
  2. https://github.com/ipfs/go-ipfs/labels/exp%2Fbeginner
  3. https://github.com/ipfs/go-ipfs/labels/exp%2Fnovice
  4. https://github.com/ipfs/go-ipfs/labels/good%20first%20issue
  5. https://github.com/ipfs/go-ipfs/labels/help%20wanted

Thanks,
Steve

@BigLep
Copy link
Contributor

BigLep commented Apr 16, 2021

A quick update on dates: Our dates have slipped by a week from what was originally planned. The dates in the Issue description have been updated. This is in part due to getting additional functionality in (e.g., repo integrations over IPFS, configurable DNS servers) and also due to prioritizing reviews/input on other projects that will land in 0.10.

@RubenKelevra
Copy link
Contributor

@BigLep thanks for the links, currently still reading code and getting used to think in go not in shellscript or python.

But soon they'll be helpful.

Was just looking for some general direction for this year, to maybe do some small stuff in the right direction :)

@BigLep
Copy link
Contributor

BigLep commented May 6, 2021

For those watching this issue, we are still actively working on this release. The issue description has been updated with the latest dates.

@Stebalien Stebalien mentioned this issue May 7, 2021
Closed
@BigLep BigLep added this to In Progress in Maintenance Priorities - Go May 7, 2021
@lidel lidel mentioned this issue May 11, 2021
Open
@BigLep
Copy link
Contributor

BigLep commented May 13, 2021

@lidel : per 2021-05-13 verbal, can you please:

  1. create the playbook for the Brave testing
  2. link to it from this release
  3. link to it from the release template

@ianopolous
Copy link
Member

ianopolous commented May 14, 2021 •
edited

It might also be worth mentioning the block api as a potential upgrade from the object api. We used to use the object api extensively in Peergos originally, and tried upgrading to the dag api 5 years ago, but it had (has?) some issues [1] making it unsuitable. But the block api works perfectly.

[1] #3503

@BigLep
Copy link
Contributor

BigLep commented May 14, 2021

For those watching this issue, @aschmahmann has been working through the checklist and we have our first rc: https://dist.ipfs.io/go-ipfs/v0.9.0-rc1

@BigLep BigLep mentioned this issue May 17, 2021
Merged
This was referenced May 18, 2021
Closed
Merged
@lidel
Copy link
Member

lidel commented May 19, 2021 •
edited

PSA: in case you are using go-ipfs as a dependency in a JS project (in E2E tests, or bundling inside of Electron app), the RC1 binary is available on NPM as go-ipfs@0.9.0-rc1

@lidel lidel mentioned this issue May 21, 2021
Merged
@BigLep BigLep mentioned this issue May 27, 2021
Closed
@aschmahmann
Copy link
Contributor Author

aschmahmann commented May 28, 2021 •
edited

Hey all, v0.9.0-rc2 is out

Changes since rc1:

@aschmahmann
Copy link
Contributor Author

Heads up to early testers:

pacman.store (@RubenKelevra)
Infura (@MichaelMure)
Textile (@sanderpick)
Pinata (@obo20)
RTrade (@postables)
QRI (@b5)
Siderus (@koalalorenzo)
Charity Engine (@Rytiss, @tristanolive)
Fission (@bmann)

If you'd like to be added/removed from this list send a PR modifying https://github.com/ipfs/go-ipfs/blob/master/docs/EARLY_TESTERS.md

@RubenKelevra
Copy link
Contributor

@aschmahmann thanks! Will change push this today to my nodes and report any issues back!

@BigLep BigLep mentioned this issue Jun 1, 2021
Merged
kallisti5 referenced this issue in haiku/website Jun 8, 2021
@kallisti5
blog: Add IPFS experiment post
9dd91bd
@BigLep
Copy link
Contributor

BigLep commented Jun 14, 2021

For those watching this issue, we are still actively working on getting this release out the door. We'll provide a status update by EOD 2021-06-16.

@andrew andrew mentioned this issue Jun 15, 2021
Closed
@BigLep
Copy link
Contributor

BigLep commented Jun 17, 2021

Latest update: We are still active here finding/fixing blocking bugs before the release. We'll provide a status update by EOD 2021-06-18.

@ianopolous
Copy link
Member

Is there any possibility this fix could be included in the release?
ipfs/go-ipfs-blockstore#77

This would be earth shatteringly awesome for us and solve a host of issues we've been having.

@aschmahmann
Copy link
Contributor Author

aschmahmann commented Jun 17, 2021 •
edited

@ianopolous thanks for PRing the fix in ipfs/go-ipfs-blockstore#78. Does it resolve your issues?

@ianopolous
Copy link
Member

ianopolous commented Jun 17, 2021 •
edited

Thank you for your help @aschmahmann It sure looks like it! Average bandwidth to S3 has dropped from 5Mb/s to <1kb/s! That's 5000X less bandwidth to S3 used on average.
5mb-to-1kb

@aschmahmann aschmahmann mentioned this issue Jun 18, 2021
Closed
@BigLep
Copy link
Contributor

BigLep commented Jun 18, 2021

Latest update: We are still finishing off the last blocking bugs before the release. Barring no new blockers identified, we will release by 2021-06-25. We will next communicate by EOD 2021-06-22.

@aschmahmann
Copy link
Contributor Author

The release is out and published on dist and GitHub. There's still some broadcasting and package manager post-release work to take care of.

@Stebalien
Copy link
Member

Looks like everything is done (I posted to reddit and pinned the forum post).

Maintenance Priorities - Go automation moved this from In Progress to Done Jun 30, 2021
@Stebalien Stebalien unpinned this issue Jun 30, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@aschmahmann aschmahmann

Labels
None yet
Projects
No open projects
Maintenance Priorities - Go
Done
Milestone
go-ipfs 0.9
Development

No branches or pull requests
7 participants
@BigLep @lidel @ianopolous @Stebalien @RubenKelevra @aschmahmann @bertrandfalguiere