Make libp2p-core default features optional in all subcrates #2181

rubdos · 2021-08-06T09:53:41Z

It's impossible to use libp2p as dependency without having libp2p-core with all its default features (example use case: I don't need/want secp256k1). The reason is that libp2p requires (a.o.) libp2p-swarm, which in turn has libp2p-core with default features.

This is probably a backwards incompatible change. I would suggest that libp2p forwards its own default feature to libp2p-*.

Additionally, we probably need to discuss which subcrates require what features from libp2p-core, and how to make sure that CI still covers the common situations in terms of feature flags.

rubdos · 2021-08-06T10:28:43Z

Ater the 888314e fixup, I got rid of secp :-)

thomaseizinger · 2021-08-06T11:14:11Z

Relevant discussion: #2146

Given lack of my own involvement on this front, I am more than happy to support merging a subset of what is discussed in that PR.

Generally, I think we should embrace opt-in more than opt-out. What do things look like if we just remove secp256k1 as a feature from libp2p-core?

rubdos · 2021-08-06T11:51:52Z

Relevant discussion: #2146

Ah, missed that. Seems like this PR then works towards #2173, and that default can soon be set to [] and a full feature flag can be added.

Given lack of my own involvement on this front, I am more than happy to support?merging a subset of what is discussed in that PR.

Generally, I think we should embrace opt-in more than opt-out. What do things look like if we just remove secp256k1 as a feature from libp2p-core?

Removing secp256k1 from core means that the default feature set is incompatible with things like IPFS, that might be not desirable.

Additional to full, you could consider a ipfs feature flag too, pulling in the standard ipfs-features... Just a suggestion though, not a thing that I'd take up.

thomaseizinger · 2021-08-06T23:10:27Z

Good idea, I'll add the proposal to issue!

mxinden

I am in favor of this step towards #2173.

This is probably a backwards incompatible change.

Yes. It would require a changelog entry.

I would suggest that libp2p forwards its own default feature to libp2p-*.

I am not quite sure I follow. Could you expand?

Additionally, we probably need to discuss which subcrates require what features from libp2p-core

Given that secp256k1 is the only feature on libp2p-core there is not much to discuss, no?

and how to make sure that CI still covers the common situations in terms of feature flags.

We have a CI step testing with all features enabled. Given that features should always be designed in an additive way, this should be good enough. Am I missing something?

rubdos · 2021-08-16T12:27:37Z

I am in favor of this step towards #2173.

Does this mean "I am in favor of this PR as a step towards #2173", or "I am in favor of discussing this approach in context of #2173"? Not sure how to read you here.

This is probably a backwards incompatible change.

Yes. It would require a changelog entry.

Ack, will add after resolving the previous point :-)

I would suggest that libp2p forwards its own default feature to libp2p-*.

I am not quite sure I follow. Could you expand?

If the meta crate has it's default enabled, it could foward that to all the subcrates:

[features]
default = [
    "yamux",
    # ...
    "libp2p-core/default",
    "libp2p-identify/default",
    # ...
]

But probably, the intention is that the meta crate works on a higher level than that, so feel free to dismiss that idea.

Additionally, we probably need to discuss which subcrates require what features from libp2p-core

Given that secp256k1 is the only feature on libp2p-core there is not much to discuss, no?

Yes, I went too abstract in my thinking too quickly :-)

and how to make sure that CI still covers the common situations in terms of feature flags.

We have a CI step testing with all features enabled. Given that features should always be designed in an additive way, this should be good enough. Am I missing something?

I would add some CI tests to check that certain features can be turned off, i.e., that they are in fact designed in an additive way.

mxinden · 2021-08-17T08:37:10Z

I am in favor of this step towards #2173.

Does this mean "I am in favor of this PR as a step towards #2173", or "I am in favor of discussing this approach in context of #2173"? Not sure how to read you here.

The former. In other words, I am in favor of both this pull request and #2173 and I think we don't have to block this pull request on #2173 and can thus already merge this pull request.

This is probably a backwards incompatible change.

Yes. It would require a changelog entry.

Ack, will add after resolving the previous point :-)

🙏 small caveat, for each crate touched in this pull request, one entry needs to be added to the corresponding CHANGELOG.md file. Sorry for the inconvenience :/

I would suggest that libp2p forwards its own default feature to libp2p-*.

I am not quite sure I follow. Could you expand?

If the meta crate has it's default enabled, it could foward that to all the subcrates:
[features]
default = [
    "yamux",
    # ...
    "libp2p-core/default",
    "libp2p-identify/default",
    # ...
]
But probably, the intention is that the meta crate works on a higher level than that, so feel free to dismiss that idea.

Ah, thanks for explaining. Yes, that would make sense to me, though I would guess we would want to forward the full feature with #2173 in mind. Either way, I think this should be a separate pull request.

and how to make sure that CI still covers the common situations in terms of feature flags.

We have a CI step testing with all features enabled. Given that features should always be designed in an additive way, this should be good enough. Am I missing something?

I would add some CI tests to check that certain features can be turned off, i.e., that they are in fact designed in an additive way.

Indeed, that would be nice to have. Would you have a test requiring said feature that is then expected to fail?

rubdos · 2021-08-17T09:24:39Z

I am in favor of this step towards #2173.

Does this mean "I am in favor of this PR as a step towards #2173", or "I am in favor of discussing this approach in context of #2173"? Not sure how to read you here.

The former. In other words, I am in favor of both this pull request and #2173 and I think we don't have to block this pull request on #2173 and can thus already merge this pull request.

Ack, I'll finish this up then.

This is probably a backwards incompatible change.

Yes. It would require a changelog entry.

Ack, will add after resolving the previous point :-)

pray small caveat, for each crate touched in this pull request, one entry needs to be added to the corresponding CHANGELOG.md file. Sorry for the inconvenience :/

Don't worry. Vim-fu will help.

I would suggest that libp2p forwards its own default feature to libp2p-*.

I am not quite sure I follow. Could you expand?

If the meta crate has it's default enabled, it could foward that to all the subcrates:
[features]
default = [
    "yamux",
    # ...
    "libp2p-core/default",
    "libp2p-identify/default",
    # ...
]
But probably, the intention is that the meta crate works on a higher level than that, so feel free to dismiss that idea.
Ah, thanks for explaining. Yes, that would make sense to me, though I would guess we would want to forward the full feature with #2173 in mind. Either way, I think this should be a separate pull request.

Makes sense.

and how to make sure that CI still covers the common situations in terms of feature flags.

We have a CI step testing with all features enabled. Given that features should always be designed in an additive way, this should be good enough. Am I missing something?

I would add some CI tests to check that certain features can be turned off, i.e., that they are in fact designed in an additive way.

Indeed, that would be nice to have. Would you have a test requiring said feature that is then expected to fail?

I think that just checking that everything compiles additively should be enough. Tests that require specific features will fail to compile anyway [citation needed]

rubdos · 2021-08-17T10:06:28Z

Hmm, on the other hand, failure testing would be nice to have; as an example: test that secp256p1 doesn't work when the feature is disabled.

Since you already test --no-default-features and --all-features, I'm not entirely sure what other feature sets to test (apart from maybe the default feature set?)

mxinden

Looks good to me. Thank @rubdos for tackling this.

I think this is a good step forward, allowing people to use libp2p without secp256k1. Will leave it open for a bit longer for others to comment as well.

CHANGELOG.md

misc/multistream-select/CHANGELOG.md

Co-authored-by: Max Inden <mail@max-inden.de>

* protocols/gossipsub: Fix inconsistency in mesh peer tracking (libp2p#2189) Co-authored-by: Age Manning <Age@AgeManning.com> * misc/metrics: Add auxiliary crate to record events as OpenMetrics (libp2p#2063) This commit adds an auxiliary crate recording protocol and Swarm events and exposing them as metrics in the OpenMetrics format. * README: Mention security@ipfs.io * examples/: Add file sharing example (libp2p#2186) Basic file sharing application with peers either providing or locating and getting files by name. While obviously showcasing how to build a basic file sharing application, the actual goal of this example is **to show how to integrate rust-libp2p into a larger application**. Architectural properties - Clean clonable async/await interface ([`Client`]) to interact with the network layer. - Single task driving the network layer, no locks required. * examples/README: Give an overview over the many examples (libp2p#2194) * protocols/kad: Enable filtering of (provider) records (libp2p#2163) Introduce `KademliaStoreInserts` option, which allows to filter records. Co-authored-by: Max Inden <mail@max-inden.de> * swarm/src/protocols_handler: Impl ProtocolsHandler on either::Either (libp2p#2192) Implement ProtocolsHandler on either::Either representing either of two ProtocolsHandler implementations. Co-authored-by: Thomas Eizinger <thomas@eizinger.io> * *: Make libp2p-core default features optional (libp2p#2181) Co-authored-by: Max Inden <mail@max-inden.de> * core/: Remove DisconnectedPeer::set_connected and Pool::add (libp2p#2195) This logic seems to be a leftover of libp2p#889 and unused today. * protocols/gossipsub: Use ed25519 in tests (libp2p#2197) With f2905c0 the secp256k1 feature is disabled by default. Instead of enabling it in the dev-dependency, simply use ed25519. * build(deps): Update minicbor requirement from 0.10 to 0.11 (libp2p#2200) Updates the requirements on [minicbor](https://gitlab.com/twittner/minicbor) to permit the latest version. - [Release notes](https://gitlab.com/twittner/minicbor/tags) - [Changelog](https://gitlab.com/twittner/minicbor/blob/master/CHANGELOG.md) - [Commits](https://gitlab.com/twittner/minicbor/compare/minicbor-v0.10.0...minicbor-v0.11.0) --- updated-dependencies: - dependency-name: minicbor dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * build(deps): Update salsa20 requirement from 0.8 to 0.9 (libp2p#2206) * build(deps): Update salsa20 requirement from 0.8 to 0.9 Updates the requirements on [salsa20](https://github.com/RustCrypto/stream-ciphers) to permit the latest version. - [Release notes](https://github.com/RustCrypto/stream-ciphers/releases) - [Commits](RustCrypto/stream-ciphers@ctr-v0.8.0...salsa20-v0.9.0) --- updated-dependencies: - dependency-name: salsa20 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * *: Bump pnet to v0.22 Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Max Inden <mail@max-inden.de> * *: Dial with handler and return handler on error and closed (libp2p#2191) Require `NetworkBehaviourAction::{DialPeer,DialAddress}` to contain a `ProtocolsHandler`. This allows a behaviour to attach custom state to its handler. The behaviour would no longer need to track this state separately during connection establishment, thus reducing state required in a behaviour. E.g. in the case of `libp2p-kad` the behaviour can include a `GetRecord` request in its handler, or e.g. in the case of `libp2p-request-response` the behaviour can include the first request in the handler. Return `ProtocolsHandler` on connection error and close. This allows a behaviour to extract its custom state previously included in the handler on connection failure and connection closing. E.g. in the case of `libp2p-kad` the behaviour could extract the attached `GetRecord` from the handler of the failed connection and then start another connection attempt with a new handler with the same `GetRecord` or bubble up an error to the user. Co-authored-by: Thomas Eizinger <thomas@eizinger.io> * core/: Remove deprecated read/write functions (libp2p#2213) Co-authored-by: Max Inden <mail@max-inden.de> * protocols/ping: Revise naming of symbols (libp2p#2215) Co-authored-by: Max Inden <mail@max-inden.de> * protocols/rendezvous: Implement protocol (libp2p#2107) Implement the libp2p rendezvous protocol. > A lightweight mechanism for generalized peer discovery. It can be used for bootstrap purposes, real time peer discovery, application specific routing, and so on. Co-authored-by: rishflab <rishflab@hotmail.com> Co-authored-by: Daniel Karzel <daniel@comit.network> * core/src/network/event.rs: Fix typo (libp2p#2218) * protocols/mdns: Do not fire all timers at the same time. (libp2p#2212) Co-authored-by: Max Inden <mail@max-inden.de> * misc/metrics/src/kad: Set query_duration lowest bucket to 0.1 sec (libp2p#2219) Probability for a Kademlia query to return in less than 100 milliseconds is low, thus increasing the lower bucket to improve accuracy within the higher ranges. * misc/metrics/src/swarm: Expose role on connections_closed (libp2p#2220) Expose whether closed connection was a Dialer or Listener. * .github/workflows/ci.yml: Use clang 11 (libp2p#2233) * protocols/rendezvous: Update prost (libp2p#2226) Co-authored-by: Max Inden <mail@max-inden.de> * *: Fix clippy warnings (libp2p#2227) * swarm-derive/: Make event_process = false the default (libp2p#2214) Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Max Inden <mail@max-inden.de> Co-authored-by: Age Manning <Age@AgeManning.com> Co-authored-by: Ruben De Smet <ruben.de.smet@rubdos.be> Co-authored-by: Thomas Eizinger <thomas@eizinger.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: rishflab <rishflab@hotmail.com> Co-authored-by: Daniel Karzel <daniel@comit.network> Co-authored-by: David Craven <david@craven.ch>

rubdos added 2 commits August 6, 2021 11:52

Make libp2p-core default features optional in all subcrates

2f01a5c

fixup! Make libp2p-core default features optional in all subcrates

888314e

mxinden reviewed Aug 10, 2021

View reviewed changes

rubdos added 2 commits August 17, 2021 12:01

Add changelog entries for libp2p#2181

7b09173

Add changelog entry in facade for libp2p#2181

5ae48a9

Merge remote-tracking branch 'origin/master' into swarm-optional

260d069

mxinden approved these changes Aug 17, 2021

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

misc/multistream-select/CHANGELOG.md Outdated Show resolved Hide resolved

rubdos and others added 3 commits August 18, 2021 09:47

Update CHANGELOG.md

2da724e

Co-authored-by: Max Inden <mail@max-inden.de>

misc/multistream-select/CHANGELOG.md: Remove entry

eb2b27c

Merge branch 'master' into swarm-optional

e32d190

mxinden merged commit f2905c0 into libp2p:master Aug 18, 2021

rubdos deleted the swarm-optional branch August 18, 2021 14:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make libp2p-core default features optional in all subcrates #2181

Make libp2p-core default features optional in all subcrates #2181

rubdos commented Aug 6, 2021 •

edited

rubdos commented Aug 6, 2021

thomaseizinger commented Aug 6, 2021 •

edited

rubdos commented Aug 6, 2021

thomaseizinger commented Aug 6, 2021

mxinden left a comment

rubdos commented Aug 16, 2021

mxinden commented Aug 17, 2021

rubdos commented Aug 17, 2021

rubdos commented Aug 17, 2021

mxinden left a comment

Make libp2p-core default features optional in all subcrates #2181

Make libp2p-core default features optional in all subcrates #2181

Conversation

rubdos commented Aug 6, 2021 • edited

rubdos commented Aug 6, 2021

thomaseizinger commented Aug 6, 2021 • edited

rubdos commented Aug 6, 2021

thomaseizinger commented Aug 6, 2021

mxinden left a comment

Choose a reason for hiding this comment

rubdos commented Aug 16, 2021

mxinden commented Aug 17, 2021

rubdos commented Aug 17, 2021

rubdos commented Aug 17, 2021

mxinden left a comment

Choose a reason for hiding this comment

rubdos commented Aug 6, 2021 •

edited

thomaseizinger commented Aug 6, 2021 •

edited