EPIC: interop testing for go and rust libp2p - Ping Test

[laurentsenta]

We want to run a "simple" test (ping) that verifies interoperability between versions and implementations.
This test should run at least before every (go|rust) libp2p release.

Use case:

• Test cross-version compatibility for Go
• Test cross-version compatibility for Rust
• Test cross-implementation compatibility for Rust and Go

We're aiming for:

• a single ping test implemented in different languages (go + rust)

• running interop + cross version for every PR on the go-libp2p repo

• running interop + cross version for every PR on the rust-libp2p repo

• @mxinden and @marten-seemann agreed to run this test if

  • it takes < 30 minutes
  • it's stable (aiming for < 1% failure rate)

• they want "reasonable" run time in the future

  • my goal right now is < 15 minutes which is the go test runtime on ubuntu,
  • I'm trying to push it down to ~ 5 minutes with caching. There is a O(n) relationship between build time & the number of tested versions and caching would make it O(1).
  • we're going to timebox this effort for a week, then discuss how far we want to go.

Todo

• implement cross-version interop testing in go
  • Introduce features required to implement interop testing in Testground testground/testground#1340
  • testground: cross-version interoperability #23
• implement cross-version interop testing in rust
  • ping/rust: introduce rust cross-version test #26
• implement cross-implementation interop testing
  • ping/rust: introduce rust cross-version test #26
• maintain & enables tests in go and rust repos
  • .github: introduce interop tests rust-libp2p#2835
  • ci: add cross-version interop go-libp2p#1725
• stability & perf improvements
  • .github/actions/setup-testground: shortcircuit testground setup #34
  • fix: testground instability due to race condition on outcomes testground/testground#1407
  • EPIC: Stability Dashboard to identify regression & discuss improvement with Maintainers and Users testground/testground#1401
  • feat: add retries & update ping/go interop test #32
  • Use CI Caching during Interop Tests #31
• Sync with the libp2p/interop repo https://github.com/libp2p/interop/issues

References:

• https://www.notion.so/pl-strflt/Roadmap-Draft-c6b7a9e67b18468bac16cccb9a8bf8fe
• https://www.notion.so/pl-strflt/Testground-Interop-Features-Plan-with-Stability-in-Mind-dad6b6d25b79451ba01e0a28da1888b3

[laurentsenta]

Gathering content in a single issue to share w/Bloxico.

[laurentsenta]

Follow-Up discussion

• Can we get rid of the building step when we use a test from another implementation:
  • We need to make sure an update in a repo doesn't break another, it's very important for stability during do cross-implementation interop tests. Fixes like the one in ping/rust: remove the master dependency #45 might be working, but it's hard to make sure we're covering all cases where a build might fail.
  • We don't want to "pin forever" the test artifacts either because a test specification might change (for example, how we sync instances).
  • What if we built every artifact for every known version on merges to libp2p/test-plans?
    • Then reuse these artifacts during test runs?
    • This way implementation langX-libp2p never attempt to build a test for langY-libp2p.
• Another important point is how to deal with new feature:
  • "as a langX-libp2p maintainer, I want to create a test in this plan than will be used by langY-libp2p"
    • It might be a feature that only exists on master,
    • It might be a feature that doesn't exist in previous versions (we're only testing ping at the moment).