error-stack test feature distribution

[indietyp]

(I don't have any other channel to contact, so I guess the discussion forum is the best forum)

I thought about the different features with glyph (and then in a couple of days™ - serde), our current test permutations are exploding.

A powerset (without any additional --optional-deps) can be expressed as $2^n$, where n is our amount of features. Currently, #794 has seven features (excluding the default feature), totaling 128 different variations!

This means that lint are about $384$ ($128 * 3$) invocations (format, clippy, doc) and $384$ for testing (I am unsure as to why it multiplies by 3)

Grouping features together has an exponential gain in test complexity and time.

After some deliberation, this is the preliminary list I came up with:

• anyhow, eyre - they can be grouped because they are both a compatibility layer and completely independent from each other
• experimental - this enables experimental features, therefore cannot be grouped
• glyph, serde - both do formatting that is entirely separate from each other
• std, hooks - hooks is dependent on std and "only" enables additional functionality
• spantrace, futures, small - separate from each other and do not influence each other. small is included here because it changes from Vec to SmallVec and does not change any other behavior.

Doing this preliminary grouping, we would only have $2^5$, which results in "only" $32$ elements.

@TimDiekmann, what do you think? Is there a better solution? A flaw?

[TimDiekmann]

The discussions forum are fine and a good way!

Your powerset calculation is not correct. Some features implies other features (like hooks -> std). cargo-hack is smart enough to detect that and only makes two runs (1. std, 2. hooks, not 3. std + hooks), this reduced the number of powersets. I'm not a mathematician but I guess this is where the x3 comes from 🙃

The groups seem fine to me, and 32 elements seem like a good compromise between test size and coverage (for normal testing).

As @nonparibus proposed, we could still do the full powerset on publish-builds, I will take this into account just before we release a new version.

Thank you for doing the research and pointing it out! 🎉

[indietyp]

Yea @nonparibus proposal is excellent, and having a lengthy test time only for each release is the best compromise.

[indietyp]

Correction: After digging (compiling cargo-hack locally and printing all permutations generated).

This is my updated list of thing, the amount of items in the powerset is actually: 9 and consists of {'hooks', 'experimental', 'futures', 'glyph', 'spantrace', 'eyre', 'anyhow', 'std', 'small'}. I forgot about eyre and anyhow as they are not explicitly stated in the [features] section. $2^9 = 512$, when then cargo-hack boils down to 384. This makes so much more sense.

Test conditions (have been normalized):

1. after cargo clean
2. 8.6s build
3. 1s test duration
4. the increased time between build and execution of tests that have been observed (nightly regression?) has been omitted
   ~> 10s per test

Correction:

• Features (including serde): $10$
• Worst case permutations: $2^{10} = 1024$
• Ideal time taken: $8.6\text{s} * 1024 = 2.4\text{h}$. Considering my previous attempts, this seems very unlikely
• Projected time taken: $\frac{5\text{h}}{348} * 1000 = 13\text{h}$, taking into consideration my previous attempts

Grouping:

• Groups (including serde): $5$
• Worst case permutations: $2^5 = 32$
• Ideal time taken: $8.6\text{s} * 32 = 4.5\text{m}$
• Projected time taken: $\frac{5\text{h}}{348} * 32 = 27\text{m}$
• Projected¹ (adjusted) time taken: $\frac{2\text{m}44\text{s}}{23} * 32 = 3\text{m}50\text{s}$

Sidenote: This is what I meant with the brutal slow down over time with that many permutations running. The time between building and execution gets longer and longer (I haven't been able to track down why), which means that more features are likely to take exponentially more time. This could be a nightly regression or just an oddity of macOS. This is unlikely, considering that the CI of #794 should have taken $\frac{2\text{m}44\text{s}}{23} * {384} = 45\text{m}$, but crashed after having not enough space at ~1.3h on both stable and nightly. This is likely a bug of rust, cargo, or nextest when building this many times.

Footnotes

1. This measure instead takes the time it takes for cargo make insta-test (new command to generate all snapshots) to finish, which has 23 permutations it goes through to create all possible combinations of snapshots. ↩