Create a list of test scenarios for releases

[p-offtermatt]

Summary

The following suggestion came up:

it would be cool if every release of ICS has a list of tests done for each feature, e.g.,
Key assignment
Test1: Scenario - Passed
Test2: Scenario - Passed
….
Jail throttling
….
…
We could link to this in the release notes.

We should find a way to do this - optimally without burdening engineers without extra manual effort.

Problem Definition

To give everyone an insight into what exactly is being tested for each release,
thus to be able to more easily identify where things might need more testing,
we should have a human-readable list of test scenarios for each release.

The solution should satisfy things along these lines:

• For each (major) feature, we get a list of test scenarios.
• The test scenarios are linked to the successful CI run that ran them
• It needs little manual effort, e.g. the list could be auto-populated from the changelog

Proposal

WIP

Next steps

I will explore some ways how we could do this and edit the issue as I go.
Eventually, the issue should either be closed (if we find that no solution for this fits our needs)
or it should present a clear proposed solution and, depending on scope, the issue might become an epic with subtasks.