Collection exclusion criteria and procedure

[Andersson007]

Summary

Issue:

Should we preemptively develop collection exclusion criteria and procedure?

This idea came to my mind when we discussed inclusion of equinix and inflobox collections.
This is still hypothetical but as we actively include collections, especially third-party ones, to the Ansible package, once it'll 99.9% happen that some of them will get unmaintained / broken / not complying to the requirements / unresponsive.

Moreover, if we can change things ourselves in repos under the ansible-collections GitHub organization, e.g. fix CI, review and merge PRs, release, find new maintainers and grant commit to them, maintain ourselves, etc., we can't do the same for third-party collection because, obviously, we don't have any permission there.

If we don't want Ansible ships a bunch of such stuff, we should develop the exclusion criteria and procedure.

Goals:

• Have them somewhere ready when needed.
• Motivate collection maintainers (especially third-party ones) to make their collections complying the collection requirements.

Solution:

• We could create a separate document under https://github.com/ansible-collections/overview
• Link to it from the collection requirements and ansible-inclusion README and from docs.ansible.com.

Initial view how it can basically look like:

1. Someone detects that the collection does not comply the collection requirements (we should go through third-party included collections from time to time to see how things are going there). If rough violations are found (I wouldn't define a precise list as it's hard to predict all possible cases and we should use our judgment and vote. If, say, a collection has a combination of a lot of issues with no response, broken CI, many not reviewed PRs OR, for example, locked issue tracker / its repo is archived, these are reasons to start the process).
2. An issue is raised in the collection repo pointing out the problem spots. If no response / no actions done, say, withing a month:
3. A dedicated issue is raised (say, in the ansible-inclusion repo) pointing out the problem spots and pinging people who submitted the collection and committers spotted in the collection repository.
4. Inform companies / individual maintainers by email if possible. If there's no reaction within a month since the dedicated issue is opened:
5. A related topic is raised in community-topics
6. Discussion, voting for / against exclusion of the collection in the next the Ansible package major release / the major release after the next one.
7. Announcement.
8. Exclusion.

Criteria (see step 1) will be abstract "Violations of the collection requirements". Each case should be considered individually.

The process can be suspended on any stage before exclusion if we see that the active work to make the collection complying is going.

Maybe it's too light:) Maybe if an exclusion decision is made, we shouldn't get back even if maintainers want to fix their collections. They can think "Ah, we have a lot of time and will fix our collection right before the exclusion and if the decision is finally made because the policy said that we can do it".
It takes a lot of time and effort for many contributors to review collection for inclusion, we will spend our meeting time to discuss and vote not discussing other important questions. Should it be easy for such collections to stay?

[felixfontein]

Currently netbox.netbox is a candidate for exclusion, since it hasn't been working with ansible-core 2.11 (so basically the modules in that collection haven't been working since Ansible 4.0.0). The collection is in search of new maintainers, and it is not clear who can actually merge things (since a fix has been around for some time).

My suggestion would be to see whether this gets resolved in time for Ansible 5.0.0; and if not, let's remove netbox.netbox from Ansible 5.

As a deadline I'd suggest October 12th, i.e. the same deadline as for including new collections into Ansible 5 (https://docs.ansible.com/ansible/devel/roadmap/COLLECTIONS_5.html).

[Andersson007]

Version 2 of Initial view how it can basically look like (see the description for the first version) in the checklist format:

• Rough violations of collections requirements / signs of abandonment are found
• Issue is raised in the collection repo
• The violations haven't been solved within a month
• (If needed, a quick discussion in the community meeting whether starting the exclusion process is sensible)
• Exclusion discussion is open, initial submitters / repo maintainers are pinged there
• Vendor representatives / individual maintainers are informed by email (if possible)
• The violations haven't been solved within a month
• Meeting topic is open
• Meeting discussion, voting
• Announcement about a possible exclusion is made via Bullhorn. If the collection lives in the ansible_collections org, in addition a new-maintainers-are-wanted announcement is made via Bullhorn
• The violations hasn't been solved within a month / new maintainers haven't been found
• Final announcement

Not sure about the stages order and the terms.

[jillr]

I may miss the meeting today. I'd like more time to process and to see the discussion that occurs before voting +1/-1 but nothing obvious stands out to me as a problem. +0 with no specific objections.

[jillr]

Maybe it's too light:) Maybe if an exclusion decision is made, we shouldn't get back even if maintainers want to fix their collections. They can think "Ah, we have a lot of time and will fix our collection right before the exclusion and if the decision is finally made because the policy said that we can do it".
It takes a lot of time and effort for many contributors to review collection for inclusion, we will spend our meeting time to discuss and vote not discussing other important questions. Should it be easy for such collections to stay?

I'd be inclined to assume positive intent initially, and if the policy is abused we can revisit it.

In this part of the second checklist:

The violations haven't been solved within a month
Meeting topic is open
Meeting discussion, voting
Announcement about a possible exclusion is made via Bullhorn. If the collection lives in the ansible_collections org, in addition a new-maintainers-are-wanted announcement is made via Bullhorn
The violations hasn't been solved within a month / new maintainers haven't been found

Are these 2 months total? One month to wait if the violations are initially solved, and another month after the meeting discussion and bullhorn announcement.

[dmsimard]

We should have brought this topic for formal discussion sooner :(

Version 2 of Initial view how it can basically look like (see the description for the first version) in the checklist format:

* [ ]  Rough violations of collections requirements / signs of abandonment are found
* [ ]  Issue is raised in the collection repo
* [ ]  The violations haven't been solved within a month
* [ ]  (If needed, a quick discussion in the community meeting whether starting the exclusion process is sensible)
* [ ]  Exclusion discussion is open, initial submitters / repo maintainers are pinged there
* [ ]  Vendor representatives / individual maintainers are informed by email (if possible)
* [ ]  The violations haven't been solved within a month
* [ ]  Meeting topic is open
* [ ]  Meeting discussion, voting
* [ ]  Announcement about a **possible** exclusion is made via Bullhorn. If the collection lives in the `ansible_collections` org, in addition a new-maintainers-are-wanted announcement is made via Bullhorn
* [ ]  The violations hasn't been solved within a month / new maintainers haven't been found
* [ ]  Final announcement

Not sure about the stages order and the terms.

My first thoughts are that this is long, (very) forgiving and places a lot of the burden on us (the community working group/steering committee) rather than the collection maintainers.

I do not feel it is appropriate to remove a collection in a minor release (to avoid breaking users) and so I would have a tendency to set the scope to be within the lifecycle of a major version rather than one or two months as suggested.

In other words, if we find a collection to be abandoned, the collection should be at risk of being excluded in the upcoming major version unless the situation is addressed in time. Even without security considerations, I would rather us not be on the hook for shipping unmaintained software and users can always install collections manually from Galaxy at their own risk if they really want to.

Now, I must recognize and acknowledge that we have not been very proactive in monitoring for inactivity or violations and so we find ourselves with a fairly long list of collections that should be considered as part of this process, ideally in time for Ansible 6 to avoid needing to wait until Ansible 7.

Let's chat about it at the meeting today.

[Andersson007]

possible indicators of abandonment:

• permanently red CI
• no releases
• open issues / PRs but no response from maintainers

[jillr]

A few thoughts based on irc conversations:

• I'd like whatever process we have to have some grace for maintainers (ie; "no response for 4 weeks and you're permabanned", "oh, I'm a solo maintainer of a small collection and I was sick, sorry but I'm back now?" would not be nice).
• I don't consider no recent commits to be unmaintained if there are no other indicators (consistently failing CI, no response/triage of any issues, etc)
• We should include a deprecation notice in the porting guide before removing a collection, with exceptions for exceptional circumstances like maybe a security problem. (acknowledging that we can't force anyone to read the porting guides)

[acozine]

Do we have criteria for this? What is our goal in removing collections? Are we trying to make the Ansible package smaller? Trying to protect users from broken plugins? Trying to incentivize maintainers/contributors? How do we know if our policy/procedure is succeeding?

[Andersson007]

users should know that a collection needs maintainers via warnings, Bullhorn, etc. to give someone a change to pick up the collection
ie. to become a new maintainer

[jillr]

@acozine I would expect users to have an assumption that if we're maintaining and shipping the ansible package with various things included, that the software it contains is also maintained in some capacity. We have no way to be certain that the included collections will continue to work if they're unmaintained (for example; servicenow ships a new version, let's call it Rome, with major API changes). So this is a way for us to communicate expectations with our users and protect them from potential breakage or unpredictable results.

[markuman]

I'd like whatever process we have to have some grace for maintainers (ie; "no response for 4 weeks and you're permabanned", "oh, I'm a solo maintainer of a small collection and I was sick, sorry but I'm back now?" would not be nice).

I fully agree here.
Maybe it's also worth to think about a bus-factor for the number of active maintainers as a inclusion criteria to prevent as much as possible an exclusion process because the project gets inactive/abonded.

[Andersson007]

should we include maintainers' responsiveness, etc. in the standards and consider such cases as violation of standards?

[mariolenz]

users should know that a collection needs maintainers via warnings, Bullhorn, etc. to give someone a change to pick up the collection
ie. to become a new maintainer

I understood that The Bullhorn is a Newsletter for the Ansible Developer Community, not for users. Of course, it doesn't hurt to use this channel of communication. But a warning when using the collection (might need a change to ansible-core) probably would reach more people.

Another good place would be the changelog.

Should we preemptively develop collection exclusion criteria and procedure?

I don't know why you say "preemptively", having collection exclusion criteria and procedures sounds like a good idea to me.

I do not feel it is appropriate to remove a collection in a minor release (to avoid breaking users) and so I would have a tendency to set the scope to be within the lifecycle of a major version

I agree. Removing a collection, even if it looks unmaintained, looks like a breaking change to me. After all, it might work for a lot of people even if it isn't maintained any more. So my suggestion is to deprecate this collection and remove it from the next major version.

Now, I must recognize and acknowledge that we have not been very proactive in monitoring for inactivity or violations and so we find ourselves with a fairly long list of collections that should be considered as part of this process, ideally in time for Ansible 6 to avoid needing to wait until Ansible 7.

Feel free to start with community.vmware. I'd like to keep it in Ansible 6 and the sooner I know you see a problem, the better. Although I can't do much at the moment since we have a CI problem I can't solve :-/ (ansible-collections/community.vmware#1260)

should we include maintainers' responsiveness, etc. in the standards and consider such cases as violation of standards?

I think "responsiveness" would be hard to define in a fair way. Don't get me wrong, I think this is important. But I also think it might be quite hard to find a valid metric for this.

[felixfontein]

users should know that a collection needs maintainers via warnings, Bullhorn, etc. to give someone a change to pick up the collection
ie. to become a new maintainer

I understood that The Bullhorn is a Newsletter for the Ansible Developer Community, not for users. Of course, it doesn't hurt to use this channel of communication.

Yes, it should be one of multiple channels used.

But a warning when using the collection (might need a change to ansible-core) probably would reach more people.

This is very non-trivial, since there's no way to do this without modifying the collection. Since we do not want to ship modified collections, this would mean we would have to get the collection to make a new release with such warnings inserted. Which is something we can only do for collections we actually control, and even there it is something we shouldn't really do IMO.

Another good place would be the changelog.

This is a place where such things must be announced IMO.

I do not feel it is appropriate to remove a collection in a minor release (to avoid breaking users) and so I would have a tendency to set the scope to be within the lifecycle of a major version

I agree. Removing a collection, even if it looks unmaintained, looks like a breaking change to me. After all, it might work for a lot of people even if it isn't maintained any more. So my suggestion is to deprecate this collection and remove it from the next major version.

IMO removal in a minor version can only be done in three cases:

1. The collection is broken and nothing from it can actually be used. (Even then, I would only remove it in the next major release.)
2. Keeping the collection is a very large security risk that cannot migitated in any other sensible way than removing it. I hope we'll never have such a situation though.
3. Some legal situation where it turns out that the collection can only be distributed under an unacceptable terms (like: you need to pay someone money to use it) that we were not aware of (otherwise it would have never been included in the first place). I also hope this never happns :)

should we include maintainers' responsiveness, etc. in the standards and consider such cases as violation of standards?

I think "responsiveness" would be hard to define in a fair way. Don't get me wrong, I think this is important. But I also think it might be quite hard to find a valid metric for this.

I think it's basically impossible to define this. No matter what definition you use, you'll always find someone matching it who shouldn't, and someone not matching it who should.

[mariolenz]

But a warning when using the collection (might need a change to ansible-core) probably would reach more people.

This is very non-trivial, since there's no way to do this without modifying the collection. Since we do not want to ship modified collections, this would mean we would have to get the collection to make a new release with such warnings inserted.

Good point, but I didn't want you to ship modified collections. I just thought it might be possible to issue a warning when using a collection that's deprecated without modifying the collection. That would probably be non-trivial, either.

I don't know enough of Python / PyPI / pip packages, but would it be possible to add a warning about deprecated collections when installing Ansible? Just an idea.

Another good place would be the changelog.

This is a place where such things must be announced IMO.

I agree, deprecating a collection must be announced in the changelog.

IMO removal in a minor version can only be done in three cases:

1. The collection is broken and nothing from it can actually be used. (Even then, I would only remove it in the next major release.)
2. Keeping the collection is a very large security risk that cannot migitated in any other sensible way than removing it. I hope we'll never have such a situation though.
3. Some legal situation where it turns out that the collection can only be distributed under an unacceptable terms (like: you need to pay someone money to use it) that we were not aware of (otherwise it would have never been included in the first place). I also hope this never happns :)

I haven't thought about critical security risks or legal issues, but I agree that those might be important enough to remove a collection within a minor version.

[Andersson007]

closed as https://github.com/ansible-collections/overview/blob/main/removal_from_ansible.rst was created, thanks everyone!