Docs [2.10] Backportapalooza 6

docs/docsite/_themes/sphinx_rtd_theme/breadcrumbs.html

@@ -44,10 +44,14 @@
               <a href="{{ meta['github_url'] }}" class="fa fa-github"> {{ _('Edit on GitHub') }}</a>
             {% else %}
               <!-- Ansible-specific additions for modules etc -->
-                {% if pagename.endswith('_module') %}
-                  <a href="https://{{ github_host|default("github.com") }}/{{ github_user }}/{{ github_repo }}/{{ theme_vcs_pageview_mode|default("blob") }}/{{ github_module_version }}{{ meta.get('source', '') }}?description=%23%23%23%23%23%20SUMMARY%0A%3C!---%20Your%20description%20here%20--%3E%0A%0A%0A%23%23%23%23%23%20ISSUE%20TYPE%0A-%20Docs%20Pull%20Request%0A%0A%2Blabel:%20docsite_pr" class="fa fa-github"> {{ _('Edit on GitHub') }}</a>
-                {% elif pagename.startswith('plugins') and meta.get('source', None) %}
-                  <a href="https://{{ github_host|default("github.com") }}/{{ github_user }}/{{ github_repo }}/{{ theme_vcs_pageview_mode|default("blob") }}/{{ github_root_dir }}/{{ pagename }}.py?description=%23%23%23%23%23%20SUMMARY%0A%3C!---%20Your%20description%20here%20--%3E%0A%0A%0A%23%23%23%23%23%20ISSUE%20TYPE%0A-%20Docs%20Pull%20Request%0A%0A%2Blabel:%20docsite_pr" class="fa fa-github"> {{ _('Edit on GitHub') }}</a>
+                {% if (pagename.endswith('_module')) or (pagename.endswith('_become'))
+                  or (pagename.endswith('_cache')) or (pagename.endswith('_callback'))
+                  or (pagename.endswith('_connection')) or (pagename.endswith('_inventory'))
+                  or (pagename.endswith('_lookup')) or (pagename.endswith('_shell'))
+                  or (pagename.endswith('_strategy')) or (pagename.endswith('_vars'))
+                 %}
+                <!--  <a href="https://{{ github_host|default("github.com") }}/{{ github_user }}/{{ github_repo }}/{{ theme_vcs_pageview_mode|default("blob") }}/{{ github_module_version }}{{ meta.get('source', '') }}?description=%23%23%23%23%23%20SUMMARY%0A%3C!---%20Your%20description%20here%20--%3E%0A%0A%0A%23%23%23%23%23%20ISSUE%20TYPE%0A-%20Docs%20Pull%20Request%0A%0A%2Blabel:%20docsite_pr" class="fa fa-github"> {{ _('Edit on GitHub') }}</a> -->
+                  <br>
                 {% elif pagename.startswith('cli') and meta.get('source', None) %}
                   <a href="https://{{ github_host|default("github.com") }}/{{ github_user }}/{{ github_repo }}/{{ theme_vcs_pageview_mode|default("blob") }}/{{ github_cli_version }}{{ meta.get('source', '') }}?description=%23%23%23%23%23%20SUMMARY%0A%3C!---%20Your%20description%20here%20--%3E%0A%0A%0A%23%23%23%23%23%20ISSUE%20TYPE%0A-%20Docs%20Pull%20Request%0A%0A%2Blabel:%20docsite_pr" class="fa fa-github"> {{ _('Edit on GitHub') }}</a>
                 {% elif (not 'list_of' in pagename) and (not 'category' in pagename) %}

docs/docsite/rst/community/development_process.rst

@@ -43,7 +43,7 @@ Here's an overview of the PR lifecycle:
 Automated PR review: ansibullbot
 --------------------------------
 
-Because Ansible receives many pull requests, and because we love automating things, we've automated several steps of the process of reviewing and merging pull requests with a tool called Ansibullbot, or Ansibot for short.
+Because Ansible receives many pull requests, and because we love automating things, we have automated several steps of the process of reviewing and merging pull requests with a tool called Ansibullbot, or Ansibot for short.
 
 `Ansibullbot <https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md>`_ serves many functions:
 
@@ -139,11 +139,7 @@ We don't merge every PR. Here are some tips for making your PR useful, attractiv
 Changelogs
 ----------
 
-Changelogs help users and developers keep up with changes to Ansible.
-Ansible builds a changelog for each release from fragments.
-You **must** add a changelog fragment to any PR that changes functionality or fixes a bug in ansible-base.
-You don't have to add a changelog fragment for PRs that add new
-modules and plugins, because our tooling does that for you automatically.
+Changelogs help users and developers keep up with changes to Ansible. Ansible builds a changelog for each release from fragments. You **must** add a changelog fragment to any PR that changes functionality or fixes a bug in ansible-base. You do not have to add a changelog fragment for PRs that add new modules and plugins, because our tooling does that for you automatically.
 
 We build short summary changelogs for minor releases as well as for major releases. If you backport a bugfix, include a changelog fragment with the backport PR.
 
@@ -152,44 +148,37 @@ We build short summary changelogs for minor releases as well as for major releas
 Creating a changelog fragment
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-A basic changelog fragment is a ``.yaml`` file placed in the
-``changelogs/fragments/`` directory.  Each file contains a yaml dict with
-keys like ``bugfixes`` or ``major_changes`` followed by a list of
-changelog entries of bugfixes or features.  Each changelog entry is
-rst embedded inside of the yaml file which means that certain
-constructs would need to be escaped so they can be interpreted by rst
-and not by yaml (or escaped for both yaml and rst if that's your
-desire).  Each PR **must** use a new fragment file rather than adding to
-an existing one, so we can trace the change back to the PR that introduced it.
+A basic changelog fragment is a ``.yaml`` file placed in the ``changelogs/fragments/`` directory.  Each file contains a yaml dict with keys like ``bugfixes`` or ``major_changes`` followed by a list of changelog entries of bugfixes or features.  Each changelog entry is rst embedded inside of the yaml file which means that certain constructs would need to be escaped so they can be interpreted by rst and not by yaml (or escaped for both yaml and rst if you prefer).  Each PR **must** use a new fragment file rather than adding to an existing one, so we can trace the change back to the PR that introduced it.
 
-To create a changelog entry, create a new file with a unique name in the ``changelogs/fragments/`` directory of corresponding repository.
-The file name should include the PR number and a description of the change.
-It must end with the file extension ``.yaml``. For example: ``40696-user-backup-shadow-file.yaml``
+To create a changelog entry, create a new file with a unique name in the ``changelogs/fragments/`` directory of the corresponding repository. The file name should include the PR number and a description of the change. It must end with the file extension ``.yaml``. For example: ``40696-user-backup-shadow-file.yaml``
 
-A single changelog fragment may contain multiple sections but most will only contain one section.
-The toplevel keys (bugfixes, major_changes, and so on) are defined in the
-`config file <https://github.com/ansible/ansible/blob/devel/changelogs/config.yaml>`_ for our release note tool. Here are the valid sections and a description of each:
+A single changelog fragment may contain multiple sections but most will only contain one section. The toplevel keys (bugfixes, major_changes, and so on) are defined in the `config file <https://github.com/ansible/ansible/blob/devel/changelogs/config.yaml>`_ for our `release note tool <https://github.com/ansible-community/antsibull-changelog/blob/main/docs/changelogs.rst>`_. Here are the valid sections and a description of each:
+
+**breaking_changes**
+  Changes that break existing playbooks or roles. This includes any change to existing behavior that forces users to update tasks. Displayed in both the changelogs and the :ref:`Porting Guides <porting_guides>`.
 
 **major_changes**
-  Major changes to Ansible itself. Generally does not include module or plugin changes.
+  Major changes to Ansible itself. Generally does not include module or plugin changes. Displayed in both the changelogs and the :ref:`Porting Guides <porting_guides>`.
 
 **minor_changes**
   Minor changes to Ansible, modules, or plugins. This includes new features, new parameters added to modules, or behavior changes to existing parameters.
 
 **deprecated_features**
-  Features that have been deprecated and are scheduled for removal in a future release.
+  Features that have been deprecated and are scheduled for removal in a future release. Displayed in both the changelogs and the :ref:`Porting Guides <porting_guides>`.
 
 **removed_features**
-  Features that were previously deprecated and are now removed.
+  Features that were previously deprecated and are now removed. Displayed in both the changelogs and the :ref:`Porting Guides <porting_guides>`.
+
+**security_fixes**
+  Fixes that address CVEs or resolve security concerns. Include links to CVE information.
 
 **bugfixes**
   Fixes that resolve issues.
 
 **known_issues**
   Known issues that are currently not fixed or will not be fixed.
 
-Each changelog entry must contain a link to its issue between parentheses at the end.
-If there is no corresponding issue, the entry must contain a link to the PR itself.
+Each changelog entry must contain a link to its issue between parentheses at the end. If there is no corresponding issue, the entry must contain a link to the PR itself.
 
 Most changelog entries will be ``bugfixes`` or ``minor_changes``. When writing a changelog entry that pertains to a particular module, start the entry with ``- [module name] -`` and the following sentence with a lowercase letter.
 
@@ -222,9 +211,7 @@ Once you've written the changelog fragment for your PR, commit the file and incl
 Backporting merged PRs
 ======================
 
-All Ansible PRs must be merged to the ``devel`` branch first.
-After a pull request has been accepted and merged to the ``devel`` branch, the following instructions will help you create a
-pull request to backport the change to a previous stable branch.
+All Ansible PRs must be merged to the ``devel`` branch first. After a pull request has been accepted and merged to the ``devel`` branch, the following instructions will help you create a pull request to backport the change to a previous stable branch.
 
 We do **not** backport features.
 

docs/docsite/rst/dev_guide/developing_api.rst

@@ -6,7 +6,7 @@ Python API
 
 .. contents:: Topics
 
-.. note:: This API is intended for internal Ansible use. Ansible may make changes to this API at any time that could break backward compatibility with older versions of the API. Because of this, external use is not supported by Ansible.
+.. note:: This API is intended for internal Ansible use. Ansible may make changes to this API at any time that could break backward compatibility with older versions of the API. Because of this, external use is not supported by Ansible. If you want to use Python API only for executing playbooks or modules, consider `ansible-runner <https://ansible-runner.readthedocs.io/en/latest/>`_ first.
 
 There are several ways to use Ansible from an API perspective.   You can use
 the Ansible Python API to control nodes, you can extend Ansible to respond to various Python events, you can

docs/docsite/rst/dev_guide/developing_collections.rst

@@ -25,6 +25,8 @@ Collections follow a simple data structure. None of the directories are required
     collection/
     ├── docs/
     ├── galaxy.yml
+    ├── meta/
+    │   └── runtime.yml
     ├── plugins/
     │   ├── modules/
     │   │   └── module1.py
@@ -194,6 +196,64 @@ command completion, or environment variables are presented throughout the
 Ansible Collection Testing because the act of installing the stable release of
 Ansible containing `ansible-test` is expected to setup those things for you.
 
+.. _meta_runtime_yml:
+
+meta directory
+--------------
+
+A collection can store some additional metadata in a ``runtime.yml`` file in the collection's ``meta`` directory. The ``runtime.yml`` file supports the top level keys:
+
+- *requires_ansible*:
+
+  The version of Ansible required to use the collection. Multiple versions can be separated with a comma.
+
+  .. code:: yaml
+
+     requires_ansible: ">=2.10,<2.11"
+
+  .. note:: although the version is a `PEP440 Version Specifier <https://www.python.org/dev/peps/pep-0440/#version-specifiers>`_ under the hood, Ansible deviates from PEP440 behavior by truncating prerelease segments from the Ansible version. This means that Ansible 2.11.0b1 is compatible with something that ``requires_ansible: ">=2.11"``.
+
+- *plugin_routing*:
+
+  Content in a collection that Ansible needs to load from another location or that has been deprecated/removed.
+  The top level keys of ``plugin_routing`` are types of plugins, with individual plugin names as subkeys.
+  To define a new location for a plugin, set the ``redirect`` field to another name.
+  To deprecate a plugin, use the ``deprecation`` field to provide a custom warning message and the removal date or version. If the plugin has been renamed or moved to a new location, the ``redirect`` field should also be provided. If a plugin is being removed entirely, ``tombstone`` can be used for the fatal error message and removal date or version.
+
+  .. code:: yaml
+
+     plugin_routing:
+       inventory:
+         kubevirt:
+           redirect: community.general.kubevirt
+         my_inventory:
+           tombstone:
+             removal_version: "2.0.0"
+             warning_text: my_inventory has been removed. Please use other_inventory instead.
+       modules:
+         my_module:
+           deprecation:
+             removal_date: "2021-11-30"
+             warning_text: my_module will be removed in a future release of this collection. Use another.collection.new_module instead.
+           redirect: another.collection.new_module
+         podman_image:
+           redirect: containers.podman.podman_image
+       module_utils:
+         ec2:
+           redirect: amazon.aws.ec2
+         util_dir.subdir.my_util:
+           redirect: namespace.name.my_util
+
+- *import_redirection*
+
+  A mapping of names for Python import statements and their redirected locations.
+
+  .. code:: yaml
+
+     import_redirection:
+       ansible.module_utils.old_utility:
+         redirect: ansible_collections.collection_name.plugins.module_utils.new_location
+
 
 .. _creating_collections_skeleton:
 
@@ -469,24 +529,70 @@ Collection versions use `Semantic Versioning <https://semver.org/>`_ for version
 
 .. _migrate_to_collection:
 
-Migrating Ansible content to a collection
-=========================================
+Migrating Ansible content to a different collection
+====================================================
+
+To migrate content from one collection to another, you need to create three PRs as follows:
+
+#. Create a PR against the old collection to remove the content.
+#. Create a PR against the new collection to add the files removed in step 1.
+#. Update the ``ansible/ansible:devel`` branch entries for all files moved.
+
+
+Removing the content from the old collection
+----------------------------------------------
+
+Create a PR against the old collection repo to remove the modules, module_utils, plugins, and docs_fragments related to this migration:
+
+#. If you are removing an action plugin, remove the corresponding module that contains the documentation.
+#. If you are removing a module, remove any corresponding action plugin that should stay with it.
+#. Remove any entries about removed plugins from ``meta/runtime.yml``. Ensure they are added into the new repo.
+#. Remove sanity ignore lines from ``tests/sanity/ignore\*.txt``
+#. Remove associated integration tests from ``tests/integrations/targets/`` and unit tests from ``tests/units/plugins/``.
+#. if you are removing from content from ``community.general`` or ``community.network``, remove entries from ``.github/BOTMETA.yml``.
+#. Carefully review ``meta/runtime.yml`` for any entries you may need to remove or update, in particular deprecated entries.
+#. Update ``meta/runtime.yml`` to contain redirects for EVERY PLUGIN, pointing to the new collection name.
+#. If possible, do not yet add deprecation warnings to the new ``meta/runtime.yml`` entries, but only for a later major release. So the order should be:
+    1. Remove content, add redirects in 3.0.0;
+    2. Deprecate redirects in 4.0.0;
+    3. Set removal version to 5.0.0 or later.
 
-You can experiment with migrating existing modules into a collection using the `content_collector tool <https://github.com/ansible/content_collector>`_. The ``content_collector`` is a playbook that helps you migrate content from an Ansible distribution into a collection.
 
 .. warning::
 
-	This tool is in active development and is provided only for experimentation and feedback at this point.
+	Maintainers for the old collection have to make sure that the PR is merged in a way that it does not break user experience and semantic versioning:
+
+	#. A new version containing the merged PR must not be released before the collection the content has been moved to has been released again, with that content contained in it. Otherwise the redirects cannot work and users relying on that content will experience breakage.
+	#. Once 1.0.0 of the collection from which the content has been removed has been released, such PRs can only be merged for a new **major** version (i.e. 2.0.0, 3.0.0, etc.).
+
+
+Adding the content to the new collection
+-----------------------------------------
+
+Create a PR in the new collection to:
+
+#. Add ALL the files removed in first PR (from the old collection).
+#. If it is an action plugin, include the corresponding module with documentation.
+#. If it is a module, check if it has a corresponding action plugin that should move with it.
+#. Check ``meta/ `` for relevant updates to ``action_groups.yml`` and ``runtime.yml`` if they exist.
+#. Carefully check the moved ``tests/integration`` and ``tests/units`` and update for FQCN.
+#. Review ``tests/sanity/ignore-\*.txt`` entries.
+#. Update ``meta/runtime.yml``.
+
+Updating ``ansible/ansible:devel`` branch entries for all files moved
+----------------------------------------------------------------------
+
+Create a third PR on ``ansible/ansible`` repository to:
 
-See the `content_collector README <https://github.com/ansible/content_collector>`_ for full details and usage guidelines.
+#. Update ``lib/ansible/config/ansible_builtin_runtime.yml`` (the redirect entry).
+#. Update ``.github/BOTMETA.yml`` (the migrated_to entry)
 
 BOTMETA.yml
 -----------
 
 The `BOTMETA.yml <https://github.com/ansible/ansible/blob/devel/.github/BOTMETA.yml>`_ in the ansible/ansible GitHub repository is the source of truth for:
 
 * ansibullbot
-* the docs build for collections-based modules
 
 Ansibulbot will know how to redirect existing issues and PRs to the new repo.
 The build process for docs.ansible.com will know where to find the module docs.