Adopt a new format for the changelog

[di]

We should probably switch to towncrier at some point to get us better CI integration etc.

Originally posted by @sigmavirus24 in #545 (comment)

[jaraco]

My biggest reservation about towncrier is it adds a step to cutting a release, a step that necessarily requires a command-line to execute. What about something like reno instead? I've been thinking about adopting that for other projects.

[sigmavirus24]

I mean, we're part of the PyPA. Other projects are using towncrier with little angst it seems. I also know the towncrier folks well enough to feel that if we approached them reasonably with "Can we collaborate on a way to make this friendlier to automate along with our automated releases?" they'd say "Sure, let's talk and get that done." In the end it would benefit everyone and wouldn't cause us to depend on a failing ecosystem (OpenStack)

[jaraco]

When I asked in Towncrier, one of the top contributors recommended reno for the use-case I'm trying to support - minimizing the environment demands to cutting a release. We use it in setuptools, and it does cause angst. I find I need to read the "releasing" docs each time to remember how releases are cut. I'd really like push-button releases for projects, and towncrier by its design inhibits that goal.

wouldn't cause us to depend on a failing ecosystem (OpenStack)

I don't think reno depends on a failing ecosystem. Its dependency tree is modest:

~ $ pip-run -q reno pipdeptree -- -m pipdeptree -p reno                                                                                                                   
reno==2.11.3
  - dulwich [required: >=0.15.0, installed: 0.19.13]
    - certifi [required: Any, installed: 2019.9.11]
    - urllib3 [required: >=1.24.1, installed: 1.25.7]
  - pbr [required: Any, installed: 5.4.3]
  - PyYAML [required: >=3.10, installed: 5.1.2]
  - six [required: >=1.9.0, installed: 1.13.0]

Can you elaborate on the risks of adopting a tool that's affiliated with OpenStack?

[sigmavirus24]

Reno is developed as an OpenStack project so that comes with it:

• The bureaucracy of:

  • Being taken seriously as a user
  • Being able to develop/contribute as an outsider and be taken seriously
  • Relying on a foundation that's rebranding itself to compete with teh CNCF and failing

• The instability of a failing foundation and more and more people stepping away from the community professionally to go where their jobs and the money takes them (the CNCF).

Reno, like other OpenStack projects will likely end un-maintained and relying on that with no one to accept patches, etc. will end causing us to need to move to something else. Twisted has been a project with consistent forward momentum (although not blazingly fast) and so I'm disappointed they're not considering this usecase.

Regardless, it seems disingenuous to me to call someone with 5 commits a "top contributor". Stephen is an OpenStack developer and would naturally try to get anyone to invest in that failing ecosystem.

Projects that want to out-live OpenStack are quickly migrating into other organizations. doc8 and bandit, for example have left for the PyCQA. I'm sure there are others I'm unaware of since I stopped working on OpenStack.

[jaraco]

Good points. How about we tackle it this way - I'll work with reno to see if I can make it work in other projects (something I've been planning anyway). If things are looking promising, I'll work with Doug Hellman to establish a sustainability plan for the project. In the meantime, you can work with the towncrier team to see if that project can support this use-case.

I'm not terribly unhappy with the current approach. It does require explicitly adding dates and seems to have a bug about detecting bugfixes in major version releases, but otherwise is lightweight and fairly nice. It also has issues with resolving conflicts, but those are usually readily solved by including both changes.

[bhrutledge]

Instead of adopting another tool, could we use a plain RST file? While working on #643, I found the build step clunky, the rendered result somewhat surprising, and the semantics around features/bugfixes/support overly-prescriptive.

Another benefit of a plain file would be that it's more easily viewed in GitHub. Some examples:

• packaging/CHANGELOG.rst
• readme_renderer/CHANGES.rst
• keyring/CHANGES.rst

If we want more structure, we could follow the format from https://keepachangelog.com/ (which I've seen used in a variety of projects).

I'm optimistic that migrating the current format could be done with regular expression replacement.

[sigmavirus24]

Instead of adopting another tool, could we use a plain RST file?

That's one option but it gives us no good way to have pull requests include changelog items without creating some bespoke tooling for it.

A single changelog file leads to merge-conflicts often times. If we're going to stick to plain RST, we can use a single file or we can try to organize our changes differently like larger projects with lots of releases actually find manageable: https://gitlab.com/pycqa/flake8/-/tree/master/docs/source/release-notes This also makes it easier to shorten the display of the versions for users

[di]

Instead of adopting another tool, could we use a plain RST file?

I don't see why we can't have both, the pip project uses towncrier and it generates a nice human-readadable .rst file as well: https://github.com/pypa/pip/blob/master/NEWS.rst

[bhrutledge]

no good way to have pull requests include changelog items

I don't think the contributing guide asks for this, and it's not something we've enforced. Of course, both of those things can change, but that would be adding a small roadblock for contributors. FWIW, I don't mind putting the changelog together prior to release, and I think that also avoids merge conflicts.

larger projects with lots of releases

the pip project uses towncrier

Given what seems like a relatively slow cadence of user-facing changes to Twine, I'm still more inclined to maintain a single flat file, rather than invest the time in adopting a new tool.

And, as a user of Python packages, I generally prefer a single changelog that I can scan/scroll/search through, rather than clicking through multiple pages

[di]

Given what seems like a relatively slow cadence of user-facing changes to Twine, I'm still more inclined to maintain a single flat file, rather than invest the time in adopting a new tool.

I think the slower cadence actually is more motivating here than less motivating. Having to go back and figure out what should be in the changelog for a new release (because we can't enforce that new PRs submit a changelog entry) is a pain.

[sigmavirus24]

I think the slower cadence actually is more motivating here than less motivating. Having to go back and figure out what should be in the changelog for a new release (because we can't enforce that new PRs submit a changelog entry) is a pain.

I whole-heartedly agree. It leads to poor release notes in general because the person generating the release often feels under time-pressure to get it out and summarize everything.

that would be adding a small roadblock for contributors.

I feel as though that's a possibility. It's also possible that it helps them think through use-cases other than their own to write the changelog item and helps reduce maintenance cost overall.

And, as a user of Python packages, I generally prefer a single changelog that I can scan/scroll/search through, rather than clicking through multiple pages

Right, that's the benefit of using a tool like towncrier rather than some homebrew nonsense[1] like what Flake8 does (which predates towncrier).

[1]: I feel justified in calling it homebrew nonsense because I'm the cause of the nonsense

[di]

To summarize:

• I think @sigmavirus24, @jaraco and I are in agreement here that we'd like to add some sort of automation and new format for the changelog. (maybe @jaraco is a +0 or a -0)
• I think @sigmavirus24 and I would prefer towncrier, while @jaraco mentioned reno, but I'm not sure what the experiments with that tool have been. I'm a strong +1 on towncrier for the same reasons @sigmavirus24 mentioned, namely that it's already being used successfully by other PyPA projects.
• @bhrutledge is proposing to keep the status quo (plain rST) and perhaps adopt a slightly different document structure.

Given all that, I'm going to propose that we:

1. implement changelog automation, and
2. that we use towncrier.

If I've misrepresented anyone's viewpoint or if anyone's a strong -1 on that plan, please comment.

[di]

Also, I'd say this doesn't need to block a 3.2.0 release.

[bhrutledge]

Thanks, @di. I'm game for towncrier.

Re: 3.2.0 release, I added this (and some others) to the milestone as "maybe" or "nice to have". Maybe further discussion on what should be included can happen in #643?

[bhrutledge]

Following up on #643 (comment), should we take this opportunity to document that Twine uses semantic versioning?

[sigmavirus24]

@bhrutledge good idea. Yeah I think you can just send a PR for that and self-merge. It should be straight-forward to explain

[bhrutledge]

For folks interested in changelog build tools (which I'm still not sold on), a new one is nedbat/scriv. There's a comparison to towncrier and reno at Philosophy — Scriv 0.9.2 documentation.

[jaraco]

I've become more and more opposed to any system that requires a separate commit to a file. There are lots of workflows where such changes are difficult to manage, the most common being where someone sends a perfect PR except it has no changelog. Adding a changelog entry, whether that be in a static file or any system that aggregates changes from committed files, cannot be amended through the Github UI, so cannot be completed without another round-trip with the contributor or grabbing a device that can check out a repo, switch forks, add files, and push to forks. I'd really like to find something that would enable a maintainer to supply a change log entry for a PR as part of the accept/merge process, so presumably that means something that's based on the commit log or some other metadata and machinery. Such an approach would enable the user to supply the change log or for the maintainer to supply/amend it as part of the merge. Ideally any such system should also avoid copy/pasting text (in the commit, in the changelog, in the pull request, and maybe other places). I do agree with a lot of the principles of scriv. I think I'll give it a try.