RFC: change our Ruby version support policy for RSpec 4

[myronmarston]

TL;DR

I propose that we change our Ruby version support policy for RSpec 4 so that dropping support for old ruby versions is no longer coupled to major version releases. Instead, I recommend we adopt a policy based around supporting old Ruby versions for N months after the Ruby core team has EOL'd
it (for some reasonable value of N).

The Current State of Things

We currently support 10 6 versions of MRI:

• 1.8.7
• 1.9.2
• 1.9.3
• 2.0
• 2.1
• 2.2
• 2.3
• 2.4
• 2.5
• 2.6
• 2.7
• 3.0

In addition, we have been strictly interpreting SemVer to mean that the only time we can drop support for an old version of Ruby is when we do a major release. As a result, we drop support for old versions very, very rarely. In fact, in all my years of involvement with RSpec (dating back to 2010), I can only think of one version of Ruby we've ever dropped support for: we stopped supporting 1.8.6 when RSpec 3 was released.

I think the amount of effort we expend supporting old versions vastly exceeds the utility to our users.

Why is this a problem?

Supporting old Ruby versions has a very real maintenance cost:

• Supporting old versions bloats our Travis build matrix, costing Travis more resources, and slowing down our feedback cycle from our CI builds.
• Our CI build does not consistently pass on old Ruby versions for reasons outside our control. For example, our CI builds get intermittent segfaults on MRI 1.9.2 and REE. Dealing with those failures (kicking the build, etc) has a very real maintenance cost.
• Continuing to support old Ruby versions has prevented us from upgrading our dependencies. For example, we are still on Cucumber 1.x, even though Cucumber 3.1 5.2 is out. Cucumber does not support ruby 1.8.7 starting with Cucumber 2, so we haven't been able to upgrade. Staying on old versions of dependencies causes us to accumulate excess tech debt.
• In my experience, there's often extra work to get my PRs green against the old Ruby versions. For example, I often accidentally use the newer hash syntax, which 1.8.7 cannot parse. After the CI build finishes, I see the failure, fix it, and force push to address the issue. This creates extra work for what should be simple changes.
• Supporting old Ruby versions limits our ability to take advantage of new Ruby features.

Regarding SemVer

The SemVer spec does not specifically document when a library is allowed to drop support for an old version of a language. I'd argue that it's a grey area that we're allowed to decide on for our project. And even if it did explicitly document it, it's up to us to decide if we want to strictly follow it. IMO, the important thing about SemVer is that your project documents the semantics of your versioning and sticks to it--not necessarily that you follow the SemVer spec. IMO, if we realize that strict adherence to the SemVer spec was inhibiting our ability to serve our users due to extra work it requires, it would be wise and appropriate to choose not to follow the spec anymore, and decide on our own semantic that we think better serves our users.

That said, I don't believe the SemVer spec requires that we maintain support for old versions of the language until the next major release.

Coupling Version Drops to Major Releases is a problem

While we strive to follow SemVer, bumping RSpec's major version is not purely a technical decision. It's also a marketing decision. The Ruby community has certain expectations around a major release of a project like RSpec containing major new features. In addition, there are lots of published materials (such as my own book) that reference RSpec's version number.

That means that we can't simply rev the major version number more often, as a means to drop old ruby versions. Besides, I think our users appreciate the stability of a major RSpec version line lasting for years.

Given that major new RSpec versions have historically only happened every 3-5 years, and our recent productivity is, if anything, less than it's been in the past, I think that coupling Ruby version support to major releases is problematic. We could better serve our users (by more productively addressing their issues, building new features, etc) if we were able to drop support for old versions of Ruby between major versions.

So, I believe that beginning with RSpec 4, we should come up with a new Ruby version policy that is not coupled to major releases.

What should we do instead?

I propose that we allow ourselves to drop support for old versions of Ruby as part of a minor release (e.g. x.y.0) in addition to major releases. In addition, I think it would be beneficial to have a specific policy about when we will drop old versions, so that it's not arbitrary based on when we feel like it. The Ruby core team announces the support schedule for each version of Ruby, and that includes an EOL date. I think we should adopt a policy of dropping support for old versions of ruby in the next minor release that comes N months after a version is EOL'd (for some reasonable value of N, such as 6, 12, 18 or 24).

Here's a concrete example. Ruby 2.0 was EOL'd on 2016-02-24. If we had adopted this policy for RSpec 3, and chose N=24, we would be dropping support for Ruby 2.0 in RSpec 3.8, which is the next minor release, since 2 years has elapsed since it was EOL'd.

How would this effect users?

• For users who use RSpec to test applications: I do not expect this to have much effect (if any) on these users. This could really only effect users who are running really old versions of Ruby that are months or years past the point where the Ruby core team is supporting it. I doubt there are any users who are allowing their Ruby version to remain so far behind while also staying on the latest version of RSpec.
• For users who use RSpec to test gems: users who support old versions of Ruby that we no longer support will be effected. They won't be able to upgrade RSpec without dropping support for old ruby versions. However, if we pick a large enough value of N, the number of users who are effected by this should be small.

Questions

1. Thoughts on the idea as a whole?
2. If we went with this proposal, what value of N should we pick?

[iGEL]

I believe that users who use a very old Ruby version but need to be on the latest Rspec version are very rare and not worth the effort you correctly mentioned. In my opinion, Nshould be 6 or max. 12 months.

[dnch]

1. Yes.
2. 12 months. Drive upgrades.

[xaviershay]

I'm most worried about ensuring we support bundler, though maybe with that being folded into rubygems it gets the same EOL policy as Ruby? In which case I'm more inclined towards a more aggressive N, maybe even matching them directly.

@indirect do you have any concerns about RSpec dropping old ruby version support (above policy)?

From the travis config, looks like bundler 2 is supporting 2.3+ but bundler 1.X is still supporting 1.8.7 https://github.com/bundler/bundler/blob/master/.travis.yml#L57

[JonRowe]

Older versions of Ruby, especially those > 2 do not cause us significant maintenance, so I'm heavily against a short N and in general not in favour of this. I'm ok with us dropping < 2 support in RSpec 4, but I don't really like the idea of dropping Ruby support in anything less than major versions, particularly because Bundler / rubygems will attempt to install the latest version regardless of your ruby specified e.g. so people who say use ~> 4.x will get error messages when we drop ruby versions without them doing anything, where as if we do it on major versions

I really don't think we should worry about the major version "expectation" we should just publicly state what we're doing and carry on, I'm also no at all concerned about references to versions in media, people expect those to be out of date, especially printed books.

[myronmarston]

Older versions of Ruby, especially those > 2 do not cause us significant maintenance,

I disagree. Every additional version of ruby that we support adds to our maintenance burden. It is a qualitatively different burden for Ruby 2.x compared to 1.x, but it's still there. And even if we felt like the 1.x burden is the main thing needing to be addressed, I'd argue that's reason enough to consider a different policy. The reason we still support 1.x is because we chose a policy that dictates that the only time we can drop support for a version is as part of a major version release, which only gives us the opportunity every few years. When we release a new version, I don't want the ruby versions supported at that time to impose a hard requirement on the RSpec maintainers for the next 3+ years due to our versioning policy.

particularly because Bundler / rubygems will attempt to install the latest version regardless of your ruby specified e.g. so people who say use ~> 4.x will get error messages when we drop ruby versions without them doing anything, where as if we do it on major versions

I assume we'd use the required_ruby_version field of our gemspec to declare what ruby versions we support in the metadata of our published gems. I thought bundler took this into account and wouldn't try to upgrade to a version that can't be installed according to it's required_ruby_version?

If bundler doesn't take that into account, it sounds like something we should push to have improved there.

I really don't think we should worry about the major version "expectation" we should just publicly state what we're doing and carry on, I'm also no at all concerned about references to versions in media, people expect those to be out of date, especially printed books.

My point was simply that we can't rev the major version as often as we'd like to be able to drop support for an old Ruby version. For example, my publisher really wouldn't have appreciated us releasing RSpec 4 right after they published my book on RSpec 3 (which took significant time and resources on their part). So I do not consider reving the major version of RSpec more often as a realistic possibility.

[JonRowe]

I disagree. Every additional version of ruby that we support adds to our maintenance burden. It is a qualitatively different burden for Ruby 2.x compared to 1.x, but it's still there.

Sure theoretically, in practise it's minimal. We spend more time solving peoples trivial issues that we do solving EOL'd Ruby bugs.

If support burden is the only reason we drop Rubies we could drop RBX and JRuby which cause us the most grief.

I assume we'd use the required_ruby_version field of our gemspec to declare what ruby versions we support in the metadata of our published gems. I thought bundler took this into account and wouldn't try to upgrade to a version that can't be installed according to it's required_ruby_version?

If bundler doesn't take that into account, it sounds like something we should push to have improved there.

It doesn't. Our own builds fall foul of this, which is why we have ruby version if blocks in them.

My point was simply that we can't rev the major version as often as we'd like to be able to drop support for an old Ruby version.

We haven't done it because we haven't felt the time is right, it's not that we can't.

For example, my publisher really wouldn't have appreciated us releasing RSpec 4 right after they published my book on RSpec 3

If we made major releases more common, without the expectation of major features, this becomes a non issue. It was only an issue for 2 -> 3 because we are made other radical feature changes at the same time. Arguably having a 3 -> 4 release (as we've discussed before) that only discards Ruby versions makes us more stable not less.

[iGEL]

It's possible that I missed the point of what you said, but it seems to me that bundler takes the required_ruby_version into account. I just tried to install Rails on Ruby 2.1/Bundler 1.15.1 and got version 4.2.10, not 5. My Gemspec was:

source "https://rubygems.org"

gem "rails"

When I tried to force version "5.1.0", I got:

Gem::InstallError: rack requires Ruby version >= 2.2.2.
An error occurred while installing rack (2.0.4), and Bundler cannot continue.
Make sure that `gem install rack -v '2.0.4'` succeeds before bundling.

In Gemfile:
  rails was resolved to 5.1.0, which depends on
    actioncable was resolved to 5.1.0, which depends on
      actionpack was resolved to 5.1.0, which depends on
        rack-test was resolved to 0.6.3, which depends on
          rack

[myronmarston]

Thanks for actually trying it, @iGEL :). Seems like we don't have to worry about bundler auto-upgrading users to a version they can't use as long as we declare required_ruby_version correctly.

Sure theoretically, in practise it's minimal. We spend more time solving peoples trivial issues that we do solving EOL'd Ruby bugs.

EOL'd Ruby bugs are a tiny part of the maintenance cost I'm talking about. Here are some things I'm talking about:

• Slower travis builds because we have so many parts to our build matrix
• More frequent flaky travis builds (again, due to so many parts of the build matrix)
• Inability to upgrade our dependencies. While dropping 1.8.7 and 1.9.2 would allow us to upgrade to cucumber 2.x, we can't upgrade to cucumber 3.x until we drop Ruby 1.9.3, 2.0 and 2.1.
• Inability to take advantage of new Ruby features. For example, it would be really nice to take advantage of Ruby 2's keyword args (especially required keyword args available in ruby 2.1+). We can't do that until we've dropped support for ruby < 2.1.

Maybe you haven't felt that continuing to support old versions of Ruby is burdensome, but I certainly have.

[xaviershay]

FWIW in practice I've found it pretty painful supporting old Ruby versions. Particularly when I was working actively with mocks.

[JonRowe]

Thanks for actually trying it, @iGEL :). Seems like we don't have to worry about bundler auto-upgrading users to a version they can't use as long as we declare required_ruby_version correctly.

Our own builds say otherwise.

From https://travis-ci.org/rspec/rspec-expectations/jobs/323984512

We, by merit of being one of the most popular testing framework have to be the lowest bar when it comes to Ruby support. If we drop a Ruby version we are effectively orphaning everyone on that Ruby version that depends on us. Currently we are able to say "upgrade to the latest version of RSpec" for a lot of enquiries from older versions, if we start dropping Ruby versions we may well have to start back porting bug fixes to the maintenance branches. I feel that burden outweighs the burden of supporting multiple older Rubies on the current branch.

Especially when we don't have any stats of how many people actively use RSpec on those versions, if rubygems was able to provide us that I'd feel more confident we could do this safely.

[tomstuart]

Speaking only as a grateful user of RSpec who otherwise has no skin in the game:

I absolutely agree that you should find a way to gracefully drop support for EOL Ruby versions. I wish you had unlimited resources but I understand that you don’t. Your time is limited and precious, so please use it to maintain & improve RSpec for the vast majority of Ruby users.

However, I want to say that I do disagree with the point about “community expectations” around the content, significance and rarity of major RSpec releases. It’s a somewhat circular argument: yes, at the moment it’s years between major RSpec versions, so they are de facto A Big Deal. But if you’re going to change something, that is an essentially harmless thing to change, versus violating the assumption that new minor releases don’t break compatibility. The scope of the “RSpec major releases are a big deal” expectation is only as large as the more engaged portion of the RSpec userbase; the scope of “minor releases don’t break things” is orders of magnitude larger, so it’s much harder to push against and, crucially, not something that you can ever change outside of your own project. But you can easily choose to cut major RSpec versions more often, and everyone will just get used to it.

In summary, my gut feeling is that you’ll piss more people off by dropping Ruby support in minor releases than you will by simply releasing more frequent major versions of RSpec. People would quickly adapt to the latter, especially given the changes proposed in rspec/rspec-rails#1967.

[JonRowe]

FWIW I would be happy dropping EOL ruby in major releases.

[fables-tales]

late to chime in here, but I also would prefer that dropping a ruby version was a major, and that we accept that RSpec majors will happen yearly with Ruby version releases.

[myronmarston]

Thanks for chiming in @tomstuart!

Regarding cutting major releases more frequently, I have a few concerns:

• I think users expect a major release to have some useful new features (which are hopefully in some sense more "major" than features we added in minor releases). I don't think users are going to be very motivated to upgrade if we have major releases that are essentially just dropping old ruby versions and/or some other breaking changes. It would feel like upgrade pain for no gain. I don't think we have the capacity to provide significant enough features to support frequent major releases.
• Currently major releases mean "breaking changes", and users know to expect it will not be as easy as upgrading a minor version. Given the relative infrequency of major releases, that has meant that RSpec has become extremely stable for users. Spec suites written against 3.0 (released in June 2014) still work entirely as-is with no changes, nearly 4 years later. Moving to a frequent major release schedule means one of two things:
  • Frequent breaking changes. (I think no one wants this).
  • Infrequent breaking changes (like now), but confusion about when breaking API changes will occur. If we release a major version each year, and they don't generally have breaking changes, then the major version number stops meaning "contains breaking API changes" and it'll cause confusion about when an RSpec release will contain a breaking change.
• If we do yearly major releases, it's going to be harder to get support from publishers like PragProg to support the development and publication of new books (or even new editions of books like my own). And since the market value of a book about RSpec X drops as soon as RSpec (X + 1) is released, due to the perception of being "out of date", it becomes harder for authors of such books to get enough income from the book to warrant writing or updating it. Having spent > 1000 hours working on such a book, I'm very hesitant to move to a policy that quickly gives the impression of it being out of date. The content of a book on RSpec 3 might be totally current and useful against RSpec 5 under this new policy, but a newcomer to the ecosystem who wants to learn RSpec isn't likely to want to give the book a shot.

In short, I think that yearly major releases for the purpose of dropping support for an EOL'd ruby version totally changes what "major release" means from containing stuff that significantly affects users (breaking changes and notable new features) to something that doesn't affect many users at all (dropping support for a Ruby version few users were using anyway).

In summary, my gut feeling is that you’ll piss more people off by dropping Ruby support in minor releases than you will by simply releasing more frequent major versions of RSpec.

This statement surprises me. Is it common and expected these days that projects only drop support for a language version in a major release? I was under the impression (perhaps mistaken) that RSpec was fairly unique in interpreting SemVer this way. For example, I don't believe Rails follows that (although I might be wrong: someone correct me if so). And as another counterexample, Elixir typically only supports the last couple Erlang/OTP versions, and drops support for old Erlang/OTP versions in nearly every minor release.

More generally, if we're dropping support for a Ruby version that was EOL'd 12 or 24 months ago, are there really many users who remain on those versions while also upgrading to the latest and greatest RSpec release?

[tomstuart]

Thanks @myronmarston, that all makes sense.

I honestly don’t know much about the nuances of people’s expectations around semantic versioning when it comes to versions of foundational pieces like Ruby or Rails (versus conventional dependencies). My guess is that in many cases it may not be more nuanced than “I expect to be able to upgrade to a new minor version without anything breaking”, which would inherently include continuing compatibility with Ruby/Rails/whatever, but I don’t have a good sense of whether that’s a reasonable or realistic expectation in general. Hopefully others can clarify this.

Regarding useful new features in new major releases: is it possible that some major release features (e.g. performance? API niceties?) could be enabled by dropping support for old Ruby versions?

[mudge]

As someone who has benefited hugely from RSpec's generous backward-compatibility with Ruby versions (e.g. in re2, embiggen, fieldhand, etc.) and has been unpleasantly surprised when other programming languages' test frameworks don't follow suit, I'd second @tomstuart's recommendation that RSpec should drop language support in a major version, not minor.

At the moment, I specify RSpec versions with an optimistic version constraint of, say, ~> 3.2 or ~> 3.7 expecting that I can happily upgrade that in any of my gems without detriment. If, say, RSpec 3.8 were to drop support for Ruby < 2.0, I'd need to switch to pessimistic version constraints (and incur the wrath of a warning from RubyGems in doing so) if I wanted to continue supporting older Ruby versions.

Of course, I happily support RSpec's decision to drop support for older Ruby versions due to the maintenance burden but I would definitely be surprised if this wasn't consider a major, breaking change.

Having said all that, I think there is a very real tension here around financially supporting the maintenance of RSpec itself: that is, the major version of RSpec is used not only for prosaic dependency management reasons but also in order to market RSpec itself and books on the subject.

Could this tension be alleviated by having some concept above major version numbers that communicates the amount of change in RSpec itself?

It's not quite right but I'm thinking of the recent introduction of "editions" in Rust where the stability of Rust version numbers is divorced from major changes to the language itself. Would it help if you were still free to pen "Effective Testing with RSpec 2018" where "RSpec 2018" describes an "edition" of RSpec even if the major version number is RSpec 6?

[lamont-granquist]

RSpec shouldn't have to support any version of ruby which is EOL'd. There really shouldn't be any time period after that, since those ruby versions have fallen off of security patch support and that means that someone is having to run potentially vulnerable instances of software in order to test and ship. Users that expect support after ruby versions are EOL are implicitly demanding that someone run potentially insecure infrastructure for them. That really shouldn't be allowed to happen.

Also 👍 on interpreting SemVer to mean that you can define your own support policy around when you drop ruby language support and don't necessarily have to only drop language versions on major version bumps. Ideally SemVer would be clarified on this point (and I think if it does go towards mandating major version bumps to drop language version support that SemVer should be dropped or forked because that produces perverse outcomes of requiring major version bumps only to drop language support).

The fact that RSpec still supports 1.8.7 and 1.9 today is an indication this policy has to change. That's a terrible thing to do to the authors.

[JonRowe]

My guess is that in many cases it may not be more nuanced than “I expect to be able to upgrade to a new minor version without anything breaking”, which would inherently include continuing compatibility with Ruby/Rails/whatever

Very much this. Its surprising behaviour much further away from the norm than just increasing major releases more frequently. I think people would be fine if we did this every year when Ruby increases in version.

If we do yearly major releases, it's going to be harder to get support from publishers like PragProg to support the development and publication of new books (or even new editions of books like my own). And since the market value of a book about RSpec X drops as soon as RSpec (X + 1) is released, due to the perception of being "out of date", it becomes harder for authors of such books to get enough income from the book to warrant writing or updating it.

Not having any author experience myself (I think you're the only current team member that has authored a book on RSpec) I feel thats flawed, there shouldn't be an expectation from the publisher to stall or affect development just because a book has been released, after all publishers don't use RSpec, the community does.

In any case in other eco systems an update is seen as a chance to update and re-market a book, e.g. Rails 3 in action was updated for Rails 4 and used to market itself as a whole new book. With coordination with the publisher it should be easy to update a book from RSpec 3 and RSpec 4 or 5, even easier with less major features.

Given that we release features in minor releases books are already out of date anyway, missing functionality that now exists. All releasing a major version does is highlight this fact.

[lamont-granquist]

It looks like bundler+rubygems may still not strip out transitive deps which no longer support the ruby version being used / in the Gemfile: rubygems/bundler#6471

[JonRowe]

They don't, they break our builds.

[deivid-rodriguez]

As a casual RSpec contributor, I'd like old version support to be dropped soon, so dependencies can be upgraded and I don't need to learn a 3 years old, almost 3 majors old, cucumber version to contribute... 😅.

Personally I agree with @myronmarston's proposal (drop EOL'd rubies support on minors), but I don't mind the alternative as long as the major version cycle becomes a bit more agile! :)

Thanks for RSpec to older and newer maintainers, by the way ❤️.

[deivid-rodriguez]

I'm fixing the issue reported by @lamont-granquist in rubygems/bundler#6728. Hopefully the required_ruby_version feature will be more trusted more in the future, it works! :)

[JonRowe]

Given that Rubygems and Bundler are now dropping Rubies < 2.3, I think its getting to the point where we should too. For RSpec 4 though there a few things I'd like to have:

• Our own "object" differ
• Parallel specs available out of the box
• Take the old should syntax into its own gem
• Take out the monkey patch mode

Probably deserves its own issue for such plans

[matkoniecz]

If we drop a Ruby version we are effectively orphaning everyone on that Ruby version that depends on us.

What should be OK for EOLed versions. As a very minor and unimportant user I am perfectly fine with EOL version being unsupported, and I consider weird that EOL version would be supported for months or longer after it is EOL.

And if you want given Ruby version to be supported for longer than you should either volunteer for maintenance or fund someone who will do this)

[pirj]

One way around dropping support for EOL'ed Rubies and only doing this in major versions is to release major versions once a year. E.g Ruby 2.4 is EOL in April 2020, RSpec 5 released in December 2021 and drops support for it. So:

RSpec version	Release date	Supported Ruby versions
RSpec 4	Dec 2020	2.3 - 3.0 (2.3 amnestied)
RSpec 5	Dec 2021	2.5 - 3.1
RSpec 6	Dec 2022	2.6 - 3.2
RSpec 7	Dec 2023	2.7 - 3.3
RSpec 8	Dec 2024	3.0 - 3.4

This way, it will be still possible to update to RSpec 5 in December 2021 even though 2.5 is EOL in April 2021, and receive all RSpec fixes until December 2022, 1.5 years after Ruby 2.5 is EOL.

I remember there is a policy to only support the latest major version. This relaxes constraints of supporting deprecations.
To be fair, we don't have too many deprecations in RSpec 3 that we plan to drop in RSpec 4, off the top of my head:

• monkey-patching should_receive/should/... with an explicit receiver
• monkey-patching mode
• shared_context_metadata_behavior
• few minor in Mocks/Expectations
• implicit block syntax expectations

Probably even less than that in RSpec 5.

[JonRowe]

RSpec supports Ruby for longer than Ruby does because real people don't always upgrade. I'm not incrementing RSpec every year just to drop Rubies, especially when they don't cause us that much trouble.

[luke-hill]

@myronmarston - Minor thing but you say

We currently support 10 6 versions of MRI:

Technically you support every version as per rubygems. Required ruby is >= 0. Could we add a small patch to the next release that sets the minimum version to whatever it is meant to be

[JonRowe]

The next major version will do so.