Drop eol and near eol versions of PHP

[aidantwoods]

Per http://php.net/supported-versions.php PHP 7.0 is in security fix only mode and will cease to be supported in 7 months.

This PR is a proposal to start 2.0 from the latest actively supported PHP release, 7.1.

---

If merged, closes #619

[aidantwoods]

See also: On PHP Version Requirements.

[PhrozenByte]

I strongly oppose this.

1. Even though a PHP version might not be supported by the PHP team itself anymore, it might still be supported by (mostly enterprise/server) Linux distributions, e.g. Debian, Red Hat, SLES. PHP 5.6 and 7.0 are still officially supported and are very common.

2. Since Parsedown hasn't the man-power to support multiple major versions at a time, we must be as circumspect as possible. Parsedown is a library. If we raise Parsedown's requirements, we inevitably cut-off any depending project from Parsedown updates if they have lower requirements (i.e. most depending projects). Raising the requirements forces depending software to release a new major version of their software - something we can't realistically demand. In practice they simply won't upgrade to Parsedown 2.0 until they make the decision to release a new major version.

3. IMO there should always be a strong technical reason to drop support for older versions. New software should always be written using the latest technology, but existing and wide-spread software (libraries in particular) unfortunately requires some conservativism.

[aidantwoods]

I'll concede that drawing the line at 7.1 is pretty aggressive (hence why this is a PR and not a commit :) )

Do you have similar objections to instead drawing the line at 5.6? (is PHP 5.3 still maintained by anyone? What about 5.4, or 5.5?). I think dropping near EOL is aggressive, but dropping EOL is just sensible.

In practice they simply won't upgrade to Parsedown 2.0 until they make the decision to release a new major version.

Bare in mind that this is our new major version. We don't get this decision again until we release a new major version. If Parsedown 1.* is anything to go by – we'll probably be supporting whatever versions we select here for a long time.

If we raise Parsedown's requirements, we inevitably cut-off any depending project from Parsedown updates [...]

1. This is a major update, so we don't do this behind the scenes.
2. There is a section in the post I linked to that addresses whether we are abandoning users.

IMO there should always be a strong technical reason to drop support for older versions

1. PHP 5.3 in particular is a real pain to support (see: one of many silly PHP 5.3 specific workarounds)
2. IMO we shouldn't give users a reason to keep using EOL software.

[aidantwoods]

Neither this nor #619 need to be merged – these are created because I think this is a topic worth at least discussing :) (even if neither end up being merged)

[PhrozenByte]

Do you have similar objections to instead drawing the line at 5.6? (is PHP 5.3 still maintained by anyone? What about 5.4, or 5.5?). I think dropping near EOL is aggressive, but dropping EOL is just sensible.

I can't really draw a exact line, it's a consideration. For example, the latest RHEL 7 comes with PHP 5.4.16 (even though you can install a newer PHP version, they guarantee bug fixes for PHP 5.4 until 2024 and security fixes until 2027 - that's ridiculous, I know). RHEL 5 comes with PHP 5.1 - and is in it's "Extended Lifecycle Support" until 2020.

I don't say that we should support PHP 5.1. My point is that the "it's EOL" argument can't be a valid reason for dropping support, because that's not the full truth. IMO the only valid reason for dropping support of older versions are technical reasons.

This is a major update, so we don't do this behind the scenes.

Being a major release (that is allowed to break BC), doesn't mean that it breaks BC for everyone. For example, if you don't use extensions (or just extensions that aren't affected by the latest changes, like Parsedown Extra) and don't have rather "special" contents, virtually nothing changes. The CommonMark specs were written with existing contents and parsers in mind. That's what I meant with "we inevitably cut-off".

There is a section in the post I linked to that addresses whether we are abandoning users.
IMO we shouldn't give users a reason to keep using EOL software.

It's the same as with #561: Parsedown is a library, we should leave this decision up to the application developer - as far as practicable, at least. That's the major difference between Parsedown being a library and end-user applications: We indeed have users that (hopefully) know what is going on and can make a informed choice on their own. It's about flexibility, not about encouriging users to do stupid things (like using a PHP version that is neither supported by PHP, nor their Linux distribution).

PHP 5.3 in particular is a real pain to support (see: one of many silly PHP 5.3 specific workarounds)

This might be one of those technical reasons. There was a major step between PHP 5.3 and 5.4, it might be reasonable to draw the line there. Maybe we can get some more opinions about that?

[aidantwoods]

IMO the only valid reason for dropping support of older versions are technical reasons.

Here are a few :)

• Return types in PHP 7: up until this point there are zero type system guarantees about what objects you will receive when calling a function
• Scalar type hints in PHP 7: up until this point there is no way to assert the scalar types that we want to accept in a function unless we manually check them in the function body. Not being able to assert that you only want string as a variable is extremely annoying.
• strict_types=1 in PHP 7: PHPs implicit type casts is mostly a headache which hides bugs in code
• Type hint collections with variadics using the ... operator from PHP 5.6: this allows us to formally accept ordered collections of objects in a type safe way

These are all especially useful if we want to define interfaces

[PhrozenByte]

These aren't technical reasons, but just a list of new features. Its use must either be required to implement something, or the alternative solution must significantly increase code's complexity, readability and/or costs, to be a technical reason. All you've mentioned is undoubtfully nice to have, but not required at all.

[aidantwoods]

Strictly speaking we should be manually checking the return type of every function call and check the argument types being accepted in the body of every function – otherwise we're assuming types without guarentees, and this can go wrong.

For example, the Parsedown wiki has apparently had a type related bug in an example extension which causes a loop to never end under certain circumstances due to PHPs implicit type casting.

See: #613, and #546 (comment)

Had types been validated, this error would have presented itself as a type error and not a hard to find infinite while loop.

[PhrozenByte]

Might be true for the public API, but since we're planning to remove the support of extensions extending the Parsedown class and rather implementing a API anyway, this won't be necessary anymore (by the way: I was about to suggest using a class for AST elements anyway, arrays require too many isset() checks).

Don't get me wrong, I'm not opposing the usefulness of these features, they just aren't really required. Again, it's a matter of consideration. Don't take it to the extreme, common sense applies. If the interpretation (of what a "technical reason" is) is too strict, we end up supporting everything since PHP 4.0. If the interpretation is to loose, we end up supporting just today's nightly build of PHP.

The truth is somewhere between PHP 5.3 and 5.6 - and due to everything mentioned before, the new requirement should be closer to PHP 5.3 than 5.6.

[aidantwoods]

(by the way: I was about to suggest using a class for AST elements anyway, arrays require too many isset() checks).

Indeed 😉

Don't get me wrong, I'm not opposing the usefulness of these features, they just aren't really required. Again, it's a matter of consideration. Don't take it to the extreme, common sense applies. If the interpretation (of what a "technical reason" is) is too strict, we end up supporting everything since PHP 4.0. If the interpretation is to loose, we end up supporting just today's nightly build of PHP.

Yup 👍 I'm only making my case for why we might want these features – it's perfectly okay if we arrive at the conclusion that these aren't enough and don't outweigh cost of dropping support (just trying to put everything on the table as it were).

I think the case for types still has merit because even without extensions we will still be interacting with code for plugins (ideally we can make it really hard to get into situations where bugs are silently swept under the rug by PHP).
That said, we can implement something like this in worst case if we are supporting pre PHP 7, and declare the PHP docblock to define public API with respect to function signatures.

[aidantwoods]

Some stats from Packagist on PHP version usage as of November 2017:

https://seld.be/notes/php-versions-stats-2017-2-edition

Version	Usage	Diff from May 2017
PHP 7.1	36.63%	(+18.99)
PHP 7.0	30.76%	(-5.36)
PHP 5.6	23.28%	(-8.16)
PHP 5.5	6.11%	(-4.5)
PHP 5.4	1.51%	(-1.6)
PHP 5.3	0.76%	(-0.22)

Obviously this isn't necessarily representative of the minimum supported version of packages.

[aidantwoods]

cc @rhukster I assume you'd be for maintaining compatibility with PHP 5.5 and up? (I note that Grav requires >=5.5.9), or are there to raise this in the near future?

[aidantwoods]

I surveyed our dependencies according to Packagist. I looked for the latest stable release of each package which depends on Parsedown (according to https://packagist.org/packages/erusev/parsedown/dependents), and then found the corresponding minimum PHP minor version from the "require": {"php": ...} key using the Packagist API.

First the minimum PHP version the package supports weighted by number of monthly downloads of each package – this should give an overall rough "size" weighting of each PHP version:

PHP Version	Usage (downloads of stable dependents)
4.0	0%
5.3	1.85%
5.4	1.54%
5.5	7.35%
5.6	5.44%
7.0	2.13%
7.1	81.69%

Note that monthly downloads does not necessarily represent downloads of the latest version but rather downloads of the package (unfortunately Packagist doesn't appear to provide this info in the API – please let me know if I missed it though).

The following is a breakdown of stable packages that depend on Parsedown (but without regard to monthly downloads)

Total stable dependents: 208

PHP Version	Usage (stable dependents)
4.0	0.48%
5.3	13.94%
5.4	16.35%
5.5	23.08%
5.6	23.08%
7.0	15.38%
7.1	7.69%

If you want to see the exact methodology for these results, please see these very hacky scripts: https://gist.github.com/aidantwoods/98d01f435d45321e10b71cc7fd224bfd

---

Excluding data from laravel, the charts are as follows (this maybe gives a better idea about the "average" package that uses Parsedown):

PHP Version	Usage (downloads of stable dependents)
4.0	0%
5.3	8.59%
5.4	7.17%
5.5	34.09%
5.6	25.24%
7.0	9.89%
7.1	15.02%

[aidantwoods]

So excluding any data from laravel (which supports 7.1 and later and has a disproportionate number of users compared to other dependencies), just over 50% of our indirect users are using us via a project that supports PHP 5.6 or later. (If we include laravel that number is almost 90%)

It's also worth noting that PHP 5.6 was shipped in Ubuntu 16.04 LTS, and that the latest LTS iteration that would have shipped with 5.5 is 14.04 LTS – which will itself hit EOL within a year. PHP 5.5 was EOL'd by the PHP team almost 2 years ago.

I also note that Grav (mentioned earlier in the thread as having a 5.5 requirement) is planning to move to 5.6 in their next release (getgrav/grav#1977).

Looking at the Packagist user specific data, it is also clear that very few people are still using PHP 5.5 and below, (and there is in-fact a major push to PHP 7.1). This indicates that it will not be hard for dependencies to upgrade because most of their users are likely already using 5.6 and above. Yes it will probably require a major release, but raising the PHP requirement is something they will have inevitably to do at some point, so we're not exactly hurting them by giving them a reason to upgrade. If they're not already thinking about it, maybe we'll be the reason they do.

On top of all this, by drawing the line at 5.6, we'd also gain the ... operator which would let us accept collections of objects in a type safe manner – which we will have to do: e.g. a collection of AST Element objects. The alternative to using the language syntax is to manually verify elements in an array. Yes, we could write a compact (to call) helper function, but using a built-in language feature is so much more readable, and likely faster.

So to summarise, if we were to draw a line at 5.6:

• We directly benefit from new features
  • Additionally by dropping 5.3 we also avoid a host of problems and stupid workarounds (e.g. 1, 2, 3, ... to name a few)
• Most users have already moved into this support range (see Packagist user data)
• Our largest dependencies already support 5.6 and onwards (see Packagist dependency data)
• The dependencies that support pre 5.6 can likely raise their requirement without losing many users (see Packagist user data), as mentioned: they will have to do this eventually anyway (and sooner rather than later).

What happens to dependencies that support pre PHP 5.6?

In my mind, they should release a new major version starting from a non EOL'd PHP version.
Not only are most of their users already using one (see Packagist data), but they gain access to new features, as well as being another reason that their users (or their hosting providers) should install safe versions of PHP: within a year Ubuntu looks like it will cease support of PHP 5.5, the PHP team did so almost two years ago. If users haven't got a plan for upgrading: they're running out of time (or may have already).

But I use $os and it still supports $oldPHPVersion:

There might be other OSs that support (and only support) PHP 5.5 and below for longer, but let's be realistic: without the PHP team supporting these, you're likely almost exclusively getting any security updates from backports. It is unlikely anyone willing to report bugs will be looking for ones specific to PHP 5.5 and below. Much less will they know who to report these bugs to: they can't really report them to teams that don't support that version. Are they going to report them to all the minor OSs supporting only these old versions individually?

For users on OSs that offer later versions: guess what? To get new features you need to install updates (update your PHP installation – you'll not only be able to use newer software on it, but it'll be much safer too). For users on OSs that don't: they should update – or change – their OS.

[PhrozenByte]

You're virtually arguing on the basis of usage statistics, the only problem is that the statistics you're using aren't relevant at all. It's like arguing that Windows is installed on roughly 90% of all computers, thus we don't have to support Linux 😉

The biggest problem is that Packagist is extremely rarely used to distribute software to end-users. I'm not sure why Grav is getting the "paradigm project", but just take Grav as an example: the usual way to install Grav is by using their pre-built package. These installations won't show up on Packagist. You can easily verify that by looking at their install count on Packagist: Do you really think Grav was installed just 22k times?

This leads us to another problem: Even the people that do use composer, usually don't use it on their productive server. They rather run composer it in their dev environment. Thus you're not getting statistics about the environments Parsedown is actually used on, but about dev environments.

The third problem is that you assume that all projects that depend on Parsedown use composer. Given that there are several Issues here on GitHub dealing with people downloading just the Parsedown.php, this assumption is insupportable.

And there's yet another problem: Many projects that depend on Parsedown aren't even end-user applications, they are frameworks - just like Laravel, Laravel surely isn't the only one. You can't take any assumption on how these downloads correspond to actual installations of some end-user software.

In my mind, they should release a new major version starting from a non EOL'd PHP version.

By assuming that Parsedown might be the reason for application developers to release a new major version of their software and raising their requirements because of Parsedown is pretty... daring 😆 Since Parsedown isn't really security-relevant (not least because Parsedown wasn't able to deal with untrusted content until just a few weeks ago) they simply won't update Parsedown. This is bad, but this is reality - the bad reality.

But I use $os and it still supports $oldPHPVersion:

So, you think how Ubuntu deals with PHP updates represents how enterprises like Red Hat deal with them? Red Hat plays in another league. RHEL and CentOS are one of the, if not the most wide-spread server distributions. We're not talking about "minor OSs" here. 😕

Additionally by dropping 5.3 we also avoid a host of problems and stupid workarounds

As I've mentioned already there might be good technical reasons to drop support of PHP 5.3. However, there unfortunately wasn't a single persuasive argumen why PHP 5.4 and 5.5 should be dropped yet.

[aidantwoods]

You're virtually arguing on the basis of usage statistics[...]It's like arguing that Windows is installed on roughly 90% of all computers, thus we don't have to support Linux 😉

I'm not just arguing on the most popular PHP version, but the minimum supported version statistics of our dependencies. Furthermore I am arguing based on the presence of new features that will be beneficial for us to use. I make a point about metrics to discuss likely impact.

Also, literally one of your first points was about cutting off too many users/projects. Like, what?

On the basis of new features alone I'd love to just drop everything below 7.1, the usage metrics are helpful to see why we should support versions prior to that.

[...] the only problem is that the statistics you're using aren't relevant at all.[...]

Really? Our dependencies are not at all relevant?

I'm not sure why Grav is getting the "paradigm project"

I only mentioned this since I referred to it earlier in the thread as requiring 5.5 (when in-fact it is moving to requiring 5.6... and then to 7.1 in their next update for what it's worth – but I digress).

The biggest problem is that Packagist is extremely rarely used to distribute software to end-users. [...]
The third problem is that you assume that all projects that depend on Parsedown use composer [...]

Of course this is just a sample. When did I say I assume absolutely everyone uses composer & packagist? Unfortunately unless you can provide additional metrics on direct installations, discussing them is just a game of "what if's?" and guessing. I would expect the distributions of PHP versions to be similar – but who knows.
Maybe in 10 years we still won't know the answer, but we better support PHP 5.3 just incase projects are still using it that we don't know about, right? Why don't we just never move forward at all. Let's just do bugfixes so we don't need the new features from new PHP versions to make the codebase better... does this sound like a good idea?

By assuming that Parsedown might be the reason for application developers to release a new major version of their software and raising their requirements because of Parsedown is pretty... daring 😆

Updating their requirements is something they have to do eventually. I doubt that Parsedown will cause them to do it. This is why I mention EOL and security. Most projects do (or should) care about security, by extension they should care if their users are running EOL software.
That said: the majority of projects by user mass in the sample will not have to update their requirements (this is the point in the metrics: to get an idea of impact).

So, you think how Ubuntu deals with PHP updates represents how enterprises like Red Hat deal with them? Red Hat plays in another league. RHEL and CentOS are one of the, if not the most wide-spread server distributions. We're not talking about "minor OSs" here. 😕

No. The comment about minor OSs was specifically addressing the case of OSs that support and only support PHP 5.5 and below. If an OS supports a later PHP version, then it is a minor upgrade away and the user should upgrade.

It doesn't matter what "league" they play in: core PHP development has moved on.

---

TL;DR: I think Parsedown will be better when using new PHP features. Most people appear in a sample to be using compatible versions of PHP. Most of our dependencies appear in a sample not to have to do anything to their min PHP version anyway. For the ones that do, they're likely already moving in that direction because 5.6 (a minor update from 5.x) was released almost 4 years ago.

Other than we leave a few people who can't update right now behind, is there any argument to support pre 5.6?

[aidantwoods]

there unfortunately wasn't a single persuasive argumen why PHP 5.4 and 5.5 should be dropped yet.

PHP 5.6 has type hinted variadics that facilitate type safe collections. Why should we keep them?

[PhrozenByte]

Why don't we just never move forward at all.

You're not getting the point. The point is that you can't use usage statistics to reason dropping support of platforms. Never, no matter what metrics you use. There is only one valid reason to drop support: Technical reason. There are indeed technical reasons to drop support of PHP 5.3, but none for PHP 5.4 and 5.5.

Of course this is just a sample. When did I say I assume absolutely everyone uses composer & packagist? Unfortunately unless you can provide additional metrics on direct installations, discussing them is just a game of "what if's?" and guessing.

It's neither a random nor a (at least assumed) representative sample - and that's the reason why it isn't relevant at all. We know that this sample is highly skewed. This is nothing up to guessing, we know that, as described above (see #618 (comment)). It's exactly like arguing that Parsedown shouldn't support Linux because Windows is installed on 90% of all computers. The problem is not that the metrics are wrong, the problem is that you can't derive your assumptions from the given metrics.

You won't ever get any valid statement from this data about the population, this is statistically impossible. You can only make statements about the sample, but not about the population - because it isn't representative for the population. The only thing you can deduce from the given data is that there are at least 111 projects still supporting PHP 5.3, 5.4 and 5.5. You can deduce that this is more than half of all projects on Packagist (not of all projects using Parsedown!). You can deduce that there were hundreds of thousands (you unfortunately didn't provide the total number of downloads) of downloads on platforms still using PHP 5.3, 5.4 and 5.5. You can make statements about absolute numbers. You can't make statements about usage or usage shares, because this would require you to apply these metrics to the population. You can't deduce anything about the population. Again, this is statistically impossible because we know that the sample is highly skewed.

I'm just trying to tell you that you can't use the given information to make the statements you've made. This is completely independent of whether these statements are relevant for this consideration or not. They are not. Even though the absolute numbers rather support my statement (having a negative impact on more than 100 projects is bad), it is nevertheless irrelevant. Again, the only valid reason to drop support is a technical reason.

The comment about minor OSs was specifically addressing the case of OSs that support and only support PHP 5.5 and below.

Just because a newer version is available doesn't mean that you can (or even should) use a newer version. This might be true for small private websites, but can't be further away from truth for enterprises. Never touch a running system unless you are required to. And there is no reason to do so until you have a technical reason - like that your Linux distribution ends support of your PHP version. However, since enterprise Linux distributions support PHP versions for ages this usually isn't going to happen anytime soon. The most common reason is that enterprises replace old servers with newer ones, using newer software. But this happens just bit by bit.

PHP 5.6 has type hinted variadics that facilitate type safe collections.

See #618 (comment)

[rhukster]

Sorry @aidantwoods to be so late in replying re. Grav, but we are actually moving quite rapidly ourselves to drop EOL PHP releases: https://www.getgrav.org/blog/raising-php-requirements-2018

• Grav 1.5 (PHP 5.6.4+) we plan on releasing in a month or two
• Grav 1.6 (PHP 7.1.3+)should be released before the end of the year.