SeleniumBase's pytest dependency is pinned to v5.3.5

[aklajnert]

This is what happens when I run pip install pytest seleniumbase:

ERROR: seleniumbase 1.36.9 has requirement pytest==5.3.5; python_version >= "3.5", but you'll have pytest 5.4.1 which is incompatible.

I'm actually a bit surprised by the pytest==5.3.5 constraint. Is there a reason for that?

[mdmintz]

@aklajnert There are several regressions in pytest after v5.3.5 (that were introduced in pytest 5.4.0), which has forced me to hardcode pytest==5.3.5 into SeleniumBase for now (since https://github.com/seleniumbase/SeleniumBase/releases/tag/v1.35.9)

These regressions include:

• self._outcome.errors no longer showing errors in tearDown() methods for failing tests, which is needed for extracting errors to log files and determining when to attach screenshots to pytest html reports.
• Complaints about the new color scheme, which leaves some console output unreadable in certain terminals, especially ones with colors.
• Multi-threaded test issues (pytest fails to run with pytest-xdist pytest-dev/pytest#6925)
• --last-failed is broken for parametrized and class tests (--last-failed is broken for parametrized and class tests since 5.4 pytest-dev/pytest#6968)
• unittest.TestCase cleanup issues (unittest.TestCase cleanup functions not invoked on test failure pytest-dev/pytest#6947)

In the 2 days after pytest 5.4.0 was released, I received a record number of complaints, so that's why SeleniumBase will be using pytest==5.3.5 for the foreseeable future. I do have high hopes that we'll be back to using the latest version of pytest once their team has resolved their issues.

[mdmintz]

I've opened pytest-dev/pytest#7000 so that the pytest people can take a look at the issue with self._outcome.errors.

[blueyed]

pytest-dev/pytest#7000 is caused by pytest-dev/pytest#6947 also.

[aklajnert]

@mdmintz thanks for the explanation.
I wonder if it wouldn't be better to slightly relax the requirements from pytest==5.3.5 to pytest<=5.3.5 before pytest will fix those issues?

[blueyed]

@aklajnert thought the same, yes. Could be <=5.4 though then (although no 5.3 release is likely to happen).

[mdmintz]

@aklajnert @blueyed
Due to issues such as pytest-dev/pytest#6517 from pytest 5.3.4 and other issues from earlier versions, I cannot safely set the pytest requirement to allow less than 5.3.5, and due to the new issues, I cannot set the requirements to allow greater than pytest 5.3.5 either, so it's pinned for now, just like a chess piece. If you need a temporary workaround, fork/clone SeleniumBase and set the requirements that you need. My main goal is to make sure everything works, and with pytest 5.3.5 there have been no reports of any unusual pytest issues, which makes it the perfect version right now.

[aklajnert]

@mdmintz thanks for clarifying, I really appreciate the great work you put into the SeleniumBase framework.

[mdmintz]

@aklajnert Today I found out that 3 key members of the pytest organization have quit unexpectedly. There's a Tweet about it here: https://twitter.com/brettsky/status/1248733160591908864?s=20
At this point I'm not counting on any future pytest updates to fix the issues above from v5.4.0 and onward, at least not without introducing more issues that may be problematic.
This means that pinning SeleniumBase to pytest v5.3.5 is more likely a feature than an issue, even though it's not the latest version. I consider pytest v5.3.5 to be the "perfect" version, as there have been no pytest issues reported from SeleniumBase users with it. I'll likely make a note of this somewhere, and close this ticket soon as the final result.

[aklajnert]

That's really sad. I wonder if having the pytest as a requirement is actually necessary for SeleniumBase. I do use it only with pytest, but as it can be used with nose (and probably unittest), I'm not sure if the pytest should be the requirement.
Also, probably not all use cases are affected by the pytest bugs. Maybe it would be possible to display a warning that the pytest version may be causing trouble and the recommended one is 5.3.5. I think people still are able to install a different version of pytest when they have SeleniumBase already installed and will do pytest update.

[mdmintz]

@aklajnert pytest provides SeleniumBase with the following important features:

• 1: Command-line argument parsing, where most of configuration is set.
• 2: The pytest-html reports plugin, which generates test reports and attaches screenshots to failing tests. (Usage: --html=report.html)
• 3: The pytest-xdist plugin, which allows you to run multiple tests at the same time by multi-threading them. (Usage: -n=NUM)
• 4: The pytest-ordering plugin, which lets you choose the order that your tests run in.
• 5: The pytest-rerunfailures plugin, which lets you rerun flaky tests. (Usage: --reruns=NUM)

The Nose test runner provides only the first of that list (command-line argument parsing). If work on pytest stops entirely due to the situation mentioned earlier, it will become like Nose... meaning no more updates, but the framework is still usable. pytest 5.3.5 has everything that SeleniumBase needs. Upgrading to the latest version of pytest right now breaks SeleniumBase functionality, so there's no reason to give users the option of using any version other than pytest 5.3.5. Most people using seleniumbase do not set a specific version of it in their requirements.txt or setup.py files, meaning that it's up to me to make sure that the latest version SeleniumBase works perfectly whenever they install it.

Corporations are using SeleniumBase in their deployment cycles (see the small list I've created: https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/happy_customers.md). Universities are starting to teach SeleniumBase in their classes (such as www.catalog.msstate.edu/search/?P=CSE%204283 , where students are using seleniumbase in their projects for assignments, eg: https://github.com/abbyg41/assignment3-4 and https://github.com/SpencerWBarnes/QA-Assignment (see the requirements files and their tests).

To keep SeleniumBase working perfectly, I've locked down most of the Python dependencies so that I get a chance to try never versions of those dependencies first. That brings us back to why pytest is pinned to pytest 5.3.5, rather than 5.4.0 or 5.4.1, which breaks unittest functionality. SeleniumBase's BaseCase class inherits unittest.TestCase, so that's a problem, but only if users have the option to install pytest 5.4.0 or 5.4.1. The SeleniumBase requirements file and setup file now pin pytest to pytest 5.3.5, so that's no longer an issue.

I've been in the corporate world for a long time, and I've learned a few things about building reliable frameworks. One big thing is that loose dependencies lead to loose results (AKA bad surprises), which means that it's OK to not use the latest version of a package/dependency. That's especially true in this case with pytest.

[aklajnert]

OK, that explains everything. I don't know the SeleniumBase internals at all, but I had the impression that it doesn't need pytest to work. If you rely on pytest, then it makes perfect sense to pin the desired version. Thanks.

[mdmintz]

@aklajnert SeleniumBase is using the latest version of pytest again (5.4.2) as of seleniumbase 1.38.0, just released: https://github.com/seleniumbase/SeleniumBase/releases/tag/v1.38.0