Include tests in source distributions

[sbraz]

Hi,
We at Gentoo rely on PyPI tarballs to build docs and run tests, could you include them in future releases so that we don't have to use GitHub's tarballs?

[tartley]

Hey. FYI, yesterday I created a PR to test releases before we push them to PyPI. When that is merged, I'll be more confident about resuming merges and releases. I'll try to look at this PR soon. Thank you for creating it!

[sbraz]

Hi @tartley. If your PR works, it means tests are included so my PR would not be needed :)

[tartley]

Understood, thanks for the clarification. I'll double check when I'm not meant to be doing my real job... :-)

[tartley]

Apologies for being AWOL forever. Interesting to hear that you want this feature. It has been my instinct not to include tests. I shall sleep on this and reconsider. Be aware, there are changes afoot that may affect this, see #293 .

[sbraz]

No worries, it's not a massive problem. We (Gentoo) simply use GitHub releases instead of PyPI packages for now. Howevern, it is good practice to include tests in sdists if they aren't too large. I know Debian also relies on this to test packages.

[tartley]

Thank you for the response. I confess I do not yet understand why it's good to bundle tests in sdists. If such a test fails for you, but passes for me (the only scenario when it is useful to have bundled them) then that (to my way of thinking) indicates that it's a poor test, i.e. depends on something contextual that isn't captured in the repo, breaking all the invariants I thought we relied upon - eg. to see a failing test start passing by editing something that is captured in the repo.

I guess maybe when the test fails for you, it's a sign that whatever made it fail possibly ought to be something that is captured / controlled by / compensated for by something that ought to be in the repo, but isn't yet. (or it's a bad test)

I'll try to think harder about how I'm wrong about this, see if I can read around it or steelman myself into understanding it. Thanks for raising the issue.

[sbraz]

If such a test fails for you, but passes for me (the only scenario when it is useful to have bundled them) then that (to my way of thinking) indicates that it's a poor test, i.e. depends on something contextual that isn't captured in the repo.

There are many reasons for failing tests downstream. Here are a few I can think of:

1. missing dependencies (maybe not declared where we expected them to be or missed by us)
2. wrong version for some dependencies (we don't usually pin versions like upstreams do because it makes dependency handling harder for the whole system)
3. of course actual bugs in packages

The main purpose of tests for us are to make sure that we aren't shipping a broken package for some reason (points 1 and 2).

But of course, we often find actual bugs that you might not have noticed (because you're not running the same Python version, because of different non-Python dependencies, etc.). All this makes the test suites very valuable to us (and indirectly to you because we can help you find bugs 😃).

[tartley]

OK then. Thank you for expanding on the idea. I'll continue to educate myself about it and see whether we can incorporate it here.

[mtelka]

+1 for including tests in sdist (including the tox.ini file). OpenIndiana have similar workflow like Gentoo (and others mentioned above) so tests in sdist would make the world better. Thank you.