Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release Pillow 6.1.0 on July 1, 2019 #3835

Closed
22 tasks done
radarhere opened this issue May 8, 2019 · 36 comments
Closed
22 tasks done

Release Pillow 6.1.0 on July 1, 2019 #3835

radarhere opened this issue May 8, 2019 · 36 comments
Assignees
Labels
Projects
Milestone

Comments

@radarhere
Copy link
Member

radarhere commented May 8, 2019

Release notes needed

Release checklist

  • Open a release ticket e.g. Release Pillow 5.2.0 on July 1, 2018 #3154
  • Develop and prepare release in master branch.
  • Check Travis CI and AppVeyor CI to confirm passing tests in master branch.
  • Check that all of the wheel builds Pillow Wheel Builder pass the tests in Travis CI.
  • In compliance with PEP 440, update version identifier in src/PIL/_version.py
  • Update CHANGES.rst.
  • Run pre-release check via make release-test in a freshly cloned repo.
  • Create branch and tag for release e.g.:
    git branch 5.2.x
    git tag 5.2.0
    git push --all
    git push --tags
  • Create source distributions e.g.:
    make sdist
  • Create binary distributions

Windows

Mac and Linux

  • Use the Pillow Wheel Builder:
    git clone https://github.com/python-pillow/pillow-wheels
    cd pillow-wheels
    ./update-pillow-tag.sh [[release tag]]
  • Download distributions from the Pillow Wheel Builder container.
    wget -m -A 'Pillow-<VERSION>*' \
    http://a365fff413fe338398b6-1c8a9b3114517dc5fe17b7c3f8c63a43.r19.cf2.rackcdn.com

  • Upload all binaries and source distributions e.g. twine upload dist/Pillow-5.2.0*
  • Create a new release on GitHub
  • In compliance with PEP 440, increment and append .dev0 to version identifier in src/PIL/_version.py

Publicize Release

Documentation

@radarhere radarhere added this to the 6.1.0 milestone May 8, 2019
@hugovk hugovk pinned this issue May 8, 2019
@aclark4life aclark4life added this to In progress in Pillow May 11, 2019
@hugovk
Copy link
Member

hugovk commented Jun 25, 2019

Less than a week to release!

Let's review some more contributor PRs.

Is there anything still needing release notes?

@aclark4life
Copy link
Member

@hugovk @radarhere Can we start a new thing where whoever is planning to do or lead the release self-assigns themself to these release issues? We now have 4, count them 4 (if you include me) folks capable of doing a release but I've lost track of what our schedule is (if we even have one). With @wiredfool and I lurking, it's typically either @hugovk or @radarhere so let's just figure out what the plan is for say, one year in advance. E.g. every other between @hugovk and @radarhere ? All four of us so as not to overwhelm one developer? I don't have any strong preference just want to firm it up … thanks.

@hugovk hugovk self-assigned this Jun 26, 2019
@hugovk
Copy link
Member

hugovk commented Jun 26, 2019

Sure, @radarhere did the last one, I'm good to do this one.

@radarhere and I have been alternating, which I think's fine, and if there's one we can't do then we can just pipe up in advance and one of the other three can do it. For example, I can't guarantee to be available every single 1st January, because I might be away on holiday and without a computer.

We've been updating the release checklist as needed each release.

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

All merged in for this release. Now waiting for master to turn green on the CIs (AppVeyor is sooo sloww, last 3 were 53/63/47 mins. We need to start shifting it to AZP).

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

(I accidentally pushed 6.2.x to the 5.2.x branch. I've reset this back to the 5.2.0 tag.)


@cgohlke Please could we have Windows binaries for 6.1.0?

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

Some new failures on pillow-wheels, TestImageFont.test_unicode_extended failed:

  • Mac MB_PYTHON_VERSION=2.7
  • Linux MB_PYTHON_VERSION=2.7 UNICODE_WIDTH=16
  • Linux MB_PYTHON_VERSION=2.7 PLAT=i686 UNICODE_WIDTH=16

With:

  • AssertionError: average pixel value difference 14.7190 > epsilon 6.2000
  • AssertionError: average pixel value difference 14.7190 > epsilon 6.2000
  • AssertionError: average pixel value difference 14.7280 > epsilon 6.2000

https://travis-ci.org/python-pillow/pillow-wheels/builds/552752918

image

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

I suggest:

  1. we fix the pillow-wheels build
  2. because the 6.1.0 tag is already pushed, create a new 6.1.1 release with the fix
  3. we don't bother with binaries for 6.1.0 but do 6.1.1 instead

Thoughts?


For 1) see #3931 to loosen the epsilon for FreeType 2.10.

@aclark4life
Copy link
Member

@hugovk How about delete the tag and just release 6.1.0? Any downside to that?

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

That is possible, a downside is the tag is already "out there" and we don't know if anyone has done anything with it already.

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

Could just delete it quickly now and hope for the best?

@aclark4life
Copy link
Member

@hugovk I'm comfortable taking responsibility for any damage done by updating the tag rev. I don't feel like that's as offensive as releasing different versions of 6.1.0 to PyPI (which we'd obviously never do). In other words, the requirement for a point/bug fix release should be that there is an actual release, not just that we've tagged a release. In this case, we just tagged it (right?) and we caught a bug so I think it's OK to delete and re-tag.

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

OK, 6.1.0 tag deleted and will re-tag the new one. Thanks!

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

The new tag is ready and pillow-wheels is green.

Take 2: @cgohlke Please could we have Windows binaries for 6.1.0?

@cgohlke
Copy link
Contributor

cgohlke commented Jul 1, 2019

I'm running into two issues:

  1. pypy3 builds are failing

_imagingft.obj : error LNK2019: unresolved external symbol _PyUnicode_AsUCS4Copy referenced in function _text_layout_raqm

  1. reproducible segfault with heap verification (Déjà vu)
================================================= test session starts =================================================
platform win32 -- Python 3.7.3, pytest-4.6.3, py-1.8.0, pluggy-0.12.0 -- X:\Python37\python.exe
cachedir: .pytest_cache
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('D:\\Build\\Pillow\\Pillow-6.1.0\\.hypothesis\\examples')
rootdir: D:\Build\Pillow\Pillow-6.1.0
plugins: hypothesis-4.24.6, palladium-1.2.1.1, cov-2.7.1, faulthandler-1.6.0, forked-1.0.2, localserver-0.5.0, pep8-1.0.6, xdist-1.29.0
collected 1355 items

Tests/test_000_sanity.py::TestSanity::test_sanity PASSED                                                         [  0%]
Tests/test_binary.py::TestBinary::test_big_endian PASSED                                                         [  0%]
<snip>
Tests/test_image_convert.py::TestImageConvert::test_p_la PASSED                                                  [ 54%]
Tests/test_image_convert.py::TestImageConvert::test_rgba_p PASSED                                                [ 54%]
Tests/test_image_convert.py::TestImageConvert::test_sanity Windows fatal exception: access violation

Current thread 0x00001abc (most recent call first):
  File "X:\Python37\lib\site-packages\PIL\Image.py", line 1056 in convert
  File "D:\Build\Pillow\Pillow-6.1.0\Tests\test_image_convert.py", line 9 in convert
  File "D:\Build\Pillow\Pillow-6.1.0\Tests\test_image_convert.py", line 31 in test_sanity
  File "X:\Python37\lib\unittest\case.py", line 615 in run
  File "D:\Build\Pillow\Pillow-6.1.0\Tests\helper.py", line 61 in run
  File "X:\Python37\lib\unittest\case.py", line 663 in __call__
  File "X:\Python37\lib\site-packages\_pytest\unittest.py", line 224 in runtest
  File "X:\Python37\lib\site-packages\_pytest\runner.py", line 123 in pytest_runtest_call
  File "X:\Python37\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "X:\Python37\lib\site-packages\pluggy\manager.py", line 81 in <lambda>
  File "X:\Python37\lib\site-packages\pluggy\manager.py", line 87 in _hookexec
  File "X:\Python37\lib\site-packages\pluggy\hooks.py", line 289 in __call__
  File "X:\Python37\lib\site-packages\_pytest\runner.py", line 198 in <lambda>
  File "X:\Python37\lib\site-packages\_pytest\runner.py", line 226 in from_call
  File "X:\Python37\lib\site-packages\_pytest\runner.py", line 198 in call_runtest_hook
  File "X:\Python37\lib\site-packages\_pytest\runner.py", line 173 in call_and_report
  File "X:\Python37\lib\site-packages\_pytest\runner.py", line 93 in runtestprotocol
  File "X:\Python37\lib\site-packages\_pytest\runner.py", line 78 in pytest_runtest_protocol
  File "X:\Python37\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "X:\Python37\lib\site-packages\pluggy\manager.py", line 81 in <lambda>
  File "X:\Python37\lib\site-packages\pluggy\manager.py", line 87 in _hookexec
  File "X:\Python37\lib\site-packages\pluggy\hooks.py", line 289 in __call__
  File "X:\Python37\lib\site-packages\_pytest\main.py", line 271 in pytest_runtestloop
  File "X:\Python37\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "X:\Python37\lib\site-packages\pluggy\manager.py", line 81 in <lambda>
  File "X:\Python37\lib\site-packages\pluggy\manager.py", line 87 in _hookexec
  File "X:\Python37\lib\site-packages\pluggy\hooks.py", line 289 in __call__
  File "X:\Python37\lib\site-packages\_pytest\main.py", line 250 in _main
  File "X:\Python37\lib\site-packages\_pytest\main.py", line 206 in wrap_session
  File "X:\Python37\lib\site-packages\_pytest\main.py", line 243 in pytest_cmdline_main
  File "X:\Python37\lib\site-packages\pluggy\callers.py", line 187 in _multicall
  File "X:\Python37\lib\site-packages\pluggy\manager.py", line 81 in <lambda>
  File "X:\Python37\lib\site-packages\pluggy\manager.py", line 87 in _hookexec
  File "X:\Python37\lib\site-packages\pluggy\hooks.py", line 289 in __call__
  File "X:\Python37\lib\site-packages\_pytest\config\__init__.py", line 82 in main
  File "X:\Python37\lib\site-packages\pytest.py", line 102 in <module>
  File "X:\Python37\lib\runpy.py", line 85 in _run_code
  File "X:\Python37\lib\runpy.py", line 193 in _run_module_as_main

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

  1. pypy3 builds are failing

_imagingft.obj : error LNK2019: unresolved external symbol _PyUnicode_AsUCS4Copy referenced in function _text_layout_raqm

Likely from #3780 (ping @nulano):

PyUnicode_AS_UNICODE is now deprecated, replace it with PyUnicode_AsUCS4Copy or PyUnicode_READ_CHAR.

What's the full version of the failing pypy3?

@cgohlke
Copy link
Contributor

cgohlke commented Jul 1, 2019

Segfault at

INT32 v = L(&palette[in[x]*4]) / 1000;

>	_imaging.cp37-win_amd64.pyd!p2i(unsigned char * out_=0x000001ef25cc5d58, const unsigned char * in=0x000001ef2202ffac, int xsize=128, const unsigned char * palette=0x000001ef142fabe7) Line 1045	C
 	_imaging.cp37-win_amd64.pyd!frompalette(ImagingMemoryInstance * imOut=0x000001ef13dcdfa0, ImagingMemoryInstance * imIn=0x000001ef142e6fa0, const char * mode=0x000001ef19dd9f40) Line 1204	C
 	_imaging.cp37-win_amd64.pyd!convert(ImagingMemoryInstance * imOut=0x0000000000000000, ImagingMemoryInstance * imIn=0x000001ef142e6fa0, const char * mode=0x000001ef19dd9f40, ImagingPaletteInstance * palette=0x0000000000000000, int dither=3) Line 1492	C
 	_imaging.cp37-win_amd64.pyd!ImagingConvert(ImagingMemoryInstance * imIn=0x000001ef142e6fa0, const char * mode=0x000001ef19dd9f40, ImagingPaletteInstance * palette=0x0000000000000000, int dither=3) Line 1542	C
 	_imaging.cp37-win_amd64.pyd!_convert(ImagingObject * self=0x000001ef2ddaafb0, _object * args=0x000001ef2dea26c8) Line 933	C
 	python37.dll!_PyMethodDef_RawFastCallKeywords(PyMethodDef * method=0x0000000000000000, _object * self=0x000001ef2ddaafb0, _object * const * args=0x000001ef2a1f8fb0, __int64 nargs=2, _object * kwnames) Line 694	C
 	[Inline Frame] python37.dll!_PyMethodDescr_FastCallKeywords(_object *) Line 288	C
 	python37.dll!call_function(_object * * * pp_stack=0x00000028847e91f8, __int64 oparg, _object * kwnames=0x0000000000000000) Line 4593	C

  | Name | Value | Type
-- | -- | -- | --
◢ | in | 0x000001ef2202ffac "\v.\v.\v\x10" | const unsigned char *
  |   | 11 '\v' | const unsigned char
◢ | out_ | 0x000001ef25cc5d58 "ÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀÀ... | unsigned char *
  |   | 192 'À' | unsigned char
◢ | palette | 0x000001ef142fabe7 "" | const unsigned char *
  |   | 0 '\0' | const unsigned char
  | v | 158 | int
  | x | 86 | int
  | xsize | 128 | int

@cgohlke
Copy link
Contributor

cgohlke commented Jul 1, 2019

What's the full version of the failing pypy3?

Python 3.6.1 (784b254d6699, Apr 16 2019, 12:10:48)
[PyPy 7.1.1-beta0 with MSC v.1910 32 bit] on win32

and

Python 3.5.3 (928a4f70d3de, Feb 08 2019, 12:56:35)
[PyPy 7.0.0 with MSC v.1910 32 bit] on win32

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

The segfault is likely from #3225, @DerDakon please could you check it?

@cgohlke
Copy link
Contributor

cgohlke commented Jul 1, 2019

How could the tests ever pass? Maybe I'm missing something, but shouldn't this be:

diff --git a/src/libImaging/Convert.c b/src/libImaging/Convert.c
index 038a83fe..5df48fb2 100644
--- a/src/libImaging/Convert.c
+++ b/src/libImaging/Convert.c
@@ -1041,7 +1041,7 @@ static void
 p2i(UINT8* out_, const UINT8* in, int xsize, const UINT8* palette)
 {
     int x;
-    for (x = 0; x < xsize; x++, in += 2, out_ += 4) {
+    for (x = 0; x < xsize; x++, out_ += 4) {
         INT32 v = L(&palette[in[x]*4]) / 1000;
         memcpy(out_, &v, sizeof(v));
     }
@@ -1060,7 +1060,7 @@ static void
 p2f(UINT8* out_, const UINT8* in, int xsize, const UINT8* palette)
 {
     int x;
-    for (x = 0; x < xsize; x++, in += 2, out_ += 4) {
+    for (x = 0; x < xsize; x++, out_ += 4) {
         FLOAT32 v = L(&palette[in[x]*4]) / 1000.0F;
         memcpy(out_, &v, sizeof(v));
     }

@DerDakon
Copy link
Contributor

DerDakon commented Jul 1, 2019

The very same change I came up with, see #3932.

@cgohlke
Copy link
Contributor

cgohlke commented Jul 1, 2019

The very same change I came up with, see #3932.

Good. All tests are passing here. No segfaults.

@hugovk
Copy link
Member

hugovk commented Jul 1, 2019

I've untagged 6.1.0.

PR #3932 is merged to fix the segfault.


That leaves the PyPy3 failure. I guess that's because PyUnicode_AS_UNICODE may well be "Deprecated since version 3.3, will be removed in version 4.0" in CPython, that doesn't necessarily mean it's deprecated in PyPy.

Our tests didn't pick it up because we don't test PyPy3 on Windows; and whilst we do test PyPy3 on Travis CI, there is a "implicit declaration of function ‘PyUnicode_AsUCS4Copy’" warning we didn't notice as all the test_imagefont tests are skipped because there's no FreeType (or RAQM) on PyPy3. We should improve these testing gaps.

I've opened a ticket with PyPy asking if they also plan to deprecate/replace/remove PyUnicode_AS_UNICODE.

And here's a PR to revert that change, in case we don't get a better solution in the short term: #3933.

I'll need to continue tomorrow unless someone in another timezone would like to pick up the reins.

Thanks everyone for the help so far!

@aclark4life
Copy link
Member

@hugovk Probably fine to wait until tomorrow, thanks for all the hard work! Much appreciated.

@nulano
Copy link
Contributor

nulano commented Jul 1, 2019

From the PyPy FAQ:

The external C-API has been reimplemented in PyPy as an internal cpyext module. We support most of the documented C-API [...]

I guess that means no modern PyUnicode functions. There might be a way to fix the original issue (#3777) using PyPy-specific functions, but probably not in time for this release.

It should be possible to revert the change for PyPy only by adding a preprocessor if block; looking through the PyPy include files, the macro PYPY_VERSION_NUM seems like a good candidate. I can try to write that tomorrow.

@aclark4life
Copy link
Member

Thanks @nulano

@nulano
Copy link
Contributor

nulano commented Jul 2, 2019

#3935 should fix the PyPy3 build failure introduced in 8d4bb33:

This disables the fix for #3777 for PyPy3, as PyPy's cpyext module hasn't been updated to support the new PyUnicode functions yet.

Our tests didn't pick it up because ... whilst we do test PyPy3 on Travis CI, there is a "implicit declaration of function ‘PyUnicode_AsUCS4Copy’" warning we didn't notice as all the test_imagefont tests are skipped because there's no FreeType (or RAQM) on PyPy3. We should improve these testing gaps.

The FreeType library was probably disabled due to a dynamic link error, not because it's not installed. FreeType works again with this PR: https://travis-ci.org/nulano/Pillow/jobs/553153042#L2976

@nulano
Copy link
Contributor

nulano commented Jul 2, 2019

Some new failures on pillow-wheels, TestImageFont.test_unicode_extended failed:

  • Mac MB_PYTHON_VERSION=2.7
  • Linux MB_PYTHON_VERSION=2.7 UNICODE_WIDTH=16
  • Linux MB_PYTHON_VERSION=2.7 PLAT=i686 UNICODE_WIDTH=16

With:

  • AssertionError: average pixel value difference 14.7190 > epsilon 6.2000
  • AssertionError: average pixel value difference 14.7190 > epsilon 6.2000
  • AssertionError: average pixel value difference 14.7280 > epsilon 6.2000

https://travis-ci.org/python-pillow/pillow-wheels/builds/552752918

For 1) see #3931 to loosen the epsilon for FreeType 2.10.

I suspect this is actually a real failure. When I tried the test on Windows without the #3780 fix, I got the same values:
E AssertionError: 2.5 not greater than or equal to 14.728 : average pixel value difference 14.7280 > epsilon 2.5000
https://ci.appveyor.com/project/nulano/pillow/builds/23676028/job/jjd897m8b68sx96a#L6647.

I suggest reverting #3931 and changing the test's skip conditions to skip on all platforms when running Python 2.

@nulano
Copy link
Contributor

nulano commented Jul 2, 2019

I suggest reverting #3931 and changing the test's skip conditions to skip on all platforms when running Python 2.

Done in #3936.

@hugovk
Copy link
Member

hugovk commented Jul 2, 2019

Thanks @nulano! Let's merge #3936 after the release to minimise changes, as it only changes a single test file.

@cgohlke Please could you test #3935? If it's fine, I'll merge it and continue the release.

@nulano
Copy link
Contributor

nulano commented Jul 2, 2019

Regarding #3935, I just realized PyPy does support the new PyUnicode_READ_CHAR function (used with basic layout), but not PyUnicode_AsUCS4Copy (used with raqm layout). In the PR I removed them both for PyPy.

I could add this back, but I think it's best to wait for the coming Python 2 end of support. At that point I would like to simplify and improve both layout functions. What do you think @hugovk?

@hugovk
Copy link
Member

hugovk commented Jul 2, 2019

Yeah, let's wait. It's only 6 months until Python 2 EOL (and 3 months until PRs can be merged). And the changes are in part to replace deprecated functions in CPython, which will be removed in CPython 4.0, due maybe 2021 or 2022 or who knows; no rush.

@cgohlke
Copy link
Contributor

cgohlke commented Jul 2, 2019

could you test #3935?

#3935 builds OK with pypy3 on Windows.

@hugovk
Copy link
Member

hugovk commented Jul 2, 2019

#3935 merged and 6.1.0 retagged.

pillow-wheels is queued up, although Travis is running slow at the moment so we'll need to be patient.

Take 3! @cgohlke Please could we have Windows binaries for 6.1.0?

@cgohlke
Copy link
Contributor

cgohlke commented Jul 2, 2019

could we have Windows binaries for 6.1.0?

Here you go.

@hugovk
Copy link
Member

hugovk commented Jul 3, 2019

🚀Pillow 6.1.0 is released! Thanks everyone for the help.

Edit: https://twitter.com/PythonPillow/status/1146339393046814721

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Pillow
  
Closed
Development

No branches or pull requests

6 participants