Memory issue PIL >= 8.3.0

[helgeerbe]

Today my picframe service crashed

I'm logging in a influxdb the system load and found this behavior.

System load is quite low. But memory consumption is strange. Picframe starts with 6% total memory. While running, memory is freed (sawtooth line). But at the end memory consumption is nearly 100% in total.

I checked an older log:

Memory consumption is as expected stable. When suddenly the sawtooth line appears. (High cpu load two most left yellow peaks are the nightly backups).

What have I done?

• apt get full-upgrade
• pull all python modules required by picframe to their latest releases

pi@picframe:~ $ python3 --version
Python 3.7.3
pi@picframe:~ $ picframe -v
INFO:start.py:starting ['/home/pi/.local/bin/picframe', '-v']
picframe version:  0+untagged.365.gfc64728

Checking required packages......
PIL :  8.4.0
exifread :  2.3.2
pi3d :  2.48
yaml :  6.0
paho.mqtt :  1.5.1
iptcinfo3 :  2.1.4
numpy :  1.21.2
ninepatch : installed, but no version info

Checking optional packages......
pyheif :  0.5.1

I rolled back some older releases of picframe, but the sawtooth line was still there. So maybe one of the python modules has an issue?

Does this occur on your pic frames (check with top if memory consumption is stable over a period of time). Which versions of the modules are you using?

[helgeerbe]

OK, found it. It's Pillow / PIL. I installed "PIL : 7.2.0" and everything is back to normal. Have to find out, which is the latest release, where everything is OK.

[paddywwoof]

@helgeerbe well done. I've got 8.1.2 on this 'big' computer, but 5.4.1 on the RPi I should think the pillow devs would want to know about this...
Rust <3
C :(
EDIT that's a heart emoji not 'prior to version 3'

[helgeerbe]

I nail it down. 8.3.0 is broken. 8.2.0 seems to work. I will let it run for a while to be sure.
One other thing. I added some new pictures while using 8.4.0 and I was wondering why they don't have any location information. I found no exif info in the database. Deleting the database and reimport the images under 8.2.0 and even the exif data is imported properly. Not sure if this is some how related.

[helgeerbe]

@paddywwoof I hope I found the root cause. Can you have a look at python-pillow/Pillow#5797 (comment)

[helgeerbe]

Should be fixed in the next Pillow release python-pillow/Pillow#5844

[helgeerbe]

Should be fixed in Pillow release 9.0.0. I updated the dependency in setup.py.
As a workaround upgrade Pillow to release 9.0.0 or later