Kodi v19 crashes when PIL imports numpy

[cg00001]

This happens on Arch Linux with all addons and skins that use PIL image libraries.
Information on TheMovieDb Helper Kodi addon where its dev did a workaround here:
jurialmunkey/plugin.video.themoviedb.helper#445

A Kodi dev mentioned that this is a PIL / Numpy issue not a Kodi issue so I should ask here:
xbmc/xbmc#19324

Thank you for any possible solutions.

[basilgello]

I played with an issue a bit and found out that the underlying issue is the lack of sub-interpreter support in C++ numpy modules
(namely multiarray).

I am opening a NumPy bug now.

[radarhere]

Thanks for investigating further.

[basilgello]

@radarhere isn't it better to provide an option for Pillow users to disable internal numpy invocation somehow?

For example, by a method or constructor option that is on by default.

See attached issue: NumPy developers claim sub-interpreter support is still in near-beginning state.

[radarhere]

If I'm understanding correctly, Pillow's only contribution to this bug is the fact that it uses numpy?

How do you feel about this idea - radarhere@420f4e9 - moving the numpy import from the start of ImageFilter into the Color3DLUT class, so that it is only imported when necessary?

[basilgello]

I like the idea. Basically @jurialmunkey did very similar fix lazy-loading Pillow.

[radarhere]

I've created #5323.

[cg00001]

I just tried Flatpak's Kodi Matrix version and there's no Pillow issue there -probably because the packaged Python libraries there are earlier versions, or it's based on another Linux distro altogether.
I mention this as an alternative until proper Arch version is fixed:
https://flathub.org/apps/details/tv.kodi.Kodi

[cg00001]

The new Pillow version was out yesterday while I was asleep.
Just tested Kodi 19 and it works fine.
Thank you very much.

[radarhere]

Hi. I don't know what you're talking about. The last Pillow release was 8.1.2, three weeks ago, and the fix that we've put together for this issue hasn't been accepted into master yet, let alone a release.

[cg00001]

Yes I was mistaken about the Pillow release.
The weird thing is I've been using the Flatpak Kodi Matrix version until yesterday, and after installing the system Kodi, it worked without crashing.

[cg00001]

@basilgello I opened an issue on numpy about this at numpy/numpy#18848 (comment) which was closed with this quote:
"[The only thing I and others can do is wait.]

Not necessarily. Try, for example, temporarily inserting a raise RuntimeError() at the top of numpy/__init__.py and starting up Kodi. I'm not familiar with Kodi, so you'll have to find out how Kodi logs Python tracebacks from it and its plugins. That traceback should show you which part of Kodi or its plugins is importing numpy (possibly through an intermediate like Pillow). You will have to modify that code to not attempt importing numpy (or the intermediate library).

Since numpy has never worked with sub-interpreters, the (re)introduced bug is in the code that is trying to import numpy, so I'm going to close this."

I'm not even close to try that, maybe you can help?

[basilgello]

Well, on Arch Linux I see the following location of packaged numpy targeting Python 3.9: /usr/lib/python3.9/site-packages/numpy/__init__.py If you confirm you have that file installed on a troubled machine, we can edit it and check kodi.log for traceback.

[cg00001]

Well, on Arch Linux I see the following location of packaged numpy targeting Python 3.9: /usr/lib/python3.9/site-packages/numpy/init.py If you confirm you have that file installed on a troubled machine, we can edit it and check kodi.log for traceback.

I have it:

[cg00001]

Well, on Arch Linux I see the following location of packaged numpy targeting Python 3.9: /usr/lib/python3.9/site-packages/numpy/init.py If you confirm you have that file installed on a troubled machine, we can edit it and check kodi.log for traceback.

But I have no clue what to insert and on what line.

[cg00001]

I'm sorry to bother you again, but numpy peeps said it's definitely a pillow issue, Kodi devs marked the issue as externally solved, and the fact remains:
Latest python pillow made Kodi 19 work fine for just one day for me.
The following day numpy was updated and Kodi stopped working.
The issue remains, Kodi doesn't work in Arch Linux, unless this issue is resolved.

[basilgello]

@CgCgCgCgCg Please apply this patch to the file:

diff --git a/usr/lib/python3.9/site-packages/numpy/__init__.py b/usr/lib/python3.9/site-packages/numpy/__init__.py
index 6021e6d..1739d5f 100644
--- a/usr/lib/python3.9/site-packages/numpy/__init__.py
+++ b/usr/lib/python3.9/site-packages/numpy/__init__.py
@@ -106,6 +106,9 @@ available as array methods, i.e. ``x = np.array([1,2,3]); x.sort()``.
 Exceptions to this rule are documented.
 
 """
+
+raise RuntimeError()
+
 import sys
 import warnings
 

[radarhere]

I'm sorry to bother you again, but numpy peeps said it's definitely a pillow issue

Could you point me to an exact comment where that was said?

[cg00001]

That's what I understood from this comment:
numpy/numpy#18848 (comment)

[cg00001]

@CgCgCgCgCg Please apply this patch to the file:

diff --git a/usr/lib/python3.9/site-packages/numpy/__init__.py b/usr/lib/python3.9/site-packages/numpy/__init__.py
index 6021e6d..1739d5f 100644
--- a/usr/lib/python3.9/site-packages/numpy/__init__.py
+++ b/usr/lib/python3.9/site-packages/numpy/__init__.py
@@ -106,6 +106,9 @@ available as array methods, i.e. ``x = np.array([1,2,3]); x.sort()``.
 Exceptions to this rule are documented.
 
 """
+
+raise RuntimeError()
+
 import sys
 import warnings
 

Error starts in line 1794:
`2021-05-16 13:19:42.868 T:10479 ERROR : EXCEPTION Thrown (PythonToCppException) : -->Python callback/script returned the following error<--

                                                - NOTE: IGNORING THIS CAN LEAD TO MEMORY LEAKS!  
                                                - 
                                               Error Type: <class 'RuntimeError'>   

                                               Traceback (most recent call last):  

                                                 File "/home/k/.kodi/addons/plugin.video.cumination/default.py", line 33, in <module>  

                                                   from resources.lib.sites import *  # noqa  

                                                 File "/home/k/.kodi/addons/plugin.video.cumination/resources/lib/sites/streamate.py", line 21, in <module>
                                                   import websocket  

                                                 File "/home/k/.kodi/addons/script.module.websocket/lib/websocket/__init__.py", line 21, in <module>
                                                   from ._abnf import *  

                                                 File "/home/k/.kodi/addons/script.module.websocket/lib/websocket/_abnf.py", line 37, in <module>
                                                   import numpy  

                                                 File "/usr/lib/python3.9/site-packages/numpy/__init__.py", line 110, in <module>
                                                   raise RuntimeError()  

                                               RuntimeError  

                                               -->End of Python script error report<--` 

kodi.log

[basilgello]

Now post the link to this excerpt to where numpy authors asked you to provide this info & restore original __inir__.py file.

[radarhere]

That's what I understood from this comment:
numpy/numpy#18848 (comment)

I don't think they are pointing the problem back to Pillow. I appreciate that you feel like this issue is just being passed around (although to be fair, both jurialmunkey/plugin.video.themoviedb.helper@d029022 and our own #5323 have tried to work around the situation).

You believe that themoviedb.helper is importing Pillow, which then imports numpy, and then somehow, numpy causes a crash in https://github.com/xbmc/xbmc, right? Except since #5323, Pillow only imports numpy from ImageFilter.Color3DLUT, and themoviedb.helper doesn't call that. So if you are using the latest version of Pillow, unless you can find code that mentions ImageFilter.Color3DLUT, Pillow isn't involved in bringing numpy into the picture.

I can't reproduce this problem from the command line in a virtual machine right, and it only happens on Linux?

[cg00001]

If I read the error correctly Kodi's script.module.websocket is the culprit that imports numpy.

[cg00001]

I decided to edit directly line 37 of /home/k/.kodi/addons/script.module.websocket/lib/websocket/_abnf.py and replace "import numpy" with:
numpy = None

After initial testing this seems to solves crashes on Arch Linux and Kodi 19.1 seems to work fine.