Variable Explorer & Numba - Unable to retrieve the values . . .

[Scoooooba]

Issue Report Checklist

• Searched the issues page for similar reports
• Read the relevant sections of the Spyder Troubleshooting Guide and followed its advice
• Reproduced the issue after updating with conda update spyder (or pip, if not using Anaconda)
• Could not reproduce inside jupyter qtconsole (if console-related)
• Tried basic troubleshooting (if a bug/error)
  • Restarted Spyder
  • Reset preferences with spyder --reset
  • Reinstalled the latest version of Anaconda
  • Tried the other applicable steps from the Troubleshooting Guide
• Completed the Problem Description, Steps to Reproduce and Version sections below

Problem Description

Hi Guys,
using a numba-jit function inside a class method causes the "Spyder was unable to retrieve the value of this variable from the console. . . ." error when trying to view the class instance in the variable explorer.

What steps reproduce the problem?

Given the following code:

from numba import jit

class Testclass_1():
    def __init__(self):
        pass
    
    def using_nb_test(self, a):
        a = nb_test(a) # COMMENT THIS OUT TO SHOW NO ERROR
        print(a)


@jit(nopython=True)#, nogil=True, cache=True)
def nb_test(a):
    return a+1

if __name__ == '__main__':    
    testclass_1 = Testclass_1()
    testclass_1.using_nb_test(1)

After executing this code (Run File), double clicking on testclass_1 in the Variable Explorer will show the error.

Additional info: The filepath is in the Pythonpath. Error confirmed on a second computer.

What is the expected output? What do you see instead?

I expect to see the class object information, like I see it when commenting out a = nb_test(a) # COMMENT THIS OUT TO SHOW NO ERROR. Instead I see the error.

This error first occurred with an other, more complex class that I use in a project. But without making any changes, sometimes the error is shown and sometimes everything works fine. It's like chasing ghosts, I could not figure out what really is the problem. But it seems like it has something to do with the numba function, that is used in one of the class methods, just like in the reduced example shown above. But that's also the reason why I report this, even though the error message told me not to do it. It seems odd, that sometimes it works, and sometimes not.

Paste Traceback/Error Below (if applicable)

None

Versions

• Spyder version: 5.3.2
• Python version: 3.8.10 (using an external interpreter 3.9.12 in a virtualenv)
• Qt version: 5.15.2
• PyQt version: 5.15.7
• Operating System name/version: Windows 11 Home

Dependencies

# Mandatory:
atomicwrites >=1.2.0                 :  1.4.1 (OK)
chardet >=2.0.0                      :  5.0.0 (OK)
cloudpickle >=0.5.0                  :  2.1.0 (OK)
cookiecutter >=1.6.0                 :  2.1.1 (OK)
diff_match_patch >=20181111          :  20200713 (OK)
intervaltree                         :  None (OK)
IPython >=7.31.1;<8.0.0              :  7.34.0 (OK)
jedi >=0.17.2;<0.19.0                :  0.18.1 (OK)
jellyfish >=0.7                      :  0.9.0 (OK)
jsonschema >=3.2.0                   :  4.7.2 (OK)
keyring >=17.0.0                     :  23.6.0 (OK)
nbconvert >=4.0                      :  6.5.0 (OK)
numpydoc >=0.6.0                     :  1.4.0 (OK)
paramiko >=2.4.0                     :  2.11.0 (OK)
parso >=0.7.0;<0.9.0                 :  0.8.3 (OK)
pexpect >=4.4.0                      :  4.8.0 (OK)
pickleshare >=0.4                    :  0.7.5 (OK)
psutil >=5.3                         :  5.9.1 (OK)
pygments >=2.0                       :  2.12.0 (OK)
pylint >=2.5.0;<3.0                  :  2.14.4 (OK)
pyls_spyder >=0.4.0                  :  0.4.0 (OK)
pylsp >=1.5.0;<1.6.0                 :  1.5.0 (OK)
pylsp_black >=1.2.0                  :  1.2.1 (OK)
qdarkstyle >=3.0.2;<3.1.0            :  3.0.3 (OK)
qstylizer >=0.1.10                   :  0.2.1 (OK)
qtawesome >=1.0.2                    :  1.1.1 (OK)
qtconsole >=5.3.0;<5.4.0             :  5.3.1 (OK)
qtpy >=2.1.0                         :  2.1.0 (OK)
rtree >=0.9.7                        :  1.0.0 (OK)
setuptools >=49.6.0                  :  63.1.0 (OK)
sphinx >=0.6.6                       :  5.0.2 (OK)
spyder_kernels >=2.3.2;<2.4.0        :  2.3.2 (OK)
textdistance >=4.2.0                 :  4.3.0 (OK)
three_merge >=0.1.1                  :  0.1.1 (OK)
watchdog                             :  2.1.9 (OK)
zmq >=22.1.0                         :  23.2.0 (OK)

# Optional:
cython >=0.21                        :  0.29.30 (OK)
matplotlib >=3.0.0                   :  3.5.2 (OK)
numpy >=1.7                          :  1.22.4 (OK)
pandas >=1.1.1                       :  1.4.3 (OK)
scipy >=0.17.0                       :  1.8.1 (OK)
sympy >=0.7.3                        :  1.10.1 (OK)

# Spyder plugins:
spyder_terminal.terminalplugin 1.2.2 :  1.2.2 (OK)

PIP LIST

backcall          0.2.0
cloudpickle       2.1.0
colorama          0.4.5
debugpy           1.6.2
decorator         5.1.1
entrypoints       0.4
ipykernel         6.15.1
ipython           7.34.0
jedi              0.18.1
jupyter-client    7.3.4
jupyter-core      4.11.1
llvmlite          0.38.1
matplotlib-inline 0.1.3
nest-asyncio      1.5.5
numba             0.55.2
numpy             1.22.4
packaging         21.3
parso             0.8.3
pickleshare       0.7.5
pip               22.0.4
prompt-toolkit    3.0.30
psutil            5.9.1
Pygments          2.12.0
pyparsing         3.0.9
python-dateutil   2.8.2
pywin32           304
pyzmq             23.2.0
setuptools        62.1.0
six               1.16.0
spyder-kernels    2.3.2
tornado           6.2
traitlets         5.3.0
wcwidth           0.2.5
wheel             0.37.1

[mhofsaess]

I have the same problem with self designed classes. The definitions of the the classes are in the PYTHONPATH.

With Spyder 5.3.1 it works, with 5.3.2 I get the error.
It looks like, that a process is doing something in the background in a endless loop or waiting for feeback.

[dalthviz]

Hi @Scoooooba thank you for the feedback! Since you are using the standalone installer most probably this is a duplicate of #18807 . The suggestion to workaround this is at #18807 (comment) or since you are using virtualenv basically creating a virtualenv where Spyder is installed along side Numba and other packages you want to use.

Let us know if something of the above helps!

[ccordoba12]

For @mhofsaess who also posted here: we fixed the way Spyder handles PYTHONPATH in 5.3.2 because it was inconsistent and not entirely right.

So to address your problem I think now you need to import your external PYTHONPATH to our manager. You can do that by clicking in the new Import button on it:

[mhofsaess]

Hi @ccordoba12 , as I wrote in the previous post, I added the path to the PYTHONPATH. I have been doing this since I started using Spyder (version 2.x or so) via the PYTHONPATH manager. I use the pip version under linux and as said with SPYDER 5.3.1 the display of the OBJECT in the variable manager works, after update to 5.3.2 unfortunately no longer. I did not change anything in the configuration, the PATH is also correct in the PYTHONPATH.

I started spyder in debug mode and import my user defined class.

from IO.datatypes.vtypes import data
a = data()

In the variable explorer the variable a appears with the correct type, in the commandline window I get follow message after double click a:

[DEBUG] [spyder_kernels.comms.commbase] -> Exception in cloudpickle.loads : No module named 'IO'

I hope this helps.

[Scoooooba]

Hi @Scoooooba thank you for the feedback! Since you are using the standalone installer most probably this is a duplicate of #18807 . The suggestion to workaround this is at #18807 (comment) or since you are using virtualenv basically creating a virtualenv where Spyder is installed along side Numba and other packages you want to use.

Let us know if something of the above helps!

Hi dalthviz, thanks for your answer! I tried your suggestions and can confirm that it works fine in two different ways:

• Installing spyder in the virtualenv using pip, and then run spyder out of the virtualenv
• Installing numba in the python runtime, that comes with the spyder standalone installation

Both ways there is no longer an error message using the example of my initial post.

Unfortunately I still get an error using the more complex classes I mentioned in my initial post, but I have not yet been able to reduce it to a simple example. It is like I have a module "SomeModule" with a class "SomeClass", and having the path of "SomeModule" in the PYTHONPATH. If I write in the console:

from SomeModule import SomeClass
a = SomeClass()

and then double click on a in the Variable Explorer (displayed as Type "SomeModule.SomeClass"), the error occurs. But when I then run (F5) the SomeModule and afterwards double-click on a (without executing a = SomeClass() again), the error does not occur.
As I said, unfortunately I have not yet been able to create an example, as this problem only seems to occur with my more complex class. I will provide an example, as soon as I could figure it out.

Until then I thank your for your previous answer, solving at least the first problem.

---

Edit:
Using the Variable Explorer and the more complex class works fine when installing spyder in the virtualenv using pip!

This raises an other question:
Currently I use a little script that extends the "Spyder.launch.pyw" file of the standalone version to enable me choosing an interpreter at spyders startup. This way I don't have to go in the preferences and change the "Python Interpreter" every time i want to switch to another virtualenv. The script simply changes some values of the "spyder.ini" and "transient.ini" files in the spyder config path.
As far as I see, the spyder version installed in the virtualenv using pip is using the same spyder config path, and so also using the same "spyder.ini" and "transient.ini". So two questions:

• Can I somehow change the path, where spyder saves its config, so that the pip version and the standalone version are no longer using the same config path?
• For the pip version: What is the first file that is executed, when running the "spyder" start-command in the console?

[ccordoba12]

@mhofsaess, you said:

in the commandline window I get follow message after double click a:

[DEBUG] [spyder_kernels.comms.commbase] -> Exception in cloudpickle.loads : No module named 'IO'

This means that the kernel is not loading PYTHONPATH correctly. @mrclary, this looks like a regression of a long-standing behavior. Could you take a look at it?

[ccordoba12]

@Scoooooba, you said:

Can I somehow change the path, where spyder saves its config

Yes, please take a look at the --conf-dir command line option for that.

What is the first file that is executed, when running the "spyder" start-command in the console?

Pip creates an executable called spyder.exe after installation and (if I'm not mistaken) that file can't be modified. If you want to alter the way Spyder is started, you need to remove that file and create your own script that runs spyder.app.start.main.

[mrclary]

Hi @ccordoba12 , as I wrote in the previous post, I added the path to the PYTHONPATH. I have been doing this since I started using Spyder (version 2.x or so) via the PYTHONPATH manager. I use the pip version under linux and as said with SPYDER 5.3.1 the display of the OBJECT in the variable manager works, after update to 5.3.2 unfortunately no longer. I did not change anything in the configuration, the PATH is also correct in the PYTHONPATH.

I started spyder in debug mode and import my user defined class.

from IO.datatypes.vtypes import data
a = data()

In the variable explorer the variable a appears with the correct type, in the commandline window I get follow message after double click a:

[DEBUG] [spyder_kernels.comms.commbase] -> Exception in cloudpickle.loads : No module named 'IO'

I hope this helps.

The new behavior for PYTHONPATH Manager is that these paths only apply to the IPython Console environment, not to Spyder's runtime environment. This provides a type of firewall between Spyder's runtime environment and the user's own development environment and is important to prevent user's from accidentally breaking Spyder. For the same reason, the user's system PYTHONPATH environment variable is not added to Spyder's own sys.path (which is used by Variable Explorer) when launching Spyder.

This issue also manifests in the difference between our "Lite" and "Full" versions of the Spyder installer for macOS and Windows: objects for numpy, matplotlib, pandas, etc. cannot be double-clicked in the Variable Explorer if those packages are not installed in Spyder's runtime environment ("Full" version).

I've been exploring some ideas to resolve this issue as well as make Variable Explorer more flexible, but in the short term I don't think there is an easy fix, sorry.

[dalthviz]

Just in case, maybe this could be interesting for the limitation of serialization/deserialization in the Variable Explorer?: cloudpipe/cloudpickle#417 and https://github.com/cloudpipe/cloudpickle#overriding-pickles-serialization-mechanism-for-importable-constructs with cloudpickle >=2.0.0

[Scoooooba]

@Scoooooba, you said:

Can I somehow change the path, where spyder saves its config

Yes, please take a look at the --conf-dir command line option for that.

What is the first file that is executed, when running the "spyder" start-command in the console?

Pip creates an executable called spyder.exe after installation and (if I'm not mistaken) that file can't be modified. If you want to alter the way Spyder is started, you need to remove that file and create your own script that runs spyder.app.start.main.

Thank you!
Not having to install spyder multiple times (in each virtualenv) would be nice, but this workaround solves the problem for me :)

[mhofsaess]

Hi @mrclary ,
if I have now understood correctly, there have been changes (Issue 17511 ??) in how the PYTHONPATH variable is processed between Spyder and IPYTHON from spyderVersion 5.3.1 to 5.3.2, which is now responsible for the fact that certain variables can no longer be displayed.
If there is no short-term solution to the problem now, can you give an outlook until when there can be a solution?

[mrclary]

Hi @mrclary ,
if I have now understood correctly, there have been changes (Issue 17511 ??) in how the PYTHONPATH variable is processed between Spyder and IPYTHON from spyderVersion 5.3.1 to 5.3.2, which is now responsible for the fact that certain variables can no longer be displayed.
If there is no short-term solution to the problem now, can you give an outlook until when there can be a solution?

Variables will still be displayed, but they may not be able to be inspected (double-clicked in the Variable Explorer). I cannot make any promises as to when this will be resolved, but we are investigating possible solutions. We agree that this functionality of the Variable Explorer is important and appreciate your patience.

[Scoooooba]

Hey Guys,
I'm sorry, but I have do dive back into this topic again..
I'm currently editing the "more complex" classes mentioned above and unfortunately I see the error again. So I have a version that's working and one that is not. So there's a difference between the two versions that causes the error. While trying to identify the problem, I of course start with the working version and make small changes towards the not working version and test it every time. But somehow, a certain version sometimes works, and sometimes shows the error. I seems that at some point Spyder is not updating it's chached version of the class, so it does not show the error because internally its using an old representation of the class. Does that make sense? So now, after every small change, I close the project, close Spyder, delete the pychache folder, start Spyder again, open the project again and run the file to see if it still works. A bit paranoid...
So basically my question is: Can there be an old representation of the class chached in Spyder? And if so, can I force a reload?

Please note, that in the IPython Console everything works fine and changes to the class are always correctly applied. But as far as I understood, the IPython console and Spyders internally used representation of classes are two different things.

Edit:
After a few test I found the problematic change in the code and so I can confirm, that I need to restart Spyder (closing/opening project and deleting pychache is not necessary) in order to make Spyder show the variable explorer error after the problematic change. I can successfully reproduce it with the following steps:

1. Run the (working) code, check variable in Variable Explorer --> no Error
2. Make problematic change to the code, run it, check variable in the Variable Explorer --> no Error
3. Restart Spyder, run the code --> Error
4. Undo the problematic change, run the code --> Error
5. Restart Spyder, run the code --> no Error

So far regarding the refreshing of the representation of the class in Spyder. By the way, the problematic change mentioned is editing the __setitem__ method of a class that inherits from dict to raise an Exception in some cases.

[mrclary]

@Scoooooba, can you post a minimum working example that reproduces the behavior? If we can properly identify the underlying issue, then we may be able to create a fix; maybe a cache refresh or reloading the IPython namespace, or something.

[Scoooooba]

Sorry, yes of course!
Given two files in the same directory.
File Testclass.py:

class Testclass(dict):
    def __setitem__(self, k, v):
        check = False
        if check:
            if k in self:
                dict.__setitem__(self, k, v)
            else:
                raise Exception(f"Key {k} not yet in dict.")
        else:
            dict.__setitem__(self, k, v)

File Testscript.py:

from Testclass import Testclass

if __name__ == '__main__':
    test = Testclass({'key1': 1})

Working directory is the one, where both files are in. Steps to reproduce:

1. Run Testscript.py. Inspect variable test --> no error
2. In Testclass.py change check = False to check = True and save. With this change, a class instance can not be pickled.
3. Run Testscript.py. Inspect variable test --> no error (But there should be one)
4. Restart Spyder
5. Run Testscript.py. Inspect variable test --> Error!
6. In Testclass.py change check = True to check = False and save. With this change, the error should no longer occur.
7. Run Testscript.py. Inspect variable test --> Error! (still..)
8. Restart Spyder
9. Run Testscript.py. Inspect variable test --> no error (as expexted)

Also, when the Variable Explorer is not working as it should, the "loading circle" is spinning forever, like mhofsaess mentioned in his first post.

---

Versions:

• Spyder version: 5.3.2 (standalone)
• Python version: 3.8.10 64-bit
• Qt version: 5.15.2
• PyQt5 version: 5.15.7
• Operating System: Windows 10

Dependencies:

# Mandatory:
atomicwrites >=1.2.0                 :  1.4.1 (OK)
chardet >=2.0.0                      :  5.0.0 (OK)
cloudpickle >=0.5.0                  :  2.1.0 (OK)
cookiecutter >=1.6.0                 :  2.1.1 (OK)
diff_match_patch >=20181111          :  20200713 (OK)
intervaltree                         :  None (OK)
IPython >=7.31.1;<8.0.0              :  7.34.0 (OK)
jedi >=0.17.2;<0.19.0                :  0.18.1 (OK)
jellyfish >=0.7                      :  0.9.0 (OK)
jsonschema >=3.2.0                   :  4.7.2 (OK)
keyring >=17.0.0                     :  23.6.0 (OK)
nbconvert >=4.0                      :  6.5.0 (OK)
numpydoc >=0.6.0                     :  1.4.0 (OK)
paramiko >=2.4.0                     :  2.11.0 (OK)
parso >=0.7.0;<0.9.0                 :  0.8.3 (OK)
pexpect >=4.4.0                      :  4.8.0 (OK)
pickleshare >=0.4                    :  0.7.5 (OK)
psutil >=5.3                         :  5.9.1 (OK)
pygments >=2.0                       :  2.12.0 (OK)
pylint >=2.5.0;<3.0                  :  2.14.4 (OK)
pyls_spyder >=0.4.0                  :  0.4.0 (OK)
pylsp >=1.5.0;<1.6.0                 :  1.5.0 (OK)
pylsp_black >=1.2.0                  :  1.2.1 (OK)
qdarkstyle >=3.0.2;<3.1.0            :  3.0.3 (OK)
qstylizer >=0.1.10                   :  0.2.1 (OK)
qtawesome >=1.0.2                    :  1.1.1 (OK)
qtconsole >=5.3.0;<5.4.0             :  5.3.1 (OK)
qtpy >=2.1.0                         :  2.1.0 (OK)
rtree >=0.9.7                        :  1.0.0 (OK)
setuptools >=49.6.0                  :  63.1.0 (OK)
sphinx >=0.6.6                       :  5.0.2 (OK)
spyder_kernels >=2.3.2;<2.4.0        :  2.3.2 (OK)
textdistance >=4.2.0                 :  4.3.0 (OK)
three_merge >=0.1.1                  :  0.1.1 (OK)
watchdog                             :  2.1.9 (OK)
zmq >=22.1.0                         :  23.2.0 (OK)

# Optional:
cython >=0.21                        :  0.29.30 (OK)
matplotlib >=3.0.0                   :  3.5.2 (OK)
numpy >=1.7                          :  1.22.4 (OK)
pandas >=1.1.1                       :  1.4.3 (OK)
scipy >=0.17.0                       :  1.8.1 (OK)
sympy >=0.7.3                        :  1.10.1 (OK)

# Spyder plugins:
spyder_terminal.terminalplugin 1.2.2 :  1.2.2 (OK)

[mrclary]

@Scoooooba, I can reproduce your issue. We do not implement a caching system for the variable explorer. However, this is what is happening:

1. The variable meta-info is transmitted from IPython Console to Spyder's Variable Explorer, which is what is displayed.
2. When inspecting the variable, a request is sent to the IPython Console to send more info. This is pickled (via cloudpickle) and received by the Variable Explorer.
3. When the info is unpickled, Variable Explorer will display the new data, if it can interpret the object class. In this case, the class module is loaded into Spyder's runtime environment.
4. If the object cannot be pickled, or if Variable Explorer cannot interpret the object class (e.g. it cannot access a required package or module), then the error message appears.

We are currently looking into features of cloudpickle that may mitigate issues related to inaccessible packages or modules. However, in your MWE, this is not the issue. Here, the issue is that Testclass module is loaded into Spyder's runtime environment and not replaced when the class is changed. You can see this without restarting Spyder and without restarting the IPython Console.

1. Follow steps 1-3 above. Do not restart Spyder in step 4; instead...
2. Open Spyder's internal console: View->Panes->Internal console
3. Remove Testclass from sys.modules

>>> import sys
>>> sys.modules.pop('Testclass')

5. Inspect the variable in Variable Explorer. You should now get an error.

You can repeat the process after changing back check = False and restore the inspection (and the spinner stops as well).

This is not a practical work-around, by an means, but merely illustrates the mechanisms at play. This is Python's standard treatment of imported modules: if it already exists in sys.modules it does not do anything.

@ccordoba12 @dalthviz, as we endeavor to improve Variable Explorer, perhaps we should also consider leveraging or replicating the User Module Reloader for use with the Variable Explorer.

[ccordoba12]

as we endeavor to improve Variable Explorer, perhaps we should also consider leveraging or replicating the User Module Reloader for use with the Variable Explorer.

Ok, that sounds like a very interesting idea. For that I think we should simply import and run our UMR before users try to open a variable in the Variable Explorer.

However, given that that could impact Spyder's stability (I don't know how it could affect the running application under all possible circumstances), I think we should leave it for Spyder 6.

[Scoooooba]

@mrclary, @ccordoba12, thanks for the explaination and outview