Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance problems with panel > 0.13 and Lab3 #3627

Closed
ItamarShDev opened this issue Jun 15, 2022 · 19 comments · Fixed by #3770
Closed

Performance problems with panel > 0.13 and Lab3 #3627

ItamarShDev opened this issue Jun 15, 2022 · 19 comments · Fixed by #3770
Labels
major issue type: bug Something isn't correct or isn't working
Milestone
Version 0.14.0

Comments

@ItamarShDev
Copy link

We are experiencing some performance behaviors with panel > 0.13
The problems we are facing are:

  • General lab slowness and input lags on Jupyter Lab
  • Random kernel crashes while using tools (can also happen in non Panel tools)
  • On local Windows machined we cannot use any panel related imports in startup.py files, using panel in said files will cause the kernel to crash and never boot.

We get the following error:

Traceback (most recent call last):
  File "/opt/conda/bin/jupyterhub-singleuser", line 10, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/app.py", line 66, in main
    return SingleUserLabApp.launch_instance()
  File "/opt/conda/lib/python3.9/site-packages/jupyter_core/application.py", line 264, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/opt/conda/lib/python3.9/site-packages/traitlets/config/application.py", line 846, in launch_instance
    app.start()
  File "/opt/conda/lib/python3.9/site-packages/jupyterhub/singleuser/mixins.py", line 645, in start
    super().start()
  File "/opt/conda/lib/python3.9/site-packages/jupyter_server/serverapp.py", line 2669, in start
    self.start_ioloop()
  File "/opt/conda/lib/python3.9/site-packages/jupyter_server/serverapp.py", line 2655, in start_ioloop
    self.io_loop.start()
  File "/opt/conda/lib/python3.9/site-packages/tornado/platform/asyncio.py", line 199, in start
    self.asyncio_loop.run_forever()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/opt/conda/lib/python3.9/asyncio/base_events.py", line 1875, in _run_once
    handle = self._ready.popleft()
IndexError: pop from an empty deque

Here is the boken system pip list:

Package                       Version                     Editable project location
----------------------------- --------------------------- --------------------------------------------------------------------
aenum                         2.2.4
aiohttp                       3.8.1
aiosignal                     1.2.0
alabaster                     0.7.12
altair                        4.1.0
analytics-python              1.4.0
anyio                         3.6.1
argon2-cffi                   21.3.0
argon2-cffi-bindings          21.2.0
asttokens                     2.0.5
async-timeout                 4.0.2
atomicwrites                  1.4.0
attrs                         21.4.0
awscli                        1.23.9
Babel                         2.10.1
backcall                      0.2.0
backoff                       1.10.0
beautifulsoup4                4.11.1
bleach                        5.0.0
bokeh                         2.4.3
boto3                         1.22.9
botocore                      1.25.9
cachetools                    5.2.0
certifi                       2022.5.18.1
cffi                          1.15.0
charset-normalizer            2.0.12
cloudpickle                   2.1.0
colorama                      0.4.4
coverage                      6.4.1
cycler                        0.11.0
debugpy                       1.6.0
decorator                     5.1.1
defusedxml                    0.7.1
docopt                        0.6.2
docutils                      0.15.2
entrypoints                   0.4
execnet                       1.9.0
executing                     0.8.3
fastjsonschema                2.15.3
filelock                      3.3.1
flake8                        3.8.3
fonttools                     4.33.3
frozenlist                    1.3.0
fsspec                        2022.5.0
gcsfs                         2022.5.0
google-api-core               2.8.1
google-auth                   2.6.6
google-auth-oauthlib          0.5.1
google-cloud-core             2.3.0
google-cloud-storage          2.3.0
google-crc32c                 1.3.0
google-resumable-media        2.3.3
googleapis-common-protos      1.56.2
hybridcontents                0.3.0
icdiff                        2.0.5
idna                          3.3
imagesize                     1.3.0
importlib-metadata            4.11.4
imubit-analytics              0.0.3
iniconfig                     1.1.1
iniherit                      0.3.9
ipyaggrid                     0.2.1
ipydatetime                   1.2.0
ipykernel                     6.13.1
ipython                       8.4.0
ipython-genutils              0.2.0
ipywidgets                    7.7.0
isort                         5.6.4
jedi                          0.18.1
jenkinsapi                    0.3.11
Jinja2                        3.1.2
jmespath                      1.0.0
joblib                        1.1.0
json5                         0.9.8
jsonschema                    4.6.0
jupyter                       1.0.0
jupyter-bokeh                 3.0.4
jupyter-client                7.3.3
jupyter-console               6.4.3
jupyter-core                  4.10.0
jupyter-server                1.17.1
jupyterlab                    3.4.2
jupyterlab-pygments           0.2.2
jupyterlab-server             2.14.0
jupyterlab-widgets            1.1.0
kiwisolver                    1.4.2
llvmlite                      0.38.1
Markdown                      3.3.7
MarkupSafe                    2.1.1
matplotlib                    3.5.1
matplotlib-inline             0.1.3
mccabe                        0.6.1
mistune                       0.8.4
mock                          3.0.5
mock-generator                1.0.0
monotonic                     1.6
multidict                     6.0.2
nbclassic                     0.3.7
nbclient                      0.6.4
nbconvert                     6.5.0
nbformat                      5.4.0
nbval                         0.9.6
nest-asyncio                  1.5.5
nose                          1.3.7
notebook                      6.4.12
notebook-shim                 0.1.0
numba                         0.55.2
numexpr                       2.8.1
numpy                         1.22.0
oauthlib                      3.2.0
packaging                     21.3
pandas                        1.2.5
pandocfilters                 1.5.0
panel                         0.13.1
param                         1.12.1
parso                         0.8.3
pickleshare                   0.7.5
Pillow                        9.1.1
pip                           22.1.2
pluggy                        0.13.1
pprintpp                      0.4.0
prometheus-client             0.14.1
prompt-toolkit                3.0.29
protobuf                      3.20.1
psutil                        5.9.1
psycopg2-binary               2.9.3
pure-eval                     0.2.2
py                            1.11.0
pyasn1                        0.4.8
pyasn1-modules                0.2.8
pycodestyle                   2.6.0
pycparser                     2.21
pyct                          0.4.8
pydantic                      1.9.0
pyflakes                      2.2.0
Pygments                      2.12.0
pymongo                       4.0.1
pyparsing                     3.0.9
PyPika                        0.48.8
pyrsistent                    0.18.1
pytest                        6.2.3
pytest-asyncio                0.14.0
pytest-cov                    2.11.1
pytest-forked                 1.4.0
pytest-icdiff                 0.5
pytest-mock                   3.6.1
pytest-repeat                 0.9.1
pytest-xdist                  2.4.0
python-dateutil               2.8.2
python-json-logger            0.1.11
python-slugify                6.1.2
pytimeparse                   1.1.8
pytz                          2022.1
pyviz-comms                   2.2.0
pywin32                       304
pywinpty                      2.0.5
PyYAML                        5.3.1
pyzmq                         23.1.0
qtconsole                     5.3.1
QtPy                          2.1.0
requests                      2.27.1
requests-oauthlib             1.3.1
rsa                           4.7.2
s3contents                    0.6.0
s3fs                          0.4.2
s3transfer                    0.5.2
scikit-learn                  1.1.1
scipy                         1.8.1
seaborn                       0.11.2
selenium                      3.141.0
Send2Trash                    1.8.0
setuptools                    62.3.3
shap                          0.40.0
simpleeval                    0.9.11
simplejson                    3.17.6
six                           1.16.0
sklearn                       0.0
slicer                        0.0.7
sniffio                       1.2.0
snowballstemmer               2.2.0
soupsieve                     2.3.2.post1
Sphinx                        3.2.1
sphinx-rtd-theme              0.5.0
sphinxcontrib-applehelp       1.0.2
sphinxcontrib-devhelp         1.0.2
sphinxcontrib-htmlhelp        2.0.0
sphinxcontrib-jsmath          1.0.1
sphinxcontrib-qthelp          1.0.3
sphinxcontrib-serializinghtml 1.1.5
stack-data                    0.2.0
tables                        3.7.0
tabulate                      0.8.9
terminado                     0.15.0
testpath                      0.6.0
text-unidecode                1.3
threadpoolctl                 3.1.0
timedb                        3.29.2
tinycss2                      1.1.1
toml                          0.10.2
toolz                         0.11.2
tornado                       6.1
traitlets                     5.2.2.post1
typing_extensions             4.2.0
urllib3                       1.26.9
wcwidth                       0.2.5
webencodings                  0.5.1
websocket-client              1.3.2
wheel                         0.34.2
widgetsnbextension            3.6.0
yapf                          0.30.0
yarl                          1.7.2
yoyo-migrations               5.0.5
zipp                          3.8.0

And here is a pip list of the working environment:

Package                           Version                     Editable project location
--------------------------------- --------------------------- ------------------------------------------------------------------
aenum                             2.2.4
aiohttp                           3.8.1
aiosignal                         1.2.0
alabaster                         0.7.12
altair                            4.1.0
analytics-python                  1.4.0
anyio                             3.6.1
argon2-cffi                       21.3.0
argon2-cffi-bindings              21.2.0
asttokens                         2.0.5
async-timeout                     4.0.2
atomicwrites                      1.4.0
attrs                             21.4.0
awscli                            1.23.9
Babel                             2.10.1
backcall                          0.2.0
backoff                           1.10.0
beautifulsoup4                    4.11.1
bleach                            5.0.0
bokeh                             2.4.3
boto3                             1.22.9
botocore                          1.25.9
cachetools                        5.2.0
certifi                           2022.5.18.1
cffi                              1.15.0
charset-normalizer                2.0.12
click                             8.0.3
cloudpickle                       2.1.0
colorama                          0.4.4
coverage                          6.4.1
cycler                            0.11.0
debugpy                           1.6.0
decorator                         5.1.1
defusedxml                        0.7.1
docopt                            0.6.2
docutils                          0.15.2
entrypoints                       0.4
execnet                           1.9.0
executing                         0.8.3
Faker                             13.3.4
fastjsonschema                    2.15.3
filelock                          3.3.1
flake8                            3.8.3
fonttools                         4.33.3
frozenlist                        1.3.0
fsspec                            2022.5.0
gcsfs                             2022.5.0
google-api-core                   2.8.1
google-auth                       2.7.0
google-auth-oauthlib              0.5.2
google-cloud-core                 2.3.1
google-cloud-storage              2.4.0
google-crc32c                     1.3.0
google-resumable-media            2.3.3
googleapis-common-protos          1.56.2
hybridcontents                    0.3.0
icdiff                            2.0.5
idna                              3.3
imagesize                         1.3.0
importlib-metadata                4.11.4
imubit-analytics                  0.0.3
iniconfig                         1.1.1
iniherit                          0.3.9
ipyaggrid                         0.2.1
ipydatetime                       1.2.0
ipykernel                         6.13.1
ipython                           8.4.0
ipython-genutils                  0.2.0
ipywidgets                        7.7.0
isort                             5.6.4
jedi                              0.18.1
jenkinsapi                        0.3.11
Jinja2                            3.1.2
jmespath                          1.0.0
joblib                            1.1.0
json5                             0.9.8
jsonschema                        4.6.0
jupyter-bokeh                     3.0.4
jupyter-client                    7.3.4
jupyter-contrib-core              0.3.3
jupyter-contrib-nbextensions      0.5.1
jupyter-core                      4.10.0
jupyter-highlight-selected-word   0.2.0
jupyter-latex-envs                1.4.6
jupyter-nbextensions-configurator 0.4.1
jupyter-server                    1.17.1
jupyterlab                        3.4.0
jupyterlab-pygments               0.2.2
jupyterlab-server                 2.14.0
jupyterlab-widgets                1.1.0
jupyterthemes                     0.20.0
kiwisolver                        1.4.3
kwargs                            1.0.1
lesscpy                           0.15.0
llvmlite                          0.38.1
lxml                              4.7.1
Markdown                          3.3.7
MarkupSafe                        2.1.1
matplotlib                        3.5.1
matplotlib-inline                 0.1.3
mccabe                            0.6.1
mistune                           0.8.4
mock                              3.0.5
mock-generator                    1.0.0
monotonic                         1.6
multidict                         6.0.2
mypy-extensions                   0.4.3
nbclassic                         0.3.7
nbclient                          0.6.4
nbconvert                         6.5.0
nbformat                          5.4.0
nbval                             0.9.6
nest-asyncio                      1.5.5
nose                              1.3.7
notebook                          6.4.12
notebook-shim                     0.1.0
numba                             0.55.2
numexpr                           2.8.1
numpy                             1.22.4
oauthlib                          3.2.0
packaging                         21.3
pandas                            1.2.5
pandocfilters                     1.5.0
panel                             0.13.1
param                             1.12.1
parameterized                     0.8.1
parso                             0.8.3
pathspec                          0.9.0
pbr                               5.8.1
pickleshare                       0.7.5
Pillow                            9.1.1
pip                               22.1.2
platformdirs                      2.4.1
pluggy                            0.13.1
ply                               3.11
pprintpp                          0.4.0
progress                          1.6
prometheus-client                 0.14.1
prompt-toolkit                    3.0.29
protobuf                          3.20.1
psutil                            5.9.1
psycopg2-binary                   2.9.3
pure-eval                         0.2.2
py                                1.11.0
pyasn1                            0.4.8
pyasn1-modules                    0.2.8
pycodestyle                       2.6.0
pycparser                         2.21
pyct                              0.4.8
pydantic                          1.9.0
pyflakes                          2.2.0
Pygments                          2.12.0
pymongo                           4.0.1
pyparsing                         3.0.9
PyPika                            0.48.8
pyrsistent                        0.18.1
PyRSS2Gen                         1.1
pytest                            6.2.3
pytest-asyncio                    0.14.0
pytest-cov                        2.11.1
pytest-datadir-ng                 1.1.1
pytest-forked                     1.4.0
pytest-icdiff                     0.5
pytest-mock                       3.6.1
pytest-pythonpath                 0.7.4
pytest-repeat                     0.9.1
pytest-xdist                      2.4.0
python-dateutil                   2.8.2
python-json-logger                0.1.11
python-slugify                    6.1.2
pytimeparse                       1.1.8
pytz                              2022.1
pyviz-comms                       2.2.0
pywin32                           304
pywinpty                          2.0.5
PyYAML                            6.0
pyzmq                             23.1.0
requests                          2.28.0
requests-oauthlib                 1.3.1
rsa                               4.7.2
s3contents                        0.6.0
s3fs                              0.4.2
s3transfer                        0.5.2
scikit-learn                      1.1.1
scipy                             1.8.1
seaborn                           0.11.2
selenium                          3.141.0
self                              2020.12.3
Send2Trash                        1.8.0
setuptools                        62.4.0
shap                              0.40.0
simpleeval                        0.9.11
simplejson                        3.17.6
six                               1.16.0
sklearn                           0.0
slicer                            0.0.7
sniffio                           1.2.0
snowballstemmer                   2.2.0
soupsieve                         2.3.2.post1
Sphinx                            3.2.1
sphinx-rtd-theme                  0.5.0
sphinxcontrib-applehelp           1.0.2
sphinxcontrib-devhelp             1.0.2
sphinxcontrib-htmlhelp            2.0.0
sphinxcontrib-jsmath              1.0.1
sphinxcontrib-qthelp              1.0.3
sphinxcontrib-serializinghtml     1.1.5
sqlparse                          0.4.2
stack-data                        0.2.0
success                           1.0.0
tables                            3.7.0
tabulate                          0.8.9
terminado                         0.15.0
testpath                          0.6.0
text-unidecode                    1.3
threadpoolctl                     3.1.0
timedb                            3.29.2
tinycss2                          1.1.1
toml                              0.10.2
tomli                             1.2.3
toolz                             0.11.2
tornado                           6.1
tqdm                              4.64.0
traitlets                         5.2.2.post1
typing_extensions                 4.2.0
urllib3                           1.26.9
versioneer                        0.22
virtualenv                        20.14.1
wcwidth                           0.2.5
webencodings                      0.5.1
websocket-client                  1.3.2
wheel                             0.34.2
widgetsnbextension                3.6.0
xeus-python                       0.13.6
xeus-python-shell                 0.2.0
yapf                              0.30.0
yarl                              1.7.2
yoyo-migrations                   5.0.5
zipp                              3.8.0
@philippjfr
Copy link
Member

I was aware of performance regressions in 0.13.0 (see #3531) but was under the impression these were fixed. You are certain you have updated to 0.13.1 in your base env (the one that the Jupyter server is running in)?

@EvyatarNezer
Copy link

Hey @philippjfr, I'm working with @ItamarShDev, we did tried 0.13.1 and even 0.14.0a1 and 0.14.0a2, but the problems listed above consisted.

@philippjfr
Copy link
Member

Thanks for the update, someone will have to do a bisection on the codebase. Can you give me a very simple, step-wise recipe that will reproduce the issue so that bisection doesn't take forever?

@EvyatarNezer
Copy link

The first two sections written above are hard to reproduce

The first one (the general slowness) is always there from the moment we starting Jupyter (with panel imported trough the startup folder).

The second one (the kernel crashes) are happening suddenly during work, and the exception mentioned above is shown in the logs.

The third one (kernel wont start / crash) I can reproduce easily in windows, I'm creating a file with just line for importing panel and save it under the startup folder (%user_home%\.ipython\profile_default\startup).

We posted the list of packages above we are using above, we are using jupyterlab version 3.4.0 and we tried panel version 0.13.* and above.

just to make it clearer when go back to older panel (0.12.7 and below) stuff are smoother, the kernel exception didn't happened during my test time and the import from windows using the startup folder worked smoothly and the kernel started without any issue.

LMK if you need any other information.

thanks.

@philippjfr
Copy link
Member

Thanks, that's very helpful!

@EvyatarNezer
Copy link

Hey @philippjfr, do you have any news for us ?

@ItamarShDev
Copy link
Author

If any tess are needed, we will be happy to test

@philippjfr
Copy link
Member

Sorry, don't have any updates yet. Been a little swamped. Hope to have some time tomorrow.

@EvyatarNezer
Copy link

Thanks for the response @philippjfr, looking forward for it.

@MarcSkovMadsen
Copy link
Collaborator

I also experience this when working on my windows laptop on the current Panel master branch.

@philippjfr
Copy link
Member

Sorry I haven't gotten to this. Unfortunately I've been travelling almost non stop for weeks and havent had access to a windows machine.

@MarcSkovMadsen
Copy link
Collaborator

MarcSkovMadsen commented Aug 3, 2022 •
edited

On this amazing youtube video by Thu Vue she has to state around minute 04:00 that you should not install Panel in the same virtual environment as Jupyterlab due to some weird issue https://www.youtube.com/watch?v=B11nK1QCPRA.

I believe this is most important bug to fix for Panel as you cannot currently work with the latest version of Panel and Jupyter lab on Windows if you are not an experienced user willing to put in some effort to get it working.

@MarcSkovMadsen MarcSkovMadsen added type: bug Something isn't correct or isn't working major issue labels Aug 3, 2022
@MarcSkovMadsen MarcSkovMadsen added this to the Version 0.14.0 milestone Aug 3, 2022
@philippjfr
Copy link
Member

Would be happy to backport this to a 0.13.x release asap if we figure out what is causing it. If someone with a windows machine could do a git bisect that would be tremendously helpful. I won't be able to do it until Sunday at the very earliest.

@ItamarShDev
Copy link
Author

@philippjfr If you could guide me through, i can give you any Windows related info.

@philippjfr
Copy link
Member

So sorry this has taken so long but I was finally able to test this on windows and I have had absolutely no luck reproducing the issue. Maybe we can sync at some point and debug together.

@philippjfr
Copy link
Member

@maximlt or @hoxbro do either of you have a windows machine where you could try to replicate?

@hoxbro
Copy link
Member

hoxbro commented Aug 22, 2022

By using the first environment on Windows and adding import panel as pn to a startup file, I get the same no response when running in Jupyter Lab. If I add another check to these lines 'jupyter_client' not in sys.modules, it seems to work as expected.

panel/panel/io/server.py

Lines 538 to 545 in 8c5ca7a

if (
sys.platform == 'win32' and
sys.version_info[:3] >= (3, 8, 0) and
tornado.version_info >= (6, 1) and
type(asyncio.get_event_loop_policy()) is asyncio.WindowsSelectorEventLoopPolicy and
'jupyter_server' not in sys.modules
):
asyncio.set_event_loop_policy(asyncio.WindowsProactorEventLoopPolicy())

I could not see any slowness except that. The log message in the first comment also seems to indicate it running on Linux and not Windows.

@philippjfr
Copy link
Member

Wow, great sleuthing thanks! Could you submit a PR adding that additional guard?

@hoxbro hoxbro mentioned this issue Aug 22, 2022
Merged
@philippjfr
Copy link
Member

Note that the IndexError: pop from an empty deque is unrelated to Panel and caused by some changes in tornado>=6.1:
jupyterlab/jupyterlab#11934

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
major issue type: bug Something isn't correct or isn't working
Projects
None yet
Milestone
Version 0.14.0
Development

Successfully merging a pull request may close this issue.

Check for jupyter_client before patching event_loop on Windows
5 participants
@EvyatarNezer @ItamarShDev @philippjfr @hoxbro @MarcSkovMadsen