Rendering stuck on cell that reads data

[Dr4cky]

Hi there,

I am having an issue and I have no idea why it is not working. The rendering gets stuck on the following cell:

path = Path(r"data.csv")
df = pd.read_csv(path, index_col=0, delimiter=';')

It doesn't show an error, it just keeps loading forever "Executing 2 out of 6".
If I just simply run this cell in my notebook, it runs instantly. Any idea what is going wrong?

Thanks!

[ggogate]

Anything in the browser Console?
Also can you try loading a different, smaller sample CSV which is comma separated.

[Dr4cky]

Anything in the browser Console? Also can you try loading a different, smaller sample CSV which is comma separated.

I just found out that it is not actually that cell that causes the problem, but the cell after:

returns_plot = widgets.Output()
with returns_plot:
    text = widgets.HTML('Time series data.')
    fig_ts = show_ts(df)
    interact(fig_ts)
    display(text)

A portion of the error that I see in the console is as follows:

id
raise web.HTTPError(404, "Kernel does not exist: %s" % kernel_id)
tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist: c5416394-05a1-485e-a9bb-595a9d5e2387)
404 GET /api/kernels/c5416394-05a1-485e-a9bb-595a9d5e2387?1702219619084 (::1) 4.17ms

[ggogate]

Try commenting out the figure part and see if the HTML displays okay. Then try moving the fig_ts part above the 'with'.

[Dr4cky]

I did a reinstall and the current error that Voila gives me is that it cannot find the module plotly, even though it is installed in my environment and I can the code in the notebook just fine. Is it calling another kernel?

[ggogate]

Try starting Jupyter/voila with --enable_nbextensions=True

[Dr4cky]

Gave me the same errors unfortunately, so plotly module not found and kernel does not exist.

[trungleduc]

Hi, could post the full reproduction code? It is easier for us if you follow the issue template.

[ggogate]

Okay, able to recreate the same on another system/vm? Usually it works very well together, so trying to eliminate any external factors.
Maybe try installing previous stable versions of libraries.
If you can share the requirements.txt using 'pip freeze' I can try to recreate it at my end.

[Dr4cky]

My code is as follows:

import numpy as np
import pandas as pd
from pathlib import Path
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets
import plotly.graph_objects as go

#%%
def show_ts(df):
    fig = go.Figure()
    for col in df.columns:
        fig.add_trace(
            go.Scatter(
                name=col,
                mode="markers+lines",
                x=df.index,
                y=df[col],
                hovertemplate=("Return: %{y:.2%}<extra></extra>"),
            )
        )
    fig.update_layout(
        title={"text": "Returns", "y": 0.9, "x": 0.5, "xanchor": "center", "yanchor": "top"},
        font_family="Times New Roman",
        title_font_family="Times New Roman",
        legend_title_font_color="green",
    )
    fig.layout.yaxis.tickformat = ",.0%"
    fig.update_xaxes(title_font_family="Arial")
    fig.update_layout(
        # height=600,
        paper_bgcolor="rgba(0,0,0,0)",
        plot_bgcolor="rgba(0,0,0,0)",
        legend=dict(font_family="Courier New", font_color="white"),
    )
    fig.show()

#%%
data = np.random.random((10, 3))
df = pd.DataFrame(data, columns=['A', 'B', 'C'])


returns_plot = widgets.Output()
with returns_plot:
    text = widgets.HTML('Time series data.')
    fig_ts = show_ts(df)
    interact(fig_ts)
    display(text)

tab = widgets.Tab([returns_plot])
tab.set_title(0, 'Data')
header_one = widgets.HTML(value="<h4>Task One</h4>", layout=widgets.Layout(width='300px'))
display(widgets.VBox([header_one, tab]))

It is stuck on 2/4. The message I see in console is:

[Voila] WARNING | Clearing invalid/expired login cookie username-localhost-8869
[Voila] WARNING | Notebook dashboard2.ipynb is not trusted
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] Kernel started: b678c387-a846-43d9-bdec-71935b14162e
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

[Dr4cky]

Okay, able to recreate the same on another system/vm? Usually it works very well together, so trying to eliminate any external factors. Maybe try installing previous stable versions of libraries. If you can share the requirements.txt using 'pip freeze' I can try to recreate it at my end.

My requirements is as follows:

ipykernel==6.27.1
ipywidgets==8.1.1
jupyterlab==4.0.9
nbformat==5.9.2
numpy==1.26.2
openpyxl==3.1.2
pandas==2.1.4
plotly==5.18.0
pre-commit==3.5.0
voila==0.5.5

Running Python version 3.12.1

[ggogate]

I was able to reproduce the issue with the same Python and library versions.
I was also able to run voila by making following changes -

1. Removing the imports to interact - #from ipywidgets import interact, interactive, fixed, interact_manual
2. Change the fig.show() to return a fig. Alternately add the with-display in the show_ts()
   #fig.show()
   return fig
3. Add a display("") before the with and move the show_ts above the with
   #%%
   data = np.random.random((10, 3))
   df = pd.DataFrame(data, columns=['A', 'B', 'C'])
   display("Before")
   fig_ts = show_ts(df)
   #display("After")
   returns_plot = widgets.Output()
   with returns_plot:
   text = widgets.HTML('Time series data.')
   display(fig_ts)
   #interact(fig_ts)
   display(text)

Alternately, move the display part inside the show_ts() function.

Attaching both versions of notebooks -
Voila-Test.zip

[Dr4cky]

Thanks for your help! Unfortunately, Voila fails to run for both of your notebooks. It seems like display(fig_ts) also does not work.

[Dr4cky]

I tried Voila version 0.4.0, after which the code does work. Why would that be?

[afonit]

fyi, I have also noticed this issue using python 3.12 and using the latest voila etc..

voila gets stuck on x of x with this in the logs:

0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] Kernel started: b678c387-a846-43d9-bdec-71935b14162e
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

Trying to read up on where best to place that or what the root cause might be.

The stopping also seems to be related to heavy ipywidget notebooks, I will post any solutions I find.

@Dr4cky , this may be related, see the comments #1395, I am looking forward to the new release to hopefully get better insight and troubleshooting.

[Dr4cky]

fyi, I have also noticed this issue using python 3.12 and using the latest voila etc..

voila gets stuck on x of x with this in the logs:

0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
[Voila] Kernel started: b678c387-a846-43d9-bdec-71935b14162e
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.

Trying to read up on where best to place that or what the root cause might be.

The stopping also seems to be related to heavy ipywidget notebooks, I will post any solutions I find.

@Dr4cky , this may be related, see the comments #1395, I am looking forward to the new release to hopefully get better insight and troubleshooting.

Thanks! I gave up trying to figure it out and just used an older version of Voila (0.4.0). Hopefully they will fix it, though,

[afonit]

I have been trying to find a minimal reproducible example to aid in the troubleshooting/solution.

What adds to the difficulty - is it can be inconsistent, for example with the all current packages in a clean enviornment:

import ipywidgets as widgets
import pandas as pd
from IPython.display import display, HTML, Markdown, Javascript

data = {'a': [1, 2, 3],
        'b': ['one', 'two', 'three'],
       }

df = pd.DataFrame(data)

dropdown = widgets.Dropdown(
    description='select one',
    options=[1, 2, 3],
    value=1,
)

base_out = widgets.Output()

with base_out:
    base_out.clear_output()
    display(Markdown("""## This is in the output"""))
    display(df)

base_out

That will render fine every time.

but if I break it up into cells with some extra getting displayed eg:

Then it will frequently get stuck on:

and if I do a CTRL+SHIFT+r then the page will refresh and load the page completely.

With nothing in the logs other than:

Jan 03 14:51:08 test-env-149200 voila[18148]: 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Jan 03 14:54:53 test-env-149200 voila[18148]: [Voila] Kernel started: 9a47c879-2b47-4855-b9f5-5a4a7fc8e0b3
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - Debugger warning: It seems that frozen modules are being used, which may
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - to python to disable frozen modules.
Jan 03 14:54:53 test-env-149200 voila[18148]: 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
Jan 03 14:54:56 test-env-149200 voila[18148]: [Voila] Connecting to kernel 9a47c879-2b47-4855-b9f5-5a4a7fc8e0b3.

[trungleduc]

Thanks for the reproduction, I will take a look at it.

[afonit]

@trungleduc, please let us/me know if there are any additional steps we could take to aid in nailing this down.

[trungleduc]

I'm able to reproduce the issue with a more compact snippet

import ipywidgets as widgets
import os
returns_plot = widgets.Output()
with returns_plot:
    display(widgets.HTML("Hello"))
    display(widgets.IntSlider())
returns_plot

with these dependencies

voila=0.5.5
ipywidgets=8.1.1
ipykernel=6.29.0
jupyter_client=8.6.0
nbclient=0.7.4                    

Voila will hang randomly after some refreshes. I think the issue is related to #1234 and #1253. Voila hangs because in this await call, nbclient does not receive the execute_reply message sent from the kernel side.

Moreover, from this comment (#1234 (comment)), it looks like this block is actually the culprit since if I remove it, Voila works as expected after multiple refreshes.

  if isinstance(stream, zmq.asyncio.Socket):
      assert stream is not None
      stream = zmq.Socket.shadow(stream.underlying)

Ping @davidbrochart @blink1073 if you have idea how to fix this.

[blink1073]

Yes, the asyncio logic is definitely convoluted, I'm working on a fix in jupyter/jupyter_client#997, but I haven't had much bandwidth recently.

[blink1073]

I don't think the issue is with that code block per se, because the effect is that it is not actually sending, because send_multipart returns an awaitable for an async socket.

[ClaytonAstrom]

Is there a better parent issue to track the discussions on this? Either on jupyter_server or something?

[jhyoo1982]

For anyone who tries to resolve this issue.
In my case, preheating kernel solved problem, although it is not an ultimate solution; it didn't make the problem when preheating kernels.

[ClaytonAstrom]

I ended up having to back port some voila 0.5 changes to a forked version of 0.4.4, using jupyter-server 1.18.1. That worked for me.