Draft: Add bi-directional comms and run operations in tasks

[fleming79]

This adds supports for asynchronous bi-directional messaging from the Python objects running in the Python kernel with the JavaScript Frontend. If this is adopted, it would be a major release.

• Bi-directional comms.
• Panel can add itself to the shell.
• Add app as a property of Panel.
• Views & Models close when the Python kernel is stopped / restarted.
• Panel renders correctly on view.
• SplitPanel renders correctly on view (need help - it renders but children don't size correctly until it is re-sized in both directions) upstream issue: SplitPanel has 0 width and height jupyterlab/lumino#675.
• Support autostart via pluggy.
• Self-contained session (some errors emitted in JS console, but works OK) - need help. Support server-side execution jupyterlab/jupyterlab#15448 (listed in 4.2 Release Plan jupyterlab/jupyterlab#15801) might be useful.
• Support for execution of nested methods in the frontend.
• List attributes/methods on the frontend app.
• Main area widget with own console can hide/show itself.
• Execute code in a separate python kernel without blocking the kernel.

[github-actions]

👈 Try it on Binder (branch fleming79/ipylab/main)

[fleming79]

Regarding creating a new session without a notebook. I'm wondering if the new session should be based around a document...

Currently, the function newSession defined int utils.ts creates a new KernelWidgetManager, however the base instance doesn't provide the models that get defined using the plugin system. Currently only a limited number known are provided with the function registerWidgets defined in utils. This means that models registered using the ipywidgets plugin system don't get automatically added.

It'd be better to use registerWidgetManager from IpyWidgets, but as you can see in the prototype below, the function is expecting a DocumentRegistry.IContext<INotebookModel> context.

export function registerWidgetManager(
  context: DocumentRegistry.IContext<INotebookModel>,
  rendermime: IRenderMimeRegistry,
  renderers: IterableIterator<WidgetRenderer>
): DisposableDelegate {

[jtpio]

Thanks @fleming79 for working on this!

And sorry for the delay, I'll try to have a look soon.

[fleming79]

@jtpio - no problem. I've been learning as I go and using it for my own purposes.

As an aside, I was looking at your PR for Ipywidgets jupyter-widgets/ipywidgets#3004 from 2020 (which is still open) and saw a bit of a discussion having a widgetManager on a per-kernel basis.

Jason Grout said:
I've been going over this code and thinking about this, and there's something I think this effort is exposing about limitations in how the current lab widget manager is written. This approach (properly, I think) tries to push the notion of a widget manager down to the kernel level (i.e., the widget manager is "owned" by the kernel id, for example). However, the notebook widget manager really lives above that, at the session context level (e.g., a notebook widget manager can be "owned" by several different kernel ids over its lifetime, and right now there really isn't a provision for a notebook widget manager that doesn't happen to be associated with a kernel, as Jeremy points out above when talking about the assertion operator.

This makes me think that perhaps we should restructure the code so that we have one concept of a widget manager that is tied to a kernel, and use composition to interface with the notebook rather than inheritance. In other words, we have a layer on top of the kernel widget manager that manages notebook state and interfaces with the (kernel) widget manager to deal with saving and restoring state from a notebook. In other words, the object at the notebook level HAS a kernel widget manager (not IS a widget manager), and the kernel widget manager can be swapped out or created as needed when the kernel changes. This way consoles and notebooks can cache and share kernel widget managers freely on equal footing. Still thinking through it, but this seems to address this weird disconnect we have between notebook widget managers and kernel widget managers.

I think this concept is worth pursuing and am currently investigating how difficult it may be to implement. It would solve the issue in this PR associated with needing to create a WidgetManager and manually register widgetmodels, and the problem of widget comms shutting down when the notebook is closed.

[fleming79]

I noticed that jupyter-widgets/ipywidgets#3004 has now been merged - that's great!
Edit:
I'm wondering if the legacy function registerWidgetManager here could be used.

[fleming79]

registerWidgetManager appears to work okay with the most recent jupyterlab_widgets mentioned above by creating a dummy context.

  // For the moment we'll use a dummy session.
  // In future it might be better to support a document...
  const session = sessionContext.session;
  const context = {};
  (context as any)['sessionContext'] = sessionContext;
  (context as any)['saveState'] = new Signal({});
  (context as any).saveState.connect(() => {
    null;
  });
  registerWidgetManager(context as any, rendermime, [] as any);