Feature: save additional mime bundles in notebook #3107

maartenbreddels · 2021-02-09T15:59:23Z

This is a proof of concept for allowing embedding of additional entries in the mime bundle, such as static images, svg, video etc from the front-end.

The basic idea is to have an API Widget.generateMimeBundle which will generate additional mime entries, which can be overridden in widget subclasses. Bqplot can inject svg, core ipywidgets can inject HTML for the HTML widget, an image for the Image widget etc.

TODO:

Get approval/blessing for this approach in general.
See if we can match the cell output to a particular view, and as the view for a mime bundle, because the output may be specific to the view (such as in ipyvolume and ipywebrtc, otherwise fallback to the model)
Implement this in jupyter lab
Remove the dummy text/html mime entry.
Implement this for a few widgets (Image, HTML)

This PR differs from #2282 in just having an API for providing the API, instead of specific methods for particular representation (although the two PR's can be seen as orthogonal).

Related: #314 #2280 #2282 higlass/higlass-python#49 glue-viz/glue-jupyter#207

maartenbreddels · 2021-02-09T16:01:28Z

As a concrete example:

import ipywidgets as widgets
widgets.Image.from_url('http://localhost:8891/static/base/images/logo.png')

Which will end up in the notebook JSON file as:

...
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "09f05d25048e449bba162bf1f25c023e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<h1>Hi!</h1>"
      ],
      "text/plain": [
       "Image(value=b'http://localhost:8891/static/base/images...', format='url')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "widgets.Image.from_url('http://localhost:8891/static/base/images/logo.png')"
   ]
  },
...

maartenbreddels · 2021-02-09T16:02:30Z

Of course, matplotlib/ipympl#16 and matplotlib/ipympl#294 and matplotlib/ipympl#150 are related as well, and big motivation for this from @martinRenou 's side I think.

martinRenou · 2021-02-09T16:30:40Z

packages/base/src/widget.ts

+    return Promise.resolve({
+      'application/vnd.jupyter.widget-view+json': {
+        model_id: this.model_id,
+        version_major: 2,


Should we take the version from what the model defines instead?

I forgot to mention that this should be removed, actually

Oh ok :) 👍🏼

martinRenou · 2021-02-09T16:32:09Z

widgetsnbextension/src/manager.js

+    this.notebook.events.on('before_save.Notebook', async () => {
+      var cells = Jupyter.notebook.get_cells();
+      // notebook.js save_notebook doesn't want for this promise, we are simply lucky when this
+      // finishes before saving.


This seems very weak :S

Yes, I think this means it will be a lab feature only

What about not making it a Promise?

we to await the model, and making `generateMimeBundle async also makes sense, since it will involve async cases in bqplot at least.

martinRenou · 2021-02-09T16:34:42Z

Thanks a lot @maartenbreddels for trying this. I will give it a try.

maartenbreddels · 2021-02-09T17:57:56Z

I cleaned it up a bit, and implemented this for HTML and Image.

For instance, GitHub renders the HTML:
https://gist.github.com/maartenbreddels/695c2eac56d8ff2114998e69f93296d9

maartenbreddels · 2021-02-09T18:20:27Z

Bad news, it seems we cannot do this in Lab either. You can connect to the save event, like we do with saving the widget state:

    context.saveState.connect((sender, saveState) => {
      if (saveState === 'started' && settings.saveState) {
        this._saveState();
      }
    });

But again, this is sync only https://github.com/jupyterlab/jupyterlab/blob/b4db7f03d2ad0b91a9c8c252c56ae5e4a9408fbb/packages/docregistry/src/context.ts#L506

Maybe we can make it sync only for now, and it will work in classic and lab.

willingc · 2021-02-09T18:27:23Z

@MSeal fyi

MSeal · 2021-02-09T19:08:43Z

Adding support for this would be quite helpful!

It would also be really nice if we made this work across UIs (remember there's a dozen of so proprietary UIs as well). Working in classic, lab, and maybe nteract -- I think those are the 3 most popular open source UIs -- would be ideal to ensure it works in other environments. In lieu of that support, we should at least consider documenting changes needed by a UI that partially or completely supports widgets currently. e.g. 8.0 Widget UI Interaction changes section in the changelog?

ianhi · 2021-02-09T20:01:32Z

This will also require changes on all the frontends? My understanding from conversation on gitter awhile ago https://gitter.im/jupyter-widgets/Lobby?at=5f0ea2e30d37916fda770757

is that the frontend chooses the precedence in the mimebundle and currently lab et al always give the widget mimetype higher precedence.

MSeal · 2021-02-10T00:01:45Z

is that the frontend chooses the precedence in the mimebundle and currently lab et al always give the widget mimetype higher precedence.

It seemed from comments that UIs would need to adjust their rendering code some to look for mimetypes within the bundle and that saving those mimetypes in the bundle had UI specific limitations.

maartenbreddels · 2021-02-10T10:47:57Z

frontend chooses the precedence in the mimebundle and currently lab et al always give the widget mimetype higher precedence.

How lab/classic/nbconvert uses the mime bundle is something we can disagree about. I think we should have a dropdown to choose/override this default, in lab, classic and nbconvert, but we don't need to address that now I think.

The main point of this PR is to have an API to ask for an extra mime bundle, an example use case (HTML and Image widget), and an implementation example for Lab and/or classic notebook on how to use that API (inject it on save).

On that last point, it seems technically not possible, for classic and lab jupyterlab/jupyterlab#9764 if generating the mime bundle needs to be async.

ianhi · 2021-02-10T16:03:53Z

frontend chooses the precedence in the mimebundle and currently lab et al always give the widget mimetype higher precedence.

How lab/classic/nbconvert uses the mime bundle is something we can disagree about. I think we should have a dropdown to choose/override this default, in lab, classic and nbconvert, but we don't need to address that now I think.

I think I agree with you? I was/am just concerned is that ipywidgets doesn't have the ability to do that overriding. So this may require work outside of ipywidgets.

The main point of this PR is to have an API to ask for an extra mime bundle

👍

martinRenou · 2021-02-11T13:08:25Z

How would this play with #3114?

maartenbreddels · 2021-02-11T13:35:19Z

They should play nice together and are independent (although I'd like both of them in 8.0), I'm working on this branch now, and I'm reusing some of the code from there to make it possible to do this in a sync way.
Both methods simply listen to the same save event.

martinRenou · 2021-02-11T14:39:05Z

widgetsnbextension/src/manager.js

+      // finishes before saving.
+      await Promise.all(
+        cells.map(async cell => {
+          var widget_output = cell.output_area.outputs.find(output => {


I am getting the following error from this line:

manager.js:228 Uncaught (in promise) TypeError: Cannot read property 'outputs' of undefined at manager.js:228 at Array.map (<anonymous>) at window._Events.<anonymous> (manager.js:227) at window._Events.dispatch (jquery.min.js:2) at window._Events.v.handle (jquery.min.js:2) at Object.trigger (jquery.min.js:2) at window._Events.<anonymous> (jquery.min.js:2) at Function.each (jquery.min.js:2) at S.fn.init.each (jquery.min.js:2) at S.fn.init.trigger (jquery.min.js:2) (anonymous) @ manager.js:228 (anonymous) @ manager.js:227 dispatch @ jquery.min.js:2 v.handle @ jquery.min.js:2 trigger @ jquery.min.js:2 (anonymous) @ jquery.min.js:2 each @ jquery.min.js:2 each @ jquery.min.js:2 trigger @ jquery.min.js:2 events.trigger @ events.js:31 Notebook.save_notebook @ notebook.js:2735 Notebook.save_checkpoint @ notebook.js:3333 handler @ actions.js:885 final_actions.<computed>.handler @ actions.js:934 ActionHandler.call @ actions.js:1019 ShortcutManager.call_handler @ keyboard.js:554 KeyboardManager.handle_keydown @ keyboardmanager.js:215 (anonymous) @ keyboardmanager.js:181 dispatch @ jquery.min.js:2 v.handle @ jquery.min.js:2 async function (async) (anonymous) @ manager.js:226 dispatch @ jquery.min.js:2 v.handle @ jquery.min.js:2 trigger @ jquery.min.js:2 (anonymous) @ jquery.min.js:2 each @ jquery.min.js:2 each @ jquery.min.js:2 trigger @ jquery.min.js:2 events.trigger @ events.js:31 Notebook.save_notebook @ notebook.js:2735 Notebook.save_checkpoint @ notebook.js:3333 handler @ actions.js:885 final_actions.<computed>.handler @ actions.js:934 ActionHandler.call @ actions.js:1019 ShortcutManager.call_handler @ keyboard.js:554 KeyboardManager.handle_keydown @ keyboardmanager.js:215 (anonymous) @ keyboardmanager.js:181 dispatch @ jquery.min.js:2 v.handle @ jquery.min.js:2

Not sure it's normal that cell.output_area is undefined. We might want to add a check for undefined before this line?

I suppose you tested with only one cell containing an output. When there is no output under one cell in the Notebook, cell.output_area seems to be undefined.

Thanks, that should be fixed now

martinRenou · 2021-02-11T14:54:23Z

Tried with ipympl, I saved the image doing:

generateMimeBundle() {
    return Promise.resolve({
        'text/html': `<img src="${this.offscreen_canvas.toDataURL('image/jpeg', 1.0)}"
                        width="${this.offscreen_canvas.width}" height="${this.offscreen_canvas.height}"/>`
    });
}

It seems to work! But when I reopen the saved Notebook I see the image for ~half a second, then it gets replaced with the following text:

My understanding is that there is still the widget repr that tries to render but fails because the model state does not exist. It would be nice to remove the widget repr in the case of ipympl, would this be possible?

maartenbreddels · 2021-02-11T18:50:02Z

It seems to work! But when I reopen the saved Notebook I see the image for ~half a second, then it gets replaced with the following text:

Yes, I think that is still expected behaviour / separate issue. I think we should have a dropdown to select which mime entry to show.

maartenbreddels · 2021-02-11T18:56:36Z

I've done a refactor of this.

We now ask the view to generate the mimebundle. Jupyter notebook will call on notebook saving, the generateMimeBundle for each view, so this function should not be too expensive, and cannot be async.

A new action (and menu item) "'Add extra to output cell mime bundle'" is added, which will call generateMimeBundleExtra , which can be async, and be more expensive since it will not be called on each save operation. I think bqplot should use this to inject the svg/png into the mime bundle, video widgets could embed the video, and a possible future Screeenshot widget can use html2canvas to inject a screenshot of it's child widget into its mime bundle.

This approach makes it such that injection of mime bundles for common cases can be done sync, and thus be done on save in classic notebook and jupyter lab, while at the same moment allow for more expensive and async generation of mime bundles on user request.

martinRenou · 2021-02-12T07:31:39Z

I think we should have a dropdown to choose/override this default, in lab, classic and nbconvert, but we don't need to address that now I think.

We should maybe brainstorm a bit about this. IMO a dropdown would not be a great UX.

Edit: If we go for the dropdown idea, maybe we should provide a way for custom widget libraries to provide a sensible default repr.

maartenbreddels · 2021-02-12T09:25:30Z

It's good to consider a few ideas on how to build on top, but indeed a dropdown is not the best UX, we could take a look at https://zeppelin.apache.org/docs/0.8.0/usage/display_system/basic.html for inspiration, but for sure a separate issue/PR.

Let's first see if we can agree on populating the mime bundle in a user-friendly way. What do you think of the latest idea (auto save cheap mime bundles, expensive ones on user request)?

martinRenou

Yeah, I think having two approaches like that is good :)

I also agree it's better to have it in the view.

martinRenou · 2021-02-12T09:58:07Z

jupyterlab_widgets/src/manager.ts

-  /**
-   * Register a widget model.
-   */
-  register_model(model_id: string, modelPromise: Promise<WidgetModel>): void {


Was it intended to remove those two methods?

Yes, they went up a base class (base manager), because we need to same feature for the notebook manager.

packages/controls/src/widget_image.ts

martinRenou · 2021-02-12T10:05:27Z

How confident are you that we could make a JupyterLab implementation? I've tried something very similar here: matplotlib/ipympl#294 but it didn't work for some reason. It seems to me that ipywidgets was overwriting whatever I was putting in the Notebook instance, but I might be wrong.

maartenbreddels · 2021-02-12T10:30:37Z

I am totally lost in Lab-land, I cannot find an easy way to match the WidgetView to the output area (in classic we have access to the DOM, so we test with element.contains), if you have ideas on that, I can help you with the rest.

martinRenou · 2021-02-12T10:38:45Z

Maybe @jtpio would have some ideas?

martinRenou · 2021-09-13T12:48:32Z

Rebased

martinRenou · 2021-09-13T15:22:37Z

@maartenbreddels if you don't mind I'll iterate on this and will try to implement the same for JupyterLab.

martinRenou · 2021-09-13T15:25:13Z

I'd also like to add a way for widgets to completely overwrite the mimebundle if they want. This would be useful for ipympl, because it has been asked many times to save a static image of the plot when saving the Notebook, which I believe make sense for ipympl.

But this should not be the default for all widgets.

This will be useful for widgets like ipympl or ipycanvas, where the model does not contain enough information to recover the widget (the kernel only contains the information about the state of the plot) so we need to store a static version of the plot (static image) for convenience, and we need to dump the widget model.

maartenbreddels · 2021-09-14T19:19:20Z

packages/base/src/widget.ts

+  async generateMimeBundle() {
+    return {};
+  }
+
+  /**
+   * Whether the `generateMimeBundle` output should
+   * overwrite the mimeBundle or extend it
+   */
+  shouldOverwriteMimeBundle(): boolean {
+    return false;
+  }
+


What about generateMimeBundle making the default mimebundle (with the widget in it).
subclases can then do:

return {'text/html': view.el.outerHTM, ... await super.generateMimeBundle()} // or return {'text/html': view.el.outerHTM}

depending on if they want to overwrite it. Does that make sense?

Makes sense!

martinRenou · 2021-09-20T07:05:12Z

@maartenbreddels this is becoming less critical for ipympl, we're looking into always putting a static version of the plot in matplotlib/ipympl#343

blois · 2021-09-23T15:45:46Z

Rather than a hook on save would it be possible to have an API like widget.set_output_mime_data(mimetype, data, metadata) which would update any display items for that widget with the new data? One benefit of this is that it allows the document to be invalidated (prompt for save). Widgets like ipympl would probably need to debounce this to avoid generating base64 image data too often.

The other side of this is allowing widgets to access this data- ipympl should be able to reach back to the display item to get the image data when re-rendering.

ianhi mentioned this pull request Feb 9, 2021

Saved ASCII output from tqdm.notebook shows zero progress tqdm/tqdm#1120

Open

8 tasks

martinRenou reviewed Feb 9, 2021

View reviewed changes

maartenbreddels mentioned this pull request Feb 9, 2021

Perform async task before saving notebook jupyterlab/jupyterlab#9764

Open

martinRenou reviewed Feb 11, 2021

View reviewed changes

martinRenou mentioned this pull request Feb 11, 2021

Implement generateMimebundle matplotlib/ipympl#300

Closed

maartenbreddels force-pushed the feat_embed_mime_bundle branch from 74f0763 to cdfcec9 Compare February 11, 2021 18:45

maartenbreddels marked this pull request as ready for review February 12, 2021 09:27

martinRenou reviewed Feb 12, 2021

View reviewed changes

jasongrout mentioned this pull request Feb 18, 2021

2021 Jupyter Widgets Dev Meetings jupyter-widgets/team-compass#1

Closed

astrojuanlu mentioned this pull request Jul 12, 2021

Widgets do not render in static documentation #3235

Closed

maartenbreddels added 4 commits September 13, 2021 14:47

feat: save additional mime bundles in notebook

a7b69ef

use async, and do not return widget mime type

883bbd3

example for HTML and Image

4f3f57b

fix: skip non-existing models

e1c5ab1

martinRenou force-pushed the feat_embed_mime_bundle branch from c98e2d0 to e1c5ab1 Compare September 13, 2021 12:48

martinRenou added 3 commits September 13, 2021 17:32

Do not fail if there is no output_area

ffd91d2

Lint

f4c89fa

maartenbreddels commented Sep 14, 2021

View reviewed changes

maartenbreddels mentioned this pull request Jan 12, 2023

Interactive Plots Not Included in Jupyter Notebook Print or Download #3073

Open

Feature: save additional mime bundles in notebook #3107

Are you sure you want to change the base?

Feature: save additional mime bundles in notebook #3107

Conversation

maartenbreddels commented Feb 9, 2021 • edited

maartenbreddels commented Feb 9, 2021

maartenbreddels commented Feb 9, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinRenou commented Feb 9, 2021

maartenbreddels commented Feb 9, 2021

maartenbreddels commented Feb 9, 2021

willingc commented Feb 9, 2021

MSeal commented Feb 9, 2021

ianhi commented Feb 9, 2021

MSeal commented Feb 10, 2021

maartenbreddels commented Feb 10, 2021

ianhi commented Feb 10, 2021

martinRenou commented Feb 11, 2021

maartenbreddels commented Feb 11, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinRenou commented Feb 11, 2021 • edited

maartenbreddels commented Feb 11, 2021

maartenbreddels commented Feb 11, 2021

martinRenou commented Feb 12, 2021 • edited

maartenbreddels commented Feb 12, 2021

martinRenou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinRenou commented Feb 12, 2021

maartenbreddels commented Feb 12, 2021

martinRenou commented Feb 12, 2021

martinRenou commented Sep 13, 2021

martinRenou commented Sep 13, 2021 • edited

martinRenou commented Sep 13, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

martinRenou commented Sep 20, 2021

blois commented Sep 23, 2021

maartenbreddels commented Feb 9, 2021 •

edited

martinRenou commented Feb 11, 2021 •

edited

martinRenou commented Feb 12, 2021 •

edited

martinRenou commented Sep 13, 2021 •

edited