Jupyter Notebook version 7

[fperez]

Summary

This PR proposes a path forward for the user experience of the jupyter notebook application to continue to be available to users, with the upcoming version 7 based on the JupyterLab TypeScript codebase and APIs. Version 7 of the notebook will use what is today RetroLab, with suitable repo restructuring as detailed in the JEP.

In #78 I created the issue, but as indicated there, I'm moving directly to making a PR, given the extensive amount of community discussion this topic has already received. This PR should be where specifics of the proposal itself and Notebook v7 plan are discussed. Any meta/process issues can be dealt with in #78, if need be.

References

For reference, these are the main issues where this topic had been previously discussed:

• [Discuss] The future of the classic notebook interface.
• Proto-JEP.
• Early draft of the JEP in my fork, that was edited by all the JEP co-authors.
• Notes for the JupyterLab team meetings.
• RISE: new approach for JupyterLab 3.

In addition, see the Enhancement Proposal guidelines for information about the goals of the process itself.

Discussion points

✔️ = resolved
❌ = unresolved

• ✔️Reassurances on stability for extension authors (link, link)
• ✔️ Should this JEP include the scope of "Jupyter Desktop" as well? (link)
• ✔️ What do do with RetroLab? (link)
• ✔️ General opinions that the focus here should be Notebook v7 as an evolution of Notebook v6, and not as an adaptation of JupyterLab. Most important thing is similar UX with Notebook v6. (link)
• ❌ Concerns about this decision being made before the governance refactor is complete, as well as concerns that JupyterLab components is not the right foundation to use (link)

Voting from the @jupyter/steeringcouncil

• @afshin
  • Yes
  • No
• @blink1073
  • Yes
  • No
• @Carreau
  • Yes
  • No
• @damianavila
  • Yes
  • No
• @ellisonbg
  • Yes
  • No
• @fperez
  • Yes
  • No
• @ivanov
  • Yes
  • No
• @jasongrout
  • Yes
  • No
• @jhamrick
  • Yes
  • No
• @minrk
  • Yes
  • No
• @mpacer
  • Yes
  • No
• @parente
  • Yes
  • No
  • Abstain
• @rgbkrk
  • Yes
  • No
• @Ruv7
  • Yes
  • No
  • Abstain
• @SylvainCorlay
  • Yes
  • No
• @takluyver
  • Yes
  • No
• @willingc
  • Yes
  • No

[fperez]

Pinging co-authors Sylvain Corlay (@SylvainCorlay), Afshin Darian (@afshin), Sharan Foga (@sharanf), Kevin Goldsmith (@kevingoldsmith), Brian Granger (@ellisonbg) , Jason Grout (@jasongrout), Zach Sailer (@Zsailer), Jeremy Tuloup (@jtpio) that the proposal is ready!

Folks - I reordered all names alphabetically, but kept myself first as I didn't want to pop one of you "in front" (that would be @SylvainCorlay given the current names we have, unless a new co-author comes in before Corlay) without your authorization. But I'm making zero claims of authorship here, and I'm perfectly happy moving my name into the P slot as long as everyone, and particularly --at least for now-- Sylvain, are OK with being in front :)

[fperez]

Note that I edited the PR title to reflect we should allow for some public input in this full version before moving on to a vote. While we've had a lot of discussion that I think warrants putting it here, the finished document only came together this morning, and the community should have a chance to mull it over and provide input so we have a solid plan moving forward with broad support.

[choldgraf]

Hey all - just to make it explicit, and to match the title, I've converted this to a "DRAFT" in github's UI as well

[afshin]

I reordered all names alphabetically, but kept myself first ...

I am happy with the way you have it right now. Thanks, Fernando!

[meeseeksmachine]

This pull request has been mentioned on Jupyter Community Forum. There might be relevant details there:

https://discourse.jupyter.org/t/jep-draft-open-for-the-notebook-v7-transition/11769/1

[fperez]

I would like to suggest @choldgraf as shepherd for this JEP - Chris has worked a lot on the JEP process itself, has enough knowledge about the notebook and our community, but is reasonably removed from the specific decisions to provide the right balance of knowledge and impartiality, I think. I consulted and he's OK with the idea.

[ I'd originally posted this in the top summary, but best to have it here as a standalone comment for the record/discussion, as we may update the description in-place ]

[bollwyvl]

users will also have to have an NPM environment installed... snippets

Ah, but the fine print about jupyter lab build: If you use JupyterLab 2.x,

It's probably worth re-iterating:

No end user feature of Notebook 7 will require nodejs or any other non-pip installable package

I think by the time Notebook 7 lands (some time after Lab 4), all important extensions where the developers intended to supporting them at all will already offer one-step, pip installable pre-built extensions.

They want a single step or command for installing extensions.

As with nbextensions in Notebook <7, end users will be able to pip install or conda install new web UI features, requiring a browser reload to use new UI features.

There were certain ways this could be made to work well in Notebook <7 without a page reload (e.g. widgets), but we could never offer sound compatibility guarantees for everything: just slamming some JS out of a kernel into an output and expecting a new top-level menu item to show up is not a capability we can maintainably offer in Notebook 7.

Server extensions (e.g. new REST APIs) will continue to require a restart of the notebook server... but this has always been a system property of the notebook server. This could theoretically be fixed at an architectural level... but has the implicit potential, especially in a managed/Hub setting, of leaving a user with a completely wedged, unstarteable, uninspectable, remote web server, which has always prevented anyone from pushing too much harder on it.

we should also include a "good enough" provider of extensions

Yeah, it would be nice... but as there never was an in-app "Extension Store," in Notebook <7, I don't think, it's in-scope for this JEP.

[gutow]

It's probably worth re-iterating:

No end user feature of Notebook 7 will require nodejs or any other non-pip installable package

I think by the time Notebook 7 lands (some time after Lab 4), all important extensions where the developers intended to supporting them at all will already offer one-step, pip installable pre-built extensions.

This is good. I think that would work with my end users. Are there clear examples I can follow that do this using menus that inject code into selected cells (the key thing I do)? My testing with the examples I have found do not work as simple pip installs. I've actually had trouble getting some of them running at all. Maybe, this reflects things changing "under-the-hood" since they were last updated? Is the API still changing a lot?

There were certain ways this could be made to work well in Notebook <7 without a page reload (e.g. widgets), but we could never offer sound compatibility guarantees for everything: just slamming some JS out of a kernel into an output and expecting a new top-level menu item to show up is not a capability we can maintainably offer in Notebook 7.

I do not wish to just "slam some JS out of the kernel". However, there are many use cases where the menus available are context sensitive (in my case some examples might be: am I collecting data using a hardware A-to-D versus processing data versus creating a worksheet for students). Thus, there needs to be clearly documented ways to control what menus and buttons show via python (ideally, also any other language running in the kernel), because a specific kernel is running, and also through the settings menu. If the API is stable this should be doable, IMHO.

[bollwyvl]

Are there clear examples I can follow that do this using menus that inject code into selected cells (the key thing I do)?

The snippets example you linked to has most of the pieces. Similarly, the dask-labextension shows some approaches for working with the context menu.

trouble getting some of them running at all

without any more info, it's hard to say... but the venue for further discussion is likely issues on the extensions themselves, where it could prove out no, this developer does not intend to support this extension (which is of course their choice).

ways to control what menus and buttons show via python

At present, there isn't a way for (any) kernel to know some "client name" against which they are running (it could be multiple, simultaneously). Again, this could be made more lax, but brings a host of other deep issues, and is outside the scope of this JEP. But basically, kernel-emitted JS code that only works in one client (e.f. Notebook <7) probably already didn't work in any other first-party (lab, nbconvert) or third-party client (nteract, vscode, spyder, colab). Part of the maintainability issue we are addressing here is precisely this... perhaps widgets or mimetypes for these kinds of things are the way to move forward. had a comment about that earlier, but it appears to be gone.

[choldgraf]

Hey all - we're now outside the "final comment period" minimum window. I'll note that several @jupyter/steeringcouncil members haven't given their thoughts or a vote yet. Could somebody send another message to that group (or tell me how I can best get in touch with the group) to let them know we'd like final thoughts and comments before moving forward?

Also, after reading the conversation above, between @gutow and @bollwyvl , I don't believe that this requires changes to the JEP language itself, since we're all in agreement already that the most important extensions should be pip-installable alone by the time NTBKv7 comes out. Do you agree, or should the language be updated?

[gutow]

ways to control what menus and buttons show via python

At present, there isn't a way for (any) kernel to know some "client name" against which they are running (it could be multiple, simultaneously). Again, this could be made more lax, but brings a host of other deep issues, and is outside the scope of this JEP. But basically, kernel-emitted JS code that only works in one client (e.f. Notebook <7) probably already didn't work in any other first-party (lab, nbconvert) or third-party client (nteract, vscode, spyder, colab). Part of the maintainability issue we are addressing here is precisely this... perhaps widgets or mimetypes for these kinds of things are the way to move forward. had a comment about that earlier, but it appears to be gone.

I understand this to mean that I would have to have my students launch different versions of JLab depending on their use case. At that point, might I be better off writing dedicated applications using QT or wx? However, I do not really want to have to duplicate the very useful simple notebook interface. The other alternative is to have a single heavy weight version of JLab with lots of menus, buttons, etc. that they don't need most of the time. In that case, I would be producing a specialized version of JLab that would have many of the problems for my use case that this JEP is trying to find a path to avoid.

I am also confused by "there isn't a way for (any) kernel to know some "client name" against which they are running." One should be able to look at either existing DOM elements or interfaces to determine what is appropriate. If the necessary information is completely hidden or obfuscated, then I believe you have lost one of the key advantages of the classic notebook.

I have no objection to having to maintain two different javascript codes during the transition from the old notebook to a JLab based one. There just needs to be a clear way to do that. I am struggling with getting what were relatively simple things to do in the old notebook working in JLab. I believe it is primarily due to the opacity of the API and lack of good examples. When I say opacity, I mean it is often hard to tell without stepping back through hundreds of lines of code exactly what options, variables, etc to pass to the high level API calls. I think a concerted effort on documentation and examples would go a long way to improving this situation.

I am strongly in favor of finding a path forward. I am just worried that as mentioned by @choldgraf and @Carreau that too much of the capabilities available when working within the Classic Notebook will be lost for many of us to effectively make the transition. I personally would like to see more language concerning support for more of the on-the-fly flexibility of the classic notebook. I do understand that that will be less compatible with things outside of JLab and has some security issues. From my perspective the ease of producing something I can use for 5 - 10 years with less development overhead is more important.

[ivanov]

Could somebody send another message to that group (or tell me how I can best get in touch with the group) to let them know we'd like final thoughts and comments before moving forward?

I just sent a nudge to the group and the individuals who haven't registered their decisions here, @choldgraf

[rgbkrk]

Thanks for the nudge.

[rgbkrk]

Let's move the notebook forward!

[Zsailer]

It's worth noting here that there's a live, weekly discussion happening around this JEP and the pending work in the weekly Notebook meeting. Here is a record of the notes from previous weeks: jupyterlab/frontends-team-compass#133.

As always, these meetings are open to anyone and everyone. Feel free to join the conversation. It's always a hoot! 😄

[jasongrout]

Thank you very much for moving this forward!

[choldgraf]

Hey all - we have had a few weeks of time for comments and conversations, and we've gone through many rounds of discussion and edits (both in this PR and in the linked issues). Here's a tally of the @jupyter/steeringcouncil votes so far:

• Yes: 11
• No: 1
• Abstain: 2
• No response: 3

Proposal

As we have a strong majority of the steering council voting in favor of this, and have had a lot of productive conversations thus far in other issues and here, I propose the following steps:

• Leave this PR open for final comments until Tuesday the 21st.
• Unless a major new piece of information arises that has not already been discussed, we merge this PR at EOD that Tuesday.
• I will update the top comment with a summary of the discussion here (with a focus on major concerns people have that could impact implementation decisions), so that it is easier to refer to in the future.
• We consider this JEP accepted

[ivanov]

I was asked privately why I voted against this proposal, and my reply was deemed to contain aspects of the issues at hand that were not covered in this or some of the other threads around the proposal, so I am reposting the bulk of it here.

This isn't exhaustive but here goes. I don't agree with changing the entire project and keeping the same name, which is what notebook7 will be. I can't think of quite an equivalent thing here, but it would be like installing notepad, but getting Word that's themed to look like notepad. I guess the positive example here was Mozilla Application Suite / SeaMonkey, the re-written browser component of which is where Firefox came from, but when Firefox came out, it was a new thing. People didn't get it as an update to SeaMonkey. It would have been more responsible to make a new name for this and let people migrate to it. I also do not think JupyterLab as the code base is the thing to get behind. Additionally, I do not think there is any urgency to considering this change now - we've been in limbo on governance and decision making for the entire project for quite some time, and yet this was pushed through without wrapping up the governance refactor before prioritizing other business. I understand that it is difficult to push forward on any front with so much up in the air, but I would have preferred for us to wrap up to the new decision making process and navigated the decisions of this JEP in the new governance configuration.

[Carreau]

I don't agree with changing the entire project and keeping the same name, which is what notebook7 will be. I can't think of quite an equivalent thing here, but it would be like installing notepad, but getting Word that's themed to look like notepad.

While I understand this point I will point you toward GCC/EGCS. I think that if the replacement is good/close enough it is worth it to have a more cohesive community. But again this is not trying to convince you, just maybe giving you a historical example. This is also why I've strongly pushed toward not phrasing comparison WRT jupyterLab, but notebook v6.

I also do see the reason for the same name with respect to user muscle memory being to pip install notebook

[blink1073]

The voting window has completed, merging as accepted, thanks everyone!

[choldgraf]

Just wanted to say thanks @ivanov for providing your thoughts here - I think it complements @Carreau's thoughts as well, that Notebook v7 should be thought of as "an evolution of Notebook v6" rather than "an adaptation of JupyterLab". And thanks @blink1073 for merging it in 👍

I will update the top comment with some of the major conversation points here, so that is it easier to discover in the future if people refer back to this PR thread.