Educational JupyterHub considerations for notebook v7 switch

[yuvipanda]

Very excited about the upcoming notebook v7! With help from @jtpio, we (Berkeley) already moved a 1500 beginner data8 class to retrolab (berkeley-dsep-infra/datahub#2422) and it has gone super well!

One of the most important considerations from instructors when we were pitching them the switch is that there was always an option to switch back to the classic notebook whenever they encounter issues. This actually led me to working on jupyterlab/retrolab#158, and getting that in before the semester started was very important in convincing instructors to switch. The pitch was - we are changing the default, but the older one is still available. I think everyone enjoyed retrolab, but it definitely took some time to move.

The other important aspect here is that a single JupyterHub often serves many different classes, 20-30, and they often have different UI needs. In our primary hub, we offer classic notebook, jupyterlab, RStudio and retrolab. The nbgitpuller link generator is used by instructors to pick which UI they want their students to launch into. All UIs are fairly heavily in use, and we've been slowly moving people away from classic notebook to jupyterlab or retrolab.

What I'd love to do with the notebook v7 transition is:

1. Offer the new UI as default!
2. Allow people to switch back to jupyterlab or classic UI (offered via nbclassic) if needed
3. Do not break existing nbgitpuller URLs, as they already pick a UI they want to be sent to. These URLs are in course websites, emails sent out to students, and some on bit.ly links printed on textbooks. https://www.w3.org/Provider/Style/URI.
4. Switch nbgitpuller.link to point to the new interface as well by default.

As a JupyterHub admin, I expect to have to put in a bunch of work to make this happen - notebook v7 is after all a big breaking change!

Here's what I'd love for how this works.

1. New endpoints being served by notebook v7 have a /v7/ (or similar?) URL prefix. This is exactly what we do with retrolab now, with the /retro/ prefix.
2. By default, with just the notebook package installed, URLs without a prefix will just redirect to /v7/. So /tree/ will redirect to /v7/tree/, so if you just have the notebook package installed, all existing links will just take you to the new UI.
3. However, allow the option to continue using nbclassic interface via the nbclassic package, provided under a /v6/ URL prefix. This will not take over / by default, but allow code to switch to this if it needs to. I think some work needs to be done in the nbclassic package for it to actually ship all the JS, instead of just shimming to notebook package as it does now.
4. Provide a traitlet or other option that lets admins control what / actually redirects to - /v6/ or /v7/.

This has a few advantages:

1. Users don't lose choice, but we still change the default.
2. This doesn't commit the notebook team to long term maintenance of classic UI, but does require a short term work in moving stuff over to the nbclassic package.
3. The default experience users get is still the new and maintained interface
4. URL versioning in general IMO is a good thing, and if necessary lets us evolve the notebook package in the future.
5. JupyterHub admins responsible for educational hubs everywhere in the world will thank you with all their hearts.

This is an exciting transition, and I'm very grateful for all the folks who have put in a lot of emotional and technical work into making this happen. Feels like we're on the home stretch, and this will hopefully help make the transition easier for everyone.

Thanks a lot to @Zsailer and @jtpio for conversations around this.

[fcollonval]

I'm not convinced that this is the responsibility of the frontend to define the prefix of the tool. The versioning as suggested here is actually not proper as versioning is about server API not the frontend (although I would also love the server API to get versioned 😉). And in this particular case the server is the same and the endpoints won't change.

However, as mentioned by @williamstein in the notebook meeting, using the base URL to distinguish the frontends and documenting how to use that approach (if it is possible) sounds like a good way forward and JupyterHub is a proxy after all; routing is what it does.

The question is probably more should nb v6 and v7 be installed in the same environment. The current answer is no and we should consider providing both similarly as providing RStudio next to JupyterLab.

[jtpio]

Thanks @yuvipanda for raising this and the detailed write-up 👍 And for promoting RetroLab / Notebook v7 to be adopted for Data 8, this is really exciting 🎉

As mentioned above, prefixing the frontend endpoints with /v7 might feel awkward. In the Notebook weekly meeting yesterday someone in the chat mentioned that URLs are user interfaces. The idea with the v6 to v7 transition is to keep the same set of endpoints without redirect. Users are already used to going copy pasting URLs to notebooks from the browser URL / search bar and it would be surprising if a new /v7 pops up.

Do not break existing nbgitpuller URLs, as they already pick a UI they want to be sent to. These URLs are in course websites, emails sent out to students, and some on bit.ly links printed on textbooks. https://www.w3.org/Provider/Style/URI.

The URLs will not break because Notebook v7 is taking the same endpoints as Notebook v6. The nbgitpuller links will just redirect to the new interface (via /tree, /notebook, /edit...).

We also need to clarify what the proposal of having v6 and v7 cohabiting means:

• are we talking about having the full Notebook v6, including the Classic Notebook Server, running side-by-side Notebook v7 (Notebook v7 runs on Jupyter Server)?
• or is it just the Notebook v6 front-end that would be served next to the Notebook v7 frontend?

Because nbclassic runs on Jupyter Server and not the Classic Notebook Server. If it's only about the v6 frontend, then probably this could work if the static assets are moved to nbclassic and served behind /v6/ or /classic/ or /nbclassic/.

This doesn't commit the notebook team to long term maintenance of classic UI, but does require a short term work in moving stuff over to the nbclassic package.

Right, but continuing to provide the classic v6 UI could give the wrong message that we are still maintaining it, which hasn't been the case for many years since the original plan was to even sunset notebook. Maybe there should be some kind of banner at the top of the classic notebook v6 UI to say this is deprecated software, and encourage the switch to v7 since there won't be any new features added to the classic v6 UI.

[yuvipanda]

Thanks for the responses, @fcollonval and @jtpio. I really appreciate all the work you and others are doing on this.

Just to answer one specific question,

We also need to clarify what the proposal of having v6 and v7 cohabiting means

This is only about the v6 frontend, and doesn't require any other API endpoints. I agree with you that this would mostly entail moving the assets to nbclassic (or another package - @Zsailer had some more specific ideas here).

I think there are two primary issues here:

1. Make it possible for users to switch between v7 and v6 user interfaces on the same JupyterHub. This requires both interfaces are available side-by-side. Moving static assets to another package (nbclassic? elsewhere?) and making it available under a prefix (I like /classic) would be one solution to this. jupyter-server will be the only thing running, no classic notebook server endpoints.
2. Should the new v7 interface use a URL prefix, with the default no-prefix URLs redirecting to this prefixed URL, but hub admins be able to configure what the default no-prefix URL redirects to, and it could be /classic/. I'll think about better ways to express this and explore other options.

(2) seems to be a bit more contentious than (1).

[yuvipanda]

The URLs will not break because Notebook v7 is taking the same endpoints as Notebook v6. The nbgitpuller links will just redirect to the new interface (via /tree, /notebook, /edit...).

Ah, apologies for using an unclear version of 'break'. Instead, let's say they might end up with a different UI than what they selected in nbgitpuller.link:

Note that the option there explicitly says Classic Jupyter Notebook, rather than just Jupyter Notebook.

There are many layers this problem can be solved - including by 'fixing' this in nbgitpuller itself. I'll think about this for another day or two. But it does fundamentally require that both the classic notebook UI and the new notebook UI are available on the same server, under different URLs.

[fcollonval]

Trying to draw diagrams to describe the scenarii - don't hesitate to edit to correct them

• Scenario A (current state):

flowchart TD
    JupyterLab --> Server[Jupyter Server]
    RetroLab --> Server
    Notebook[Notebook v6] --> Server

• Scenario B (@yuvipanda wish):

flowchart TD
    JupyterLab --> Server[Jupyter Server]
    RetroLab[Notebook v7] --> Server
    Notebook[Notebook v6] --> Server

• Scenario C (what I'm supporting):

flowchart TD
    JupyterLab --> Server[Jupyter Server]
    RetroLab[Notebook v7] --> Server
    Notebook[Notebook v6] --> Server2[another Jupyter Server]

One important issue with scenario B is linked to future developments. If we want to break the Jupyter server API, we will either:

• break scenario B and force scenario C; so it is better to not support scenario B at all.
• publish a server with versioned API that can serve both the new and the old API to support scenario B.

[jtpio]

Note that the option there explicitly says Classic Jupyter Notebook, rather than just Jupyter Notebook.

Maybe we should also consider dropping the "Classic Jupyter Notebook" from the nbgitpuller. And instead have the following entries:

• Jupyter Notebook
• JupyterLab
• RStudio
• Shiny
• Custom URL

As a side effect, there will be no difference for those who have pinned on notebook==6. For those who have notebook=7 and supposedly nbclassic with the v6 frontend, they would have to make the extra effort of constructing the URL themselves (or just tweaking it to add /classic). By making it harder to find we also try to encourage folks to not use the deprecated v6 frontend, while still leaving the door open for those who want to.

[yuvipanda]

@fcollonval aha, so your fundamental concern is that we don't want to constrain the server endpoints to continue to be compatible with classic notebook UI. That if the classic notebook UI is still talking to the same jupyter-server endpoints as the new UI, there will need to be fixes to the classic notebook UI if something breaks. Is that an accurate summary of the primary concern?

I agree that's definitely not long term sustainable, and not something I want us to do at all.

[yuvipanda]

@fcollonval to expand on scenario C, it'd look more like this

flowchart TD
    JupyterLab --> Server[Jupyter Server]
    RetroLab[Notebook v7] --> Server
    jsp[jupyter-server-proxy] --> Server
    notebookv6[Notebook API Server] --> jsp
    notebookv6UI[Classic Notebook UI] --> notebookv6

The most important difference from your scenario B (which was what I was advocating for!) is that classic notebook will not be talking to the primary jupyter-server, but to a different server process that doesn't receive guarantees of maintenance. Does that sound right?

[minrk]

By invoking jupyter-server-proxy, I think your scenario B+C may work today with the right configuration - you need to tell the subprocess notebook server about its new prefix, but notebook already works fine on any URL prefix. If this is the right way to go, it's extensible to any combination of versions. Installation might be the hardest part, since the 'classic' server will eventually need to be in its own env to avoid conflicts on jupyter-client/other dependency versions. Might need to be careful about getting auth right (jupyter-server will enforce the hub auth, notebook should only require requests from jupyter-server rather than redoing hub auth or its own token auth). But this should mostly be a matter of getting the right config/env. I'm sure we can make that smoother after we show what is needed to make it work with what we have now.

[yuvipanda]

Me and @ian-r-rose actually did pretty much exactly this in jupyterlab/jupyterlab#5952, for the jupyterlab dev mode setup on binder. The latest released version of jupyterlab + jupyter-server starts up, but then we use jupyter-server-proxy to run another jupyter-server instance from the current master of jupyterlab in dev mode, so new changes can be tested out. It works fairly well, with the primary problem being one of confusion around serverextensions and nbextensions - since they share config, the serverextensions are loaded by both I think. But mostly alright.

[yuvipanda]

We discussed this during the JupyterHub team call today. I think the primary goal here is to make sure that JupyterHub admins are prepared for the notebook v7 release, and know what their options are. We want to avoid frustration on their part, as well as preempt any negativity that might get thrown towards the notebook team. @choldgraf just opened jupyterhub/team-compass#498 to track and discuss what would be needed from a JupyterHub perspective.

[Zsailer]

Here's a proposal for discussion to achieve a balance of encouraging forward progress while giving people a less-stark transition opportunity.

We can create a new, separate classic-notebook-ui pypi package that provides the Notebook JS. This is where "maintenance" will happen for Notebook JS.

Here is a list of practical steps and outcomes:

• The classic-notebook-ui (PyPI) package could be installed in environments with either notebook v6 or v7.
• NBClassic and Notebook v6.5.x will both depend on this package for the Javascript source.
  • NBClassic would need to prefix its endpoints with e.g. /nbclassic to avoid endpoint collision.
• The classic notebook UI could be served above the classic Notebook API server or the new Jupyter Server (via nbclassic) in environments with either notebook v6 or v7 installed.
• The classic Notebook UI could be served side-by-side with Notebook.v7, via nbclassic.

It has the added advantage that we can transfer issues targeted at the classic Notebook UI from of the notebook repo to the classic-notebook-ui.

To help visualize all the moving parts/packages necessary, here is my attempt at drawing a new technical diagram with all the moving parts.

graph TD
    subgraph Classic Notebook UIs
    nbclassic
    notebook.v6
    end
    subgraph JupyterLab-based Notebook UIs
    notebook.v7
    jupyterlab
    end
    subgraph REST API servers
    jupyter_server
    notebook-api-server
    end
    notebook.v7 --> jupyter_server;
    jupyterlab --> jupyter_server;
    nbclassic --> jupyter_server;
    nbclassic --> classic-notebook-ui;
    notebook.v6 --> classic-notebook-ui;
    notebook.v6 --> notebook-api-server;

(For the sake of clarity, I didn't include notebook_shim here, which a package that shims server-side config/traits.)

In the diagram above, we run the risk that classic-Notebook UI falls behind Jupyter Server development (mentioned by @fcollonval), but that's on the shoulders for classic-notebook-ui maintainers. Jupyter Server doesn't "feel" the responsibility of ensuring the classic Notebook UI works with it.

[Zsailer]

The downside of this approach is that we'd likely need to create a bunch of mostly empty python submodules in the notebook (v7) package to shim the old Python APIs to jupyter-server.

For example, we'd have to do something like:

# notebook/services/contents/manager 
from jupyter_server.services.contents.manager import *

for every module in notebook. This would become really painful over time...

[minrk]

This would become really painful over time...

Back in the old days of The Big Split, we used import shims (ipython/ipython#8186) to help manage the maintenance of shims as implementations moved between packages.

[fcollonval]

Thanks a lot @Zsailer for the detailed explanation and diagram and @yuvipanda for highlighting a similar setup for jlab dev on Binder.

I'm 👍 on Zach proposal.

[choldgraf]

Just want to say thanks everybody for your brainstorming and thoughtful considerations here.

I want to re-iterate one other point, that seems relevant to comments like "this will be painful over time to maintain". I think we are all in agreement here that "simultaneously use notebook v6 and v7 on the same environment" is not the long-term goal. I see this as a mechanism for speeding up the transition of many communities to v7 by reducing the perceived risk of using v7 for these communities, even though we temporarily give ourselves more maintenance burden.

In my opinion, we should consider putting a clear message in the "shim repository / project" that it is not a long-term solution, we make no long-term promises about maintaining it, and consider putting a date where we'll actively stop maintaining the project in the future (with the implication that communities should have migrated to notebook v7 by then).

[damianavila]

I like @Zsailer proposal as well! Even when the latest @yuvipanda proposal (the Binder one) is super interesting, I feel it is too complex for the average user/admin and I foresee them getting pretty confused about the underlying model.

[echarles]

This has just been discussed at today classic notebook meeting, and the outcome is summarised on https://hackmd.io/gl6x_SL0S_aYVn0Y0lSBpQ (see section Outcome of the discussion is a consensus on...).

[jtpio]

@yuvipanda just checking if you had the opportunity to try the latest versions of JupyterLab 4, NBClassic 1 and Notebook 7 and see if they address the concerns raised in this issue? Thanks!

[jtpio]

Looks like this issue can be closed now that Notebook 7 final is released: https://github.com/jupyter/notebook/releases/tag/v7.0.0

There is also a migration guide in the documentation for running the classic notebook UI alongside Notebook 7 via nbclassic: https://jupyter-notebook.readthedocs.io/en/stable/migrating/multiple-interfaces.html

Please open new issues here on the team compass repo or on the notebook repo if there are remaining items to be addressed. Thanks!