Drop managing source extensions with jupyter labextension?

[jtpio]

Problem

Having two ways to refer to extensions can be confusing to users and extension authors.

It also requires writing documentation (and keeping it up-to-date) about the differences:

• Managing Extension: https://jupyterlab.readthedocs.io/en/stable/user/extensions.html#managing-extensions-with-jupyter-labextension
• https://jupyterlab.readthedocs.io/en/stable/extension/extension_dev.html#source-extensions
• https://jupyterlab.readthedocs.io/en/stable/extension/extension_dev.html#prebuilt-extensions

It also holds complexity within the JupyterLab code base.

Proposed Solution

We drop support for managing source extensions with jupyter labextension:

Pros:

• Remove the extension manager, or update it to make it work with prebuilt extensions only -> less confusing to the end user who might not know the difference between the two types
• Remove quite a bit of code from the lab code base (install, update, uninstall commands) -> better maintenance
• Easier to reason about how extensions work in JupyterLab
• Focus efforts on a single way to provide extensions to JupyterLab. We can then build more focused tools for extension authors wanting to quickly hack around (a la classic notebook, also related Getting Started Docs for Non-Developers #11075), without worrying about breaking the second way of installing extensions with jupyter labextension install
• Maybe a good opportunity to streamline the bootstrap.js / index.js boilerplate and make it easier to reuse in other lab-based apps like retrolab, jupyterlite, voila, quetz-frontend
• Most of the other JupyterLab distributions reusing JupyterLab extensions do not support managing source extensions with jupyter labextension install / update /uninstall. They only load prebuilt extensions at startup:
  • JupyterLab Desktop
  • RetroLab
  • JupyterLite
  • Voila (soon to be a lab-based app)

Cons:

• in some cases it can be useful to be able bundle everything together. Dropping support for source extensions would also drop support for this

Additional context

This came up during casual discussions. Also something we have been thinking about over in JupyterLite. Trying to imagine what it would look like if everything was a federated extension, even the core plugins.

Also many third-party extensions have now been updated to be prebuilt extensions since it's a much better experience for the end user. There should now be a lot less users installing extensions with jupyter labextension install.

---

Opening this issue now to check whether this is something we would like to consider for 4.0.

It was also something briefly considered for 3.0, but since the prebuilt extension system was still new at the time it was better to keep both system for the 3.0 release.

[jasongrout]

Opening this issue now to check whether this is something we would like to consider for 4.0.

I would like to keep source extensions at least for the 4.0 cycle. It seems some places upgrade every other version (like some skipped version 2 and moved from 1 to 3), so having it in one more cycle I think would be safe, if it's not a ton of trouble.

I would also like to hear from institutional deployments that may want to compile a base system distribution by installing source extensions to see how it is going with them moving to prebuilt extensions.

Regardless, I think we should still support the example app, i.e., compiling a custom remix of JupyterLab that relies on jupyterlab_server, even if we remove the framework in JupyterLab for users to install and recompile the base distribution.

[krassowski]

1. What are the size trade-offs? I know there is some de-duplication but my (possible incorrect) impression was that the federated/prebuilt extensions are slightly heavier or have some other impact on loading time. I never compared them so I'm not sure.
2. If we decide to do this, can we first have a release with a deprecation warning in the command line telling users who invoke jupyter labextension install that this is deprecated and they should migrate?

[jtpio]

Regardless, I think we should still support the example app, i.e., compiling a custom remix of JupyterLab that relies on jupyterlab_server, even if we remove the framework in JupyterLab for users to install and recompile the base distribution.

Agree. Normally this should still be supported if the source of the extension is published like we recommend to do in https://jupyterlab.readthedocs.io/en/stable/extension/extension_migration.html#publishing-the-extension-to-pypi-and-conda-forge.

can we first have a release with a deprecation warning

This issue is more about dropping the set of install, update, uninstall commands and all the logic behind it. A warning sounds good so it doesn't come too much as a surprise.

[jtpio]

This issue is more about dropping the set of install, update, uninstall commands and all the logic behind it. A warning sounds good so it doesn't come too much as a surprise.

I have updated the title of the issue and the top comment to make that clearer.

[jasongrout]

This issue is more about dropping the set of install, update, uninstall commands and all the logic behind it. A warning sounds good so it doesn't come too much as a surprise.

Does that include all of the logic behind the user being able to rebuild JupyterLab?

I am often rebuilding JupyterLab with the --dev-build option to get non-minified js that I can debug better when I am working on a 3rd party extension.

[jtpio]

I am often rebuilding JupyterLab with the --dev-build option to get non-minified js that I can debug better when I am working on a 3rd party extension.

We should indeed keep the possibility to build JupyterLab, since this is what we recommend in the cookiecutter to generate the non-minified assets: https://github.com/jupyterlab/extension-cookiecutter-ts/tree/3.0/%7B%7Bcookiecutter.python_name%7D%7D#development-install

[agoose77]

I'm a big (+1) on this. Anecdotally, the biggest pain point for extensions since 3.0 (in my experience) is users not understanding the difference between source and pre-built, ultimately leading to hard to diagnose problems.

[jtpio]

Updated the top comment to also mention that dropping this would help make the extension ecosystem more consistent across JupyterLab distributions reusing JupyterLab prebuilt extensions, such as:

• JupyterLab Desktop
• RetroLab
• JupyterLite
• Voila (soon to be a lab-based app)

Another idea would be to reuse the jupyter labextension install / update / uninstall commands to target prebuilt extensions published to npm. The commands would then extract the npm tarball and place the static assets under PREFIX/share/labextensions, similar to what pip does.

Linking to these issues which are also related:

• Improve the user experience and awareness about prebuilt extensions #10554
• Publishing an npm prebuilt extension package that is not a source extension #9807
• Rename package.json in prebuilt output #11284

[jtpio]

Another idea would be to reuse the jupyter labextension install / update / uninstall commands to target prebuilt extensions published to npm. The commands would then extract the npm tarball and place the static assets under PREFIX/share/labextensions, similar to what pip does.

This could actually be implemented independently of JupyterLab and live in a separate repo.

[krassowski]

Another idea would be to reuse the jupyter labextension install / update / uninstall commands to target prebuilt extensions published to npm. The commands would then extract the npm tarball and place the static assets under PREFIX/share/labextensions, similar to what pip does.

This is a very interesting solution. This way we could keep the extension manager without extensive changes; it would require authors to publish extensions to npm, but not many changes other than that I guess?

[jtpio]

it would require authors to publish extensions to npm, but not many changes other than that I guess?

Probably this would need to be a separate package than the source distribution? Or packaged with the source, but with the prebuilt assets in a static/ or dist/ subfolder (will need a fix for #11284 first)

[agoose77]

Another idea would be to reuse the jupyter labextension install / update / uninstall commands to target prebuilt extensions published to npm. The commands would then extract the npm tarball and place the static assets under PREFIX/share/labextensions, similar to what pip does.

I wonder whether we want to keep this interface around at all? One of the benefits of the prebuilt extensions is that the server-extension and the frontend-extension can be bundled in the same Python package. This means that users don't run into problems when they invariably only install one of the two packages. Even as an experienced user, I occasionally made this mistake.

If we persist a labextension install that uses npm, it will suffer when

• developers don't publish to npm (which is not unreasonable, even though it makes it harder for others to extend their code)
• extensions need server-extensions too.

[bollwyvl]

to target prebuilt extensions published to npm

implemented independently of JupyterLab

Sure! Any archive-driven ecosystem that has a well-known organization (or a canonical place to put schema-validateable metadata) could work.

For reference, on lite (at build time) we support harvesting federated_extensions from pypi .whl and conda .tar.bz2 files... but can't support .tar.gz or .conda, as it takes extra layers of analysis and dependencies (and potentially arbitrary code execution) to find/create them.

In an npm package, I'd wager we just add a package.json#/jupyterlab/distributionDir to the schema and bob's your uncle. Or abuse/reuse package.json#browser, though it wouldn't be strictly accurate.

npm also provides nicer metadata search mechanism... though we went through the trouble of getting the trove classifers, there is no /browse/ API endpoint.

But what this wouldn't help us with is the dependency resolution problem, which real package managers give us "for free" (ha). I can't underline enough how bad life will get if we have to implement our own: for reference, see the UX/provenance nightmare of last two decades that was "get eclipse-for-XYZ running". VSCode gets away with this by being a walled garden, but it has taken a significant effort to get a reasonable open source thing working.

developers don't publish to npm

And that.

and live in a separate repo.

all of the logic behind the user being able to rebuild JupyterLab?

More broadly, I would like to see all nodejs/npm-specific user-time concerns of lab core moved into a separate package (though probably not a separate repo).

• move all of the source related tools (backend and frontend) from jupyterlab and jupyterlab_server into a separate package, e.g. jupyterlab_source
  • anything that calls node
  • anything that calls jlpm
  • jlpm itself
  • the existing, node-and-npm.org extension manager service
    • to be replaced in the future by something not implicitly bound to a single ecosystem/provider
  • @jupyterlab/extension-manager UI

In the 4.0 timeframe, either jupyterlab or jupyterlab_server could depend on this new package, providing shims to all of the APIs, endpoints, etc. In 5.0, the dependency could be reversed, such that most users didn't have jupyterlab_source, and the shims would be replaced with gentle suggestions to install jupyterlab_source.

Or, refactored differently, jupyterlab could become a metapackage that depended on jupyterlab_core and jupyterlab_source and jupyterlab_server.

Selfishly, from a downstream perspective, this would allow, to put a real dependency on nodejs for jupyterlab_source, but get by with much less complexity in the "normal" case.

if everything was a federated extension, even the core plugins.

And of course, I support this!

[rfox12]

Questions / concerns.... mostly from an extension developer perspective. I'm sorry if some of this is off topic... I'm happy to move or open another issue if appropriate.

Are source extensions deprecated? Are developers being encouraged to move over?

"Prebuilt" extensions rely on Webpack's federated module capabilities right? Has the Jupyter dev team already signaled a long-term dependency on Webpack? I've seen notes saying not to rely on it.

Has anyone figured out hot-module reloading for federated Webpack modules? Or more generally... what strategy gives developers the fastest cycle from change to viewing that change? Along these lines Vite is getting a lot of attention on pre-building assets.

I use Vue extensively in my extension... the webpackConfig option gives me the capability to add vue-loader which makes Vue work for me in JupyterLab. I believe my path towards "prebuilt" would leave me with including Vue as UMD in a script tag... and probably more UMD module dependencies in general... so less tree shaking...

I also worry about performance of moving everything to pre-built...

[jtpio]

Thanks @rfox12 for chiming in 👍

This is exactly the kind of feedback and discussion we would like to have with the community to know if or when such change should be made.

Are source extensions deprecated? Are developers being encouraged to move over?

There has been a lot of good feedback about the prebuilt extension system, especially from the end users since this considerably simplifies how to install JupyterLab extensions, and does not require having Node.js installed anymore on the end user machine. So in general we recommend distributing a prebuilt extension for this particular reason, but still recommend extension authors to publish the source to npm so it can be installed with jupyter labextension install or included in a custom lab build by default.

"Prebuilt" extensions rely on Webpack's federated module capabilities right? Has the Jupyter dev team already signaled a long-term dependency on Webpack? I've seen notes saying not to rely on it.

That's right. Also we try to not expose too much of the underlying tooling such as webpack. The webpackConfig is one of these hooks that "leaks" to the outside, but the documentation clearly mentions it's an experimental feature and that it might change in the future: https://jupyterlab.readthedocs.io/en/stable/extension/extension_dev.html?#custom-webpack-config

Regardless of whether JupyterLab will continue using webpack in the future, one thing for sure is that it will be really difficult to go back to an extension system that requires Node.js on the end user machine. So whatever comes next (if any) should also handle a similar federation mechanism.

I also worry about performance of moving everything to pre-built...

Good point. Probably this could already be measured so we have a clearer idea of the impact. Especially since many extensions are now distributed as prebuilt extensions.

[jtpio]

Quick summary from the weekly call today:

• Source extensions should work in JLab 4. Some source extensions (like Elyra) have not had the bandwidth to move to prebuilt yet. Other institutions may have skipped version 3, upgrading straight from 2 to 4, and thus missed the window to update to prebuilt extensions.
• Should the scope be limited to the user facing commands? (install / uninstall / update) So it's still possible to use the build machinery for advanced users or system administrators
• Perhaps the part of jlab that is doing a monolithic build can be moved into a separate package that has a proper nodejs dependency.

cc @vidartf @ajbozarth since you had some input during the call on how this is used at big institutions. It would be great if you could share more about these use cases, thanks!

[vidartf]

Some benefits to source extensions:

• If a security advisory goes out against a package in the dependency tree of one or more extensions (including core ones), a user/admin can upgrade that dependency in their build without having to wait for extension authors to release new versions of their prebuilt bundles. CVEs do not track JS dependencies in the bundles released on PyPI.
• For deploys with a large amount of extensions, the total amount of data that needs to be transferred is presumably larger with prebuilt extensions than with an optimized source build. For large deploys with a lot of users, this might translate to significant differences. It would be good to set up some best/common/worst case scenarios, and enumerate the differences, to get a handle on the severity of this issue.

[jtpio]

Opened #12386 to start deprecating managing source extensions with jupyter labextension.

This will be a first step that can go in 4.0. So there is plenty of time for folks until 5.0 where we can consider dropping it.

[jtpio]

#12386 was merged and a warning will be shown when using the jupyter labextension install command in JupyterLab 4.

Moving this issue to the 5.0 milestone.

[ekungurov]

Will the ability to enable/disable extensions from cli be preserved (currently achievable via jupyter labextension disable and jupyter labextension enable)?

[bollwyvl]

ability to enable/disable extensions from cli

That just updates a JSON file, and doesn't invoke nodejs.