Host a JupyterLab metapackage

[fcollonval]

Following the discussion at the JupyterLab dev meeting, @tonyfast proposed to maintain a JupyterLab metapackage in this organization.

This is a great idea, now the question is which extensions should we include?

To make thing as simple as possible, I propose that people add an unique comment with the extensions they would like to see. Using the first comment as example.

---

As a first guess, extension appearing in 50% of the comments will get included.

[fcollonval]

Extensions wish list:

• jupyter-archive
• jupyterlab-favorites
• jupyterlab-recents
• jlab-enhanced-launcher
• jlab-enhanced-cell-toolbar
• jupyterlab-lsp
• jupyterlab-execute-time

[jasongrout]

Cool! Perhaps this metapackage does not depend on core jlab, but only the extensions, as sort of an extension pack that you add on top of the core jlab package? That will help it to not be confused with core jlab or official Project Jupyter status.

[tonyfast]

perhaps a nice way to do this would to include all the package names in a setup.cfg and comment out the ones that don't ship. here's a skeleton example with most, if not all, of the jupyterlab contrib org.

[metadata]
name = jupyterlab-batteries

[options]
install_requires =
    jupyterlab_search_replace
    jupyterlab-unfold
    jupyterlab-spreadsheet-editor
    jupyterlab-fileopen
    jupyter-archive
    jupyterlab-spellchecker
    # jupyter-videochat
    # jupyterlab-kernelspy
    jupyterlab-tour
    jupyterlab-plugin-graph
    jupyterlab_vim
    jupyterlab-cell-flash
    # jupyterlab-link-share
    jupyterlab-favorites
    jupyterlab-recents
    jupyterlab-topbar
    # jupyterlab-logout
    jlab-enhanced-launcher
    jlab-enhanced-cell-toolbar

another option would be to have categories of extras_requires for the installations not distributed with core.

[options.extras_require]
binder =
  jupyter-videochat
developer=
  jupyterlab-kernelspy

[fcollonval]

Cool! Perhaps this metapackage does not depend on core jlab, but only the extensions, as sort of an extension pack that you add on top of the core jlab package?

I'm not sure this is gonna work because the extension are likely to pin a major version of JupyterLab

That will help it to not be confused with core jlab or official Project Jupyter status.

I agree that the naming should be clear on that point.

[krassowski]

-1 on jupyterlab-vim; I mean I love the extension, but no one will be able to escape notebooks!

[krassowski]

On more serious note: I really think that jupyterlab-git could be included in JupyterLab Desktop because we have our own conda env and can ensure that a specific version of git is installed; I am not sure if we can ensure this with a metapackage.

Some extensions are easier are more universal than others; a very simple set of ones that I expect to always work would be:

• jupyter-archive
• jupyterlab-tour
• jupyterlab-spellchecker
• jupyterlab-recents
• jupyterlab-favorites
• maybe jupyterlab-spreadsheet-editor

[krassowski]

Also, a metapackage with simple visual tests could be useful for testing new minor versions of JupyterLab for breaking changes.

[bollwyvl]

extension appearing in 50% of the comments will get included.

Good interest here, and i certainly like the concept of thinking of a curation process that involves data driven techniques.

Also, a metapackage with simple visual tests could be useful for testing new minor versions of JupyterLab for breaking changes.

☝️ this. i think an ideal baseline to proposing an extension would be at least one, relatively representative user workflow of "does the thing," which generates good screenshots of key milestones, that make their way all the way out to, e.g. a versioned RTD site. In this particular case, I feel like a human-readable BDD/ATDD approach would be the more effective approach.

i think the greater opportunity is not for "the one lab++", but rather for a number of layered, curated configurations to meet specific goals, unified by a common set of (automated) documentation, packaging, and test techniques. This is not entirely unlike...

• https://github.com/jupyter/docker-stacks
• https://spins.fedoraproject.org

...except we're working with a different set of opportunities (e.g. naively, pip is more composable than a single Dockerfile) and constraints (e.g. must work on windows). So in parallel to a list of extensions (which is good), we might consider sets of uses cases which might represent enough of a persona to be a full installer, e.g.

• i want to write a scholarly article
• i want to develop a jupyterlab extension
• i want to make a poster for a poster session
• i want to reverse engineer a data protocol
• ...

Under the hood, i could imagine that would map to some tasks:

• writing technical prose
• running acceptance tests
• spell checking

And then mapping tasks to extensions.

Luckily, I didn't make this crap up, and it already has clever buzzwords of art like AHP and QFD.

And yes, of course at the end of this, things like jupyterlab-git will bubble up to the top.

Putting the above info, as data, into the repo, would let us not only document the effectiveness of a particular extension (e.g. appears in 5 Labs) on an RTD site, but also give said site users the ability to click the tasks they are interested in, and get out a requirements.txt, environment.yml, Dockerfile that is tailored to exactly what they want. Heck, i bet we could wire up a click-to-gist-to-binder test drive.

[bollwyvl]

I am not sure if we can ensure this with a metapackage.

Sure, not a pip metapackage. But a pip upstream, which versions the relevant python parts, and can be backstopped by conda-forge can pretty much answer the mail on that. Someone would then be free to figure out an automation approach for package-manager-of-the-distro-of-the-week, but in the meantime, we'd have something that could drive to full, userspace installers across many platforms.

The underlying tool the desktop project is using (constructor) has been demonstrated to provide any number of heavyweight, tailored installers for different use cases:

• quantum chemistry
• automated testing
• learning reinforcement learning

So if we cooked up the "JupyterLab: Contributor Edition" installer, we could:

• point to it in documentation (isntead of... get python and node and .... somehow)
• use it in workshops
• use it in CI with https://github.com/conda-incubator/setup-miniconda

[fcollonval]

Thanks a lot @bollwyvl for sharing all those ideas and pointers.

Going in that direction does anyone knows about a cross-platform tool that could offer installation customization as usually encountered on Windows; like

[krassowski]

I would love JupyterLab Desktop to have such a choice of options of extensions and language pack see jupyterlab/jupyterlab-desktop#213. Given how lightweight many extensions and language packs are it would be a relatively small size increase (compared to the weight of the electron itself). I am not sure if we want to maintain two installers separately or just handle it in Desktop. The role of contrib team/repo would be to curate a list of packages that should be available to select in the installer (whether included in the installer or fetched during install time).

[bollwyvl]

I am not sure if we want to maintain two installers separately or

As mentioned a few times: even two isn't enough!

just handle it in Desktop

A stopgap for Desktop might be to think about it's installer akin to repo2docker (or even @conda-incubator/setup-miniconda). Basically a contract that it can realize "a working lab environment" from anything that has (or can be made to have) jupyterlab:

• Mini (bundled, ~125 packages)
• Dev (points at a hard-coded CDN URL e.g. github, archive.org, x00 packages, like git)
• conda-pack archive or constructor
• some other installer+package manager, with a description of how to "get stuff"

Or a description of an environment that can have the Mini environment updated, e.g.

• requirements.txt that gets overlaid on top of Mini
• meta.yaml
• conda.lock
• some other cross-platform description of an environment
  • humorously, minikube might actually be a viable option, and docker seldom actually makes anything simpler

However, my general concern with supporting "lightweight" installers: given a "fat" installer on a USB stick, it's going to work and keep working. Given "half an installer," there's every likelihood that service could be disrupted, blocked, cache evicted, slow, etc. when students try to use it next semester, or in the next country over. Flashbacks to try to get eclipse set up, configuring update sites, etc. just to run one thing that only works in a specific version of eclipse.

a cross-platform tool

Despite a lot of pascal written, a constructor or conda-pack with mamba in it is the closest i've come up with: at the end of a constructor run (or probably any such bundler thing), there's one chance to launch a post-install.*, and that can open a "ring zero" native GUI (e.g. tk), a TUI, or a local/tunneled web server which could do some extra installation, configuration, desktop icon building, and finally, launch/manage the application-of-interest.

that could offer installation customization

Welp, I guess Desktop's electron sorta is that, but now splits the deployed lab JS/CSS assets from package-manager installed things, and would need some "appliance" part of if that can help get back to a working state if/when things break. And efforts spent only on electron is... not exactly wasted, but ignores the hub space entirely.

Of particular note, such an appliance, either electron or other, could offer configuration of jupyter_server and serverextensions in ways that are not supported by the existing architecture.

how lightweight many extensions and language packs are

As mentioned above, it's not the lightweight ones we need to worry about: these could already be covered by an elegant solution like cxfreeze/pyinstaller/etc. or... cp. But it takes very little to run aground on "i need all of POSIX on windows" or "any qt whatsoever."

[manics]

Getting a bit off-topic, but would workspaces/layout/UI customisation come under this? One issue brought up in jupyter/notebook#6210 are the different use cases for Jupyter users. JupyterLab vs Retro handles some of those cases, but is there scope for e.g. providing some preset workspaces/layouts for other types of users?

[bollwyvl]

Anything that's "last in wins" or otherwise changes a single file is hard to package in a composable manner. settings could probably grow an "overrides.d" to complement overrides.json. workspaces are harder, perhaps, as I don't even know what a composable workspace would be like.

[miwojc]

• jupyterlab-vim
• jupyterlab git
• jupyterlab-lsp
• collapsible headings
• jupyterlab spellchecker
• jupyterlab code formatter

[bollwyvl]

Here's a gitter thread about citation management (cc @krassowski @baggiponte)... in addition to that, i think it's fairly clear there's a need for a latex/scholarly writing base layer with things like:

• jupyterlab-latex
• jupyterlab-lsp + texlab

This particular stack, done "the hard way" is empirically not only very large, but also difficult for a novice to stand up on e.g. windows, docker, etc... certainly more difficult than git + friends.

[baggiponte]

I would say:

• jupyterlab-lsp is the most important package - I basically would not use jupyter without it.
• collapsible_headings is really handy, especially when I have to deal with long notebooks (AKA always). I believe this should be pulled into main.
• jupyterlab_code_formatter with black is handy at times.
• geojson-extension is great since I have been working a lot with spatial data [ˆ]
• I get that jupyterlab-git is popular and an amazing achievement, but I mostly use git via CLI/terminal so it's not a dealbreaker for me.

The jupyterlab-citation-manager can be more of a niche - not all Python users are academic, but I think that a jupyter-scholar package would be crucial to reduce the gap with R (distill, bookdown, oxforddown...) and Julia, who launched Books.jl. It could come along with jupytext and jupyter-books, and, of course, jupyterlab-spellchecker.

I just discovered a lot of cool stuff in jupyterlab-contrib. Most of them are quite useful and are features I would "expect" in an IDE/default JupyterLab: jupyterlab-archive, jupyterlab-recent, jupyterlab-enhanced-launcher, jupyterlab-fileopen. Some of them I haven't tried yet, sometimes just because there are no screenshots on the webpage and I don't really know what to expect.

As a side note, mamba-gator offers an equivalent of the Anaconda Navigator from inside JupyterLab. It needs some improvements but I believe it's a worthy addition. I tried to set it up: it takes time. But anyway since I mainly create envs from the CLI I barely used it.

Besides, jupyterlab-sql can be really useful as I always have Jupyter AND Postgres (at home) or Hadoop User Interface (at work) open to query. I mean, psycopg is great but I still just use it to copy-paste the optimal query I worked on somewhere else.

[ˆ] extensions like fasta, geojson... and plot viewers for plotly and bokeh should be a package of their own, like jupyter-spatial-data and jupyter-interactive-plots

[jtpio]

Since the first JupyterLab 4 beta is approaching, maybe it could be a good time to start experimenting with such a JupyterLab metapackage.

The metapackage could first include extensions already compatible with JupyterLab 4.