Custom kernel pod created from docker-stacks images terminates with 'Succeeded' so cannot connect

[citronella3alain]

What works:

• loading a custom kernelspec (/kernels) directory from a Docker image (in a private CR); helm chart --> values.yaml:kernelspecs is correctly configured. Or, Custom Container image for k8s deployment works.
• deploying a custom kernel based on elyra/kernel-py (like this). For example, a kernel based on this Dockerfile:

FROM elyra/kernel-py:3.2.1
USER root
RUN pip install flask gradio
USER $NIB_UID

shows up in the list of kernels, runs without terminating, and executes a user command such as import gradio or import Flask. These libraries were chosen solely on the basis that they aren't in the python_kubernetes kernel so would signify successful library installation.

• loading kernelspec and kernel images from private CR

What doesn't work: deploying a custom python kernel based on docker-stacks.
Here's an example kernel Dockerfile based on here:

FROM jupyter/minimal-notebook:python-3.10
USER root
RUN wget https://github.com/jupyter-server/enterprise_gateway/releases/download/v3.2.0/jupyter_enterprise_gateway_kernel_image_files-3.2.0.tar.gz &&\
        tar -xvf jupyter_enterprise_gateway_kernel_image_files-3.2.0.tar.gz -C /usr/local/bin &&\
        rm -f jupyter_enterprise_gateway_kernel_image_files-3.2.0.tar.gz &&\
        fix-permissions /usr/local/bin

USER $NIB_UID
ENV KERNEL_LANGUAGE python
CMD /usr/local/bin/bootstrap-kernel.sh

I've tried variations such as scipy-notebook, different versions thereof, installing pycryptodome with both root and jovyan, using the fix-permissions script, etc.

What happens: A pod is spawned, but after the image finishes pulling, the pod terminates with status "Succeeded", then gets cleaned up. Instead, we want it to continue running so that Jupyter Server can connect to it.

Other Notes:

• We want to build from a docker-stacks image so that we can control the environment, i.e., Python version (3.8), in order to install a pre-built wheel.
• The kernel image puller has been disabled. However, given that the kernel-py base image works, I suspect this shouldn't be the issue. That said, I've observed on initial deployment for Jupyter Server to time out after image pull took too long. Subsequent pulls were successful after I changed timeout time in jupyter-server-config.py, however.
• ImagePullPolicy has been set to Always for testing purposes in scripts/kernel-pod.yaml.j2 and Helm chart: values.yaml:kernelspecs to make sure images aren't stale.
• I've tried various scipy-notebook and minimal-notebook versions.
• Enterprise Gateway deployed with Helm, image is elyra/enterprise-gateway:3.2.0
• In the meantime, I'll try to rework the image here and I'll see how that turns out.

Thanks! It's been really exciting to learn more about Jupyter Enterprise Gateway and its architecture! I'm rather new to k8s so better practices (how to get logs from these successful terminated pods) in relation to Gateway would be welcome too!

[kevin-bates]

Hi @citronella3alain. It's strange that you're having issues with basing your custom kernel images off docker-stacks images since that's precisely what we've done in EG. What tag are you using on your docker-stacks FROM image? I ask because shortly after 2022-01-24, docker-stacks included health-check logic - which makes perfect sense for "notebook server" images (like those in docker-stacks), but not so much for kernels. As a result, we updated our Dockerfiles to disable the health check and I wonder if you didn't do that AND are using more recent base images if that's what is causing your issues.

FYI, I contributed a new base image to docker-stacks for the sole purpose of deriving kernel images from that. The image is jupyter/docker-stacks-foundation and all other docker-stacks images derive from it. (It does not include a health check.) This way folks can use it to install the packages they need and not have full web servers and other jupyter software on their kernel images. We use this as our base image in gateway-provisioners - which will be what EG uses in its 4.0 release.

Also, you might find it easier to build kernel images using the tooling in gateway provisioners and these should be compatible as EG kernel images.

[citronella3alain]

Hi @kevin-bates,
Thank you for the quick response! It's helped me resolve the issue; custom kernel images built from jupyter/base-notebook and jupyter/docker-stacks-foundation now work!

First, to answer your questions:

• What tag are you using on your docker-stacks FROM image? Since I'm trying to pin to Python 3.8, I tried variations on it: jupyter/{minimal-notebook, scipy-notebook}:python-3.8.13. I also tried with a later version, I think Python 3.10. In either case, this leads to the next point:
• Whether the health check was disabled + are using more recent base images: Not disabled health check + recent images (from 3 months ago) => doesn't work. Disabled health check + misc observation 1 (below) + recent image => works.

Misc Observations:

1. Some subset of the packages in RUN mamba install --quiet --yes -c conda-forge cffi future pycryptodomex && ... (from here) seems to be required since the spawned kernel terminates if this line is omitted. This page seems to imply otherwise (it also uses a deprecated library, pycrypto).
2. docker-stacks-foundation images seem to only inherit from python 3.10. I've thus chosen minimal-notebook as the base image since there's a python 3.8 option. I do see that this template works regardless of python version. Is it possible / encouraged to push such images to docker-stacks?
3. Thanks for the link to gateway-provisioners!

Let me know if I should test / keep track of something else for documentation purposes. Thanks again!

[kevin-bates]

Hi @citronella3alain - glad to hear you're moving forward.

Some subset of the packages in RUN mamba install --quiet --yes -c conda-forge cffi future pycryptodomex && ... (from here) seems to be required since the spawned kernel terminates if this line is omitted. This page seems to imply otherwise (it also uses a deprecated library, pycrypto).

Yes, pycryptodomex is required to perform the encryption operations on the connection info. I agree that the documentation is out of date and should be updated. Seems like we could "advertise" the docker-stacks-foundation image here as well - although that might have a trickle effect in other areas of the docs.