Conda environment for specific dependencies

[titu1994]

🚀 Feature

While the current container tests are run against bare python, there are many important libraries that require conda to be available to properly install dependencies. Therefore conda support would be the request.

Motivation

A use case for this in NeMo is Numba - we utilize it for custom CUDA kernels for RNNT loss, custom spec augment kernels and in the future for jit compiled CPU code as well. We have plenty of unit tests that are optional (and hence skipped) when these are not installed.

Another is Pynini (for WFST based text normalization and inverse text normalization) - however since the graph building takes significant time if not cached, we do not recommend performing this CPU test during your ecosystem tests. It is just an example.

The reason we prefer Numba to be installed from conda (whereas it does have a pip install path) is that often an increment in numba version is accompanied by an increment in the llvmlite library too - and this causes significant issues. llvmlite is a library that will often throw errors during upgrade, and it can be forced via --force-reinstall (for pip) but that brings about an unstable environment. Conda bypasses all of this.

Pitch

Install miniconda on the fly (or use a base container that supports conda).

Alternatives

Leave conda tests out, and request ecosystem members to make their tests skippable if dependencies are not installed. NeMo can do this (and currently does skip numba tests if its not installed or doesn't support a recent cuda version), it may be cumbersome on a large scale.

[Borda]

I see this as a very feasible request and I hear it already from the inside PL team... 🐰