Environments Road Map
- 5/25 12pm PST: extensive rewrite which is more focused on use cases
- 5/25 2pm PST: "environment" in command syntax is replaced with "context". A context can be used to modify a user's environment (e.g. PATH) but also can be used to group a set of packages together
Spack makes it easy to install anything you want, and you can install many different versions and configurations of a package simultaneously. We want to make it easier to use these packages. That includes running them and developing with them (linking to them).
An environment is a context describing a consistent working set of packages.
- A context is a set of packages.
- A context can be used for:
- Running packages from the command line, as a user.
- Developing packages, potentially outside of Spack.
- Contexts can be shared and rebuilt (reproduced) by different Spack instances.
- Spack guarantees that packages in a context will be consistent.
- Consistency depends on how the environment is to be used (run or develop)
Spack will provide commands to create, share, modify, swap, and rebuild environments.
A user creates an environment and installs some packages. The user can:
- export a description of the packages
- give it to another user, who can then rebuild it.
The user may be working at a different site. Their system may not have the same compiler and may not be the same OS (this might be trickier so I'm curious who agrees with it). They may want to build with the same instance of Spack (the same core code and the same package repository).
A system administrator supports two teams that build their programs against different MPI/BLAS/LAPACK stacks with different compilers.
- The system administrator should be able to name the stacks and specify which versions/configurations of MPI/BLAS/etc. are in the stacks.
- The users should be able to build their software against the stacks either:
- by loading modules, or
- by pointing their build system at per-stack directories
Given two stack descriptions, C1 and C2:
spack install --context=C1 mpi%gcc@5.1
spack install --context=C1 netlib-lapack%gcc@5.1
spack install --context=C2 mpi%gcc@6.3
spack install --context=C2 atlas%gcc@6.3
A sysadmin uses Spack to provide an environment with MPI, BLAS, and LAPACK. The environment uses Open MPI 1.10.1 and they want to upgrade it to the latest version. They are OK with rebuilding the environment to support this but they want all other packages to use the same versions (and variants) as before.
A user has created Spack package descriptions for multiple packages they are developing. They want to use Spack to create an environment where they can develop one or more of these packages. If the packages are dynamically linked, they may want the option to rebuild a package and have the dependents make use of it without rebuilding them. This environment is self-contained in the sense that they are not going to build additional packages outside of the original set. The user may want to export spack settings to the format of another build system that is supported by an IDE.
Both this and (U1) describe cases where users are building with packages created by Spack. This describes a case where Spack can help out more if the user is willing to describe all the packages they are developing with spack.
(currently spack setup and spackdev are anticipated to fulfill this requirement)
Do people want this? Note this is different than an environment with python3 where some packages are built with python2.
The sysadmin provides a set of binaries for a user to run. For any two binaries provided, they must use the same run dependencies: if you have binaries X, Y, and Z, and X and Y run Z, then they must run the same version of Z. This may seem trivial from a user perspective but it places requirements on the concretizer.
A sysadmin provides a set of libraries for the user to build against. They do not have to use Spack. In this case, for any two libraries in view/lib, they must use the same dependency libraries: if you have libraries X, Y, and Z, and X->Z and Y->Z, then both X and Y must link to the same Z.
The self-contained environment described in (U3) is not bound by this constraint, but (U1) is
A sysadmin may want to provide some packages that only need to be run (e.g. emacs) along with a set of libraries. The set of run-only and build packages have the constraints described above.
A user can create a context and associate a set of specs with it; these contexts can be used to maintain different sets of (potentially conflicting) packages for different users. A context tracks the following information:
- For each package the user installed (and associated with the context), what did they explicitly ask for: this is the user spec, which omits all of the details that are not specified by the user
- For each package that was installed, what exactly was installed: this includes details like the exact compiler used, and all details that Spack "fills in"
- As an implementation detail, it may also need to maintain an intermediate representation between these two
The command syntax for this may look like:
spack context create C1
This creates a directory spack/var/contexts/C1
spack context add C1 mpi
spack context add C1 boost
spack context concretize C1
spack context install C1 #install all the concretized specs
(U1) suggests a shorthand for adding a spec to a context and installing it:
spack install --context=C1 mpi
When a user wants to upgrade a specific package they produce a new context:
spack context replace C1 C2 foo@...
This would create a new context C2 which has all the same specs, except that foo differs between them.
Add functionality to allow:
- Loading the modules for all packages in a context
- Creating a view (a merged prefix with bin/ and lib/)
spack context module-load C1
Track the version (git commit id) of spack used to install each package
If the user enables it, modifying the context can automatically update a view - as the user installs packages to the context, they automatically become available in the view.
spack install --context=C1 python@3.5.1+foo
spack install --context=C1 py-flake # this uses the python install from before
Installing a new package X->Z' to a context when you have Y->Z already, but X is just a command-line utility:
spack install --context=C1 X
>> Cannot install X, it requires Z' and Z is already installed in C1
spack install --context=C1 --run-only X
An interim solution (while concretizer updates are pending) may simply allow users to be explicit about this:
spack install --context=C1 X ^[0]python # install X and use the python installed with the first spec added to the context