Massimiliano Culpo's developer requirements

From Massimiliano via email

I won't also be able to attend the meeting on Wednesday but I would like anyhow to share a couple of thoughts on what will be discussed there.

The experiment I tried below is to write a few initial 'stories' based on the requirements I read and how I envision developing software using Spack. I hope they will help in the analysis part of the problem and they could be a good starting point for further refinement at least on some issues.

As a software developer using Spack I want the same source folder to be used by many build configurations, so that I don't have to swap source directory to work on different configurations simultaneously

Unlike what happens for normal builds, the source directory should be shared by specs coming from the same package when we develop. The point is to be able to have multiple build configurations in place for the same package at the same time.

Example: I develop hdf5 and am interested in trying to build 'hdf5%gcc+mpi^openmpi', 'hdf5%gcc+mpi^mpich', etc. I want to work on a unique source directory for all these configurations.

As a software developer using Spack I want a local view of my dependencies, so that I won't have to chase where libraries and binaries are located

I think it makes sense to use @brettviren views as an intermediate layer before configuring a build somewhere. Roughly the directory structure I have in mind for development will be something like:

hdf5/  # package I want to develop
├── codename1 # Name of the first configuration
│   ├── build # place where the build is configured
│   └── dependencies # spack view of all the relevant dependencies
│       ├── bin
│       ├── include
│       └── lib
├── codename2 # Name of the second configuration
│   ├── build
│   └── dependencies # Dependencies may be different from above
│       ├── bin
│       ├── include
│       └── lib
├── sources # Sources I want to modify
└── .spack # Among other things `package.py` should be stored here for further modifications during development

As a software developer using Spack I want installations of development software and of "stable" software to be done under different root folders

It's very likely that packages under active development will follow a different workflow from packages that just need to be installed and used. I would make this clear duplicating at least the install and the module file root folders.

As a software developer using Spack I want only VCS fetch strategies to be used for development, so that I won't start developing on an expanded tarball

There has been some discussion in various Github issues on what we should do to track 'moving' versions. One idea could be allow them only in development mode and have them under the control of the developer.

This boils down to:

• add a few attributes like git, svn, etc. to packages that are being developed (so that we know what repository we should use for the package)
• disallow things like branch and maybe tag in the version directives that use VCSFetchStategy of any kind

When developing, only fetches that go through a VCS system should be allowed (and the default branch will be fetched). When installing "stable" packages, only 'non-moving' versions can be used in directives.

As a software developer using Spack I want to be able to depend on both software under development and stable releases, so that I can develop multiple related packages at once

Example: I am developing 'netcdf' and 'hdf5' together. I want the build of 'netcdf' to depend on the version of 'hdf5' under development (and rely on stable versions for everything else). A.k.a. Cyrus requirement # 1

As a software developer using Spack I want to be able to use the software I am developing so that I can try it as if it was deployed

This may be controversial but, while I would still generate module files for packages under development on explicit request, I would like spack to provide a virtual env. I think that a command similar to:

$ spack development activate hdf5%gcc+mpi^openmpi

should create a new shell with the environment set-up to develop on hdf5%gcc+mpi^openmpi. We should keep track that we are already under a development environment, have the binaries we need already in PATH, etc.

The reason not to use module files for this is that their installation is triggered as a post-install hook in the usual work-flow. It is very likely though that during development the installation of a new version of the package will be in control of the developer who will just issue a make install command in the appropriate build folder. Going in and out of a shell is in my opinion a quicker way to update the environment during development than unloading a module file, regenerating it, reloading it.

Module files can still be useful to let other people try / test what we are currently developing.

As a software developer using Spack I want to be able to query every relevant information on the packages being developed

I think it could make sense to have a db with development only informations stored in the root folder for packages under development.