Brett Viren's ideas

From: Brett Viren

Copy/pasted/approximated from email.

1. I am sure we will find the need to generate configuration files (eg, EM TCL files) or other textual content in various formats based on meta data associated with packages of a concrete Spack DAG.

   To do this I think we should bring back the "spack view text" feature (but with a different cmd name) that briefly lived in the "spack view" PR. I recognize "we should" probably means "I should".

   • Elizabeth: I would imagine the way to go is to write the Spack DAG in a standardized format in the install (already done -- spec.yaml), and then write scripts to read that format and generate config files in whatever other format is desired.

   • Brett agrees.

   This will avoid a proliferation of custom text generator codes while still allowing support for outputting a wide array of configuration file formats by supplying for each the appropriate template. Users could then do their own thing but Spack could also have some batteries include and provide templates for EM and whatever other popular user environment systems we might want to provide for.

   I'd have to dig up the PR but I recall two sticking points for this feature. First, it should not duplicate some existing code that is already in the core of Spack which creates a dictionary of metadata for each package in the concrete DAG. The existing code needs to be factored/reused. The second is that doing templating is easiest by depending on a high quality templating engine (eg, jinja or mako). One can go pretty far with just Python's 'str.format()' but I don't know of a way to easily add support for branching and looping. Adding a new dependency shouldn't be done lightly and I expect people have strong opinions about this.

2. One anti pattern I hope we can avoid is to never make "spack install" depend on anything we are about to add in support of managing user run-time environment. I can go into the why of this but maybe it's not controversial. I say it to draw fire to see if anyone disagrees so we can have that discussion.

   • Elizabeth: Can you please say more about that? I don't understand what you are getting at.

   • Brett: Well, as my Fermilab colleagues probably already guessed, my mind here is on the software ecosystem at FNAL. I don't want to trot out the problems just to poke but they do provide a good case study. Certainly I hope they will correct me if I wrongly characterize anything.

     I guess I can break down the ecosystem into 4 interlocking parts:

     • UPS :: user environment management very similar to EM. Given a binary installation area, it provides an equivalent to "module load ". It also has its own package build feature but that is essentially unused nowadays.

     • cetbuildtools :: binding CMake to find dependencies via UPS. This includes finding both external packages as well as other collaboration packages. It's functions are used in collaboration package's CMakeLists.txt files and it requires some additional auxiliary/invented control files.

     • mrb :: a developer's "helper" shell script which depends on the above and replaces a lot of required tedious commands with a lot of required mystical ones.

     • SSI :: (I think this is the right moniker) is the meta-build system based based on a body of "shim" shell scripts (moral equivalent to Spack's package.py) which build an external package into a UPS binary. It relies on UPS to set up the build environment.

     Goto 1.

     As a consequence, building the ecosystem from "first principles" is essentially impossible. To build our code we must replicate the entire ecosystem and that essentially limits us to building on platforms supported by the build experts at FNAL.

     Hopefully airing this laundry is useful. I don't want it to come off hurtful!

     As long as "spack install" never requires any user environment setup then most of the cycle seen above will not form.

     But, creating one or more developer environment management systems with Spack is an opportunity to fall into some of these kinds of traps.