Telcon: 2019 05 16
- Peter Scheibel (host)
-
1:00-1:30 PDT - any issues that people want to bring up
- Python3 Migration Strategy: Python2 is retiring January 1, 2020! We need to agree on a traisition plann SOON, so we have time to implement and test it properly. See https://github.com/spack/spack/issues/11468
-
1:30-2:00 PDT - HEP working items document (continuation of discussion from https://github.com/spack/spack/wiki/Telcon%3A-2019-05-02 and 4/11)
- (@chissg) Updated FNAL Spack Work Items to SpackWorkItems-0.4.pdf. Please note that this document describes issues that we (Fermilab) have encountered while trying to satisfy the needs of HEP collaborations and users. It is by no means intended to be a comprehensive, prioritized, or sanctioned list of things that must be done to Spack. That said, @citibeth, you raise some interesting points and I would be glad to have you involved in discussion this afternoon or any other time.
Here are my thoughts on the document SpackWorkItems-0.3.pdf:
2.1: In general, I believe that these kinds of "plug-in" systems (external repos, external commands) are a complicated/specific way of doing what Git already does for us in a more simple/general way. I believe we should encourage users (i.e. it should be standard practice) to fork the Spack repo, and to put their custom commands, custom packages, etc. directly into their fork. If your extensions are in a separate file (as they would be for additional packages, additional build systems, etc), then there will never be any conflicts when pulling from the latest Spack version. I don't think we've seriously considered a cost/benefit analysis of just using Git, vs. making everything pluggable. If we did, I wouldn't be surprised if we concluded that pluggable stuff is too much effort for too little gain.
3.1: Can also be addressed TODAY by using git effectively.
2.2: The history under 2.2/Status is woefully incomplete. See #2177 and #642 for the earliest (known) mentions of this idea. We should also consider #3156 (Spack Build Server). Maybe its time has come?
2.3: Consider the alternate possibility; which will probably work for some users but not all: that we encourage people to rebuild their entire software stack semi-frequently. This only sounds onerous in a pre-spack world where building software is a difficult, manual task. Spack automates that. So we should encourage users to let Spack do what it does best; that is, build software. In a robust multi-user environment (see #3156), central admins will play less of a role in installing software themselves (since users can always send appropriate specs to the server); and more of a role of helping users set up their Spack Environments. Thus software building becomes more decentralized and self-service. In the end, I feel that too many of us are simply using Spack as an automated way to do what people used to do manually; rather than think deeply about how life changes in a Spack world, and how to adapt to that. For example... pre-emptively building 24 versions of a "core" package for different compiler/MPI/etc. combinations just in case some user wants that combination makes less sense in a Spack world.
2.4: I believe this will come naturally when we stop having compilers be something special, and just have them be another build/link/run/etc. dependency. Therefore, I think we should work on turning comilers into a dependnecy, rather than specifically supporting compiler-agnostic packages.
2.5: We've had PRs for this in the past. But the stumbling block (for me) is the idea that such packages should be creatable on-the-fly in a lightweight manner; in other words, the "concretize together" feature. In the end, I think that will be more flexible and useful than heavyweight no-source packages.
3.4.1: The core problem here is, how do we put a total ordering on a small and finite set of irregular version strings? We've spilled far too much ink on complicated solutions to a simple problem. Whatever else we do, I'd like to see a simple solution to this problem: i.e. have the user tell Spack the ordering explicitly, one way or another, at least in some cases. It should be a simple and powerful fallback if nothing else works. See discussion at end of #8421.
3.4.2: This becomes more important if a simple user-defined ordering non-numeric versions is implemented in 3.4.1.
3.4.3: This would be solved with 3.4.1
3.5: See #391, #2594. Consider the pros and cons of bottom-up vs. top-down variant forwarding. (Top-down is what's described in 3.5. Bottom-up means that if A->B, then A configurs its features based on variants found on B). Currently, some packages in Spack use bottom-up and some use top-down. It would be good if we could come to consensus on best practice here. (See also 4.4 and evaluate in terms of bottom-up vs top-down forwarding).
4.1 The existing mariadb package is GPL and therefore not usable by HEP commu- nity in a linking context.
I'm not sure if that is true. GPL restrictions only kick in if you redistribute software. If you just keep it in-house (i.e. on your local HPC), then I believe you can do anything you want. Of course, talk to lawyer...
=========================================
Items not currently on the list:
\subsection{Spack Setup}
Allow for a way to configure source builds within a Spack environment.
\paragraph{Status}
PR #10403 (and previous versions) was submitted many years ago, it is currently moribund. A better/simpler way may also be possible: to provide an easy way to export the build environment of any Spack package. It would be exported, for example, as a bash script. This would allow Spack setup-like functionality to work more generally for all packages, not just CMake-based packages (and no longer having to build a new way for Spack Setup for each build system).
\subsection{Spack Environment Export}
"Render" a Spack environment by creating a single module that loads it; rather than today's options of (a) creating multiple modules and loading them in a shell script or (b) creating a View. A single-module Spack Environment render would make (a) obsolete.
\paragraph{Status}
Discussed a few times, nothing done yet.
\subsection{Spack Environment Garbage Collection}
If a Spack site uses Spack Environments exclusively, then garbage (obsolete installed packages) can be identified and removed. This is helpful.
\paragraph{Status}
A PR was submitted on Issue #8291. It is waiting for Spack Setup to be merged.