break build systems and dependency type
s down into more granular resource dependencies
#20523
cosmicexplorer
started this conversation in
Ideas
Replies: 1 comment
-
Appears similar in intent to #18054. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Probably blocking on #20530 to resolve this, but the type of solution we get to #20530 might come from the discussion in this issue.
This is a similar attempt to #10411, but with a broader scope. We attempt to explicitly represent interaction with build systems by introducing the "resource dependencies" concept instead of defining a dependency by build step names alone.
Unlike #20256, any extensions of the spec syntax are out of scope for this issue. Once this is done, we would continue to allow build, link, run, and test dependencies, but internally they would each be implemented as an umbrella virtual dependency, covering each underlying resource dependency, for no change in functionality.
Rationale
We have long desired to make a package work in more than 1 type of Spack "build system" at a time, e.g. if a package switches from using autotools to cmake. See Multi Build System Packages [Prototype; Please Review] #12941 for context.
Separately from build systems, it also seems spack may be conflating several concepts into the dependency
type
s that can be declared in package.py files. An excerpt like:from this comment #3768 (comment) (quoted immediately above) seems to point out several very specific dependency relationships which don't seem to fit into the current model, (where a build "type" determines multiple very specific things, despite having the same name as a build "step").
There was a discussion just now with @tgamblin and @scheibelp where it was agreed that the definition of a spack "test" dependency seemed to be the least obvious of this set of words that mean two very different things:
spack python
exists?Even with just two dependencies at once like at the bottom of that comment, it seems to become very difficult to predict the result when Spack evaluates the spec. Furthermore, it seems to me that having to go back and exhaustively post-hoc categorize all of the intended changes as is done in that comment is far more effort than writing this code with an API that explicitly accesses and provides these very specific resources.
That precise argument also seems to apply to the current situation of monolithic, non-composable build system abstractions. I believe there may be a solution to both of the perceived issues with monolithic build systems and monolithic dependency types at once.
Description
This is a general approach I'm thinking might address this:
Require Explicit Dependencies on
.libs
and.headers
, and Validate ThemThis is a paradigm that pants, another build tool, has been moving to over the past few years for defining packages ("targets" in pants parlance). The site has a great blurb on what this means (at https://www.pantsbuild.org/docs/target-api-concepts#a-field-driven-api):
I ranted and raved about this to @becker33 a while ago: pants implements this via this very basic "UnionRule" concept I made up: pantsbuild/pants#7116. I don't think any of the pants rule DSL is relevant to this, but I can explain the mechanics of it further if necessary.
And the part I think is particularly relevant here is that
Field
has its own entire class hierarchy which decouples validating outputs from consuming inputs. As described in adding custom validation, it's mainly expected that theField
will take care of validating itself once you add it. The pants approach is particularly nice because both the provider and the consumer need to explicitly turn it on, and because it attaches a validation process that is always run to check that the output artifacts (libs, headers) actually work. Spack essentially has all of this, but the validation part isn't shared across packages at all, possibly attracting mild bitrot.Separately, this paradigm of explicit resource dependency on a particular package and implicit resource validation may even be a first step to Spack inferring which resources are available via binary analysis (CMake does a pale but useful imitation of binary analysis here).
Proposal
The concrete proposal here is:
Separate all the fine print in our dependency types into "resources" which can be depended upon individually.
provides(...)
.provides()
would be updated to accept a python class as well as a string.libs()
andheaders()
methods in package.py files -- the only difference we propose is that each of those hook methods be explicitly assigned to a specific purpose-built class definition, instead of requiring it for every package with inheritance.(Try to) represent all spack's known deptypes with these resource dependencies, as per Document the meaning of deptypes #3768 (comment).
build
,link
,run
, andtest
dependencies, but internally they would each be implemented as an umbrella virtual dependency covering each underlying resource dependency for no change in functionality.(Try to) convert all spack build systems into collections of resource dependencies that package.py files can choose thoughtfully between.
Additional information
As described above, pants has used the indirection of
@dataclass
objects and highly granular dependencies to avoid surprises in its build logic. This approach is definitely compatible with mypy, for example.I would appreciate any thoughts on how to make this something spack would actually merge!
Beta Was this translation helpful? Give feedback.
All reactions