a proposal for how to represent virtual dependencies in the spec syntax

[cosmicexplorer]

#15569 expands the spec syntax really thoughtfully, making use of our new magic powers from the clingo concretizer. These are several ways we could consider expanding on that, along with an example spec grammar.

"DO NOT ERASE"

The current state of this issue lies at comment #20256 (comment)! Everything below this line is for archival purposes!

---

Rationale

1. There's no distinction between real and virtual dependencies in the spec syntax. This can be confusing sometimes.
   • Finer selection of virtual providers #15569 takes a huge step towards solving this, while at the same time also addressing the separate issue of:
2. There's no way to signal in a spec that you intend to use this specific spec to satisfy this specific virtual dependency. That desired use case is missing.
   • Finer selection of virtual providers #15569 solves this -- see https://github.com/spack/spack/pull/15569/files#diff-3170fb66c33cc3b75ca53fd212a3668610eb9bc78dcafb9b88e456500e0c7e70R1208-R1218:

The ^<virtual>=<spec> syntax tells Spack to use <spec> to satisfy the
requested virtual (so ^mpi=intel-parallel-studio tells Spack to use
intel-parallel-studio as the mpi provider) and to consider its other virtual
dependencies after any other explicitly mentioned package. Adding ^openblas
to the spec thus means that openblas will then be used for lapack, and
blas because it is specified explicitly.

In this example, that covers all the virtual dependencies, but if there were other
virtual dependencies, Spack could try to satisfy them with intel-parallel-studio.
For instance, if another package depended on tbb or daal (which intel-parallel-studio
also provides), Spack could use intel-parallel-studio to satisfy them as a last resort.

3. There's no way to represent in a single spec that a dependency needs to be built with a separate compiler.
   • Specifying the compiler in a depends_on directive sets the compiler only on that node #20127 describes how Spack currently approaches this: by only applying the depends_on(...) directive to the current node (not ideal).
   • However, this also touches on a further more general issue:
4. There are (known and intentional) disparities between the Spack CLI spec syntax and the directives accepted in a package.py.
   • For example, while the depends_on(...) directive is able to specify a compiler (just for that node, see (3)), a user can't do that on the command line.
   • However, I think this is actually understandable, because:
5. Spack specs have no concept of scope or nesting -- they're totally flat.
   • This is also a really great feature of the syntax, though, which makes it 100x more helpful on the command line instead of having to match paired delimiters.
   • But this lack of scope or nesting is also directly what makes it difficult and/or impossible to disambiguate e.g. a %gcc@7 intended to be applied on one dependency, but %gcc@8 in another, when provided with the current spec syntax.
     - Finer selection of virtual providers #15569 introduces the first paired delimiter -- ^[virtual=package@version], which exposes this new type of constraint: "please ensure that my mpi implementation is going to come from exactly this package".
6. Specifying version ranges requires hacky workarounds with .999: (imho) to emulate the behavior of < or > with the single : operator as we currently parse it.
   • These workarounds may make users' spack specs harder to read and maintain.
   • pip-style requirements specifiers are able to represent several version relationships we can't -- the lossy conversion is very helpfully detailed in our docs here.
   • However, opening up new types of version relationship symbols is currently hampered by the same lack of nesting as described in (5).
7. It's currently nontrivial (?) to answer a question like "if I change this depends_on('x@y.z') version y.z to something else, what conflicts will arise?"
   • The clingo solver is basically answering that question, but its internal machinations aren't transparent to us yet -- we haven't figured out how best to interpret a .unsat_core(), for example.
   • We have to create big lookup tables to map our spec logic into ASP, see Experimental ASP-based concretizer #19501 and Finer selection of virtual providers #15569:

     spack/lib/spack/spack/provider_index.py

     Lines 340 to 352 in 4622489

     	class IndexWithBindings(abc.Mapping, _IndexBase):
     	def __init__(self, specs, bindings):
     	"""Provider index that ensures some virtual dependencies to be
     	bound to certain providers.
     	Args:
     	specs: specs used to initialize this provider index.
     	bindings (dict): dictionary mapping a virtual dependency name to
     	its preferred provider. All the other virtual dependencies
     	provided by the same package will be given the lowest possible
     	priority.
     	"""
     	self.bindings = bindings

8. Two parts:
   • A provider of a virtual dependency currently makes absolutely zero guarantees about the files it provides to any dependent build/link/run steps.
     • The only validation of outputs is done by the provider class impl itself, or implicitly by dependent builds that start mysteriously failing.
   • The phases "build", "link", and "run" differentiate between how the output is consumed, but they don't at all address the inverse questions of "what types of output can this package provide?" or "what types of output does this package require from a specific dependency?"
     • Variants are often used to signal this information, since they are the only declarative (i.e. visible to clingo) method Spack provides to represent this concept of "things i will build" vs "things i need to build/run". But that meaning is often informal, as package variants are not propagated to all transitive dependencies the way the list of un/satisfied virtual dependencies are with Finer selection of virtual providers #15569.

Description

Three major areas I think we could consider:

Generalized Non-Nestable Scoping

It seems that the spec syntax intentionally avoids any paired delimiters or scopes (one reason: in order to ensure command-line specs can be unambiguously interpreted). However, there is likely room to extend the square brackets [^virtual=x@y.z] proposed in #15569 in order to start representing further new classes of information.

In particular, if we can establish the types of spec syntax items that it makes sense to be able to group together, that also would immediately allow us to implement the conflicts() directive by allowing a negation operator to be applied to any grouped item. i.e. if we can agree that [x@y.z] makes sense, then we can also express the unambiguous idea of [!x@y.z].

Establishing an unobstrusive, non-nestable scoping paradigm by generalizing the [] from #15569 may allow us to avoid many further changes to the spec semantics, even if we end up adding a few new tokens.

I described some ways we could specify and interact with scoped specs in

spack/test.py

Lines 64 to 67 in 8777b1d

	# TODO: We have seen '!' used above for negation, but only while referring to a precise single
	# lexeme for version strings. Can we apply it more generally?
	# Proposal: define an unambiguous grouping methodology for valid spec strings, where each
	# character is either:

and below (lots of ideas in that file, though).

This addresses (3), (4), (5), (6).

A Spec is a Conjunction of Assertions <=> A Package Definition is Serializable

Consider:

1. If we can successfully add enough room in the spec syntax to represent all types of relevant constraints in an unambiguous, user-friendly way which is the same across both the CLI and package.py files,
2. If we can extend the spec syntax to allow representing conjunctions of sub-specs (which can be unambiguously normalized, or shown to conflict),

That means we will have created a spec syntax which has a bijection with the exact information we provide to clingo. We may then be able to avoid creating any tables at all within spack (see (7)) while building up clingo inputs, instead computing effects on-demand. This is likely to make spec resolution easier to test and easier to implement in a different type of solver. It is also likely to make the task of pretty-printing .unsat_core()s much easier, if we can unambiguously map each constraint provided to the ASP solver back to a single unique depends_on(...) directive, or perhaps to a single CLI argument.

Some further paths:

• We would be able to unambiguously specify the identity of each package (i.e. the change to the global constraint set induced by evaluating package.py) with the exact same representation the user provides on the command line.

  • It's possible this could make the grounding phase faster, if we can provide clingo a known finite domain of atoms that's linear in the number of the total packages (~5000, IIRC).

• We would then be able to make clingo work offline to produce all (or some subset of) valid (possibly overlapping) models which would satisfy a given spec. This opens the possibility of pre-concretizing for all architectures, e.g. on an x86 laptop. Since a spack environment can now be represented as a spec, we can avoid bootstrapping full binaries for difficult platforms, and suddenly we just created a highly compressed file format that can be executed on all supported platforms at once without modification.

Require Explicit Dependencies on .libs and .headers, and Validate Them

This is a paradigm that pants, another build tool, has been moving to over the past few years for defining packages ("targets" in pants parlance). The site has a great blurb on what this means (at https://www.pantsbuild.org/docs/target-api-concepts#a-field-driven-api):

Pants plugins do not care about specific target types; they only care that the target type has the right combination of field types that the plugin needs to operate.

For example, the Python autoformatter Black does not actually care whether you have a python_library, python_tests, or custom_target target; all that it cares about is that your target type has the field PythonSources.

And the part I think is particularly relevant here is that Field has its own entire class hierarchy which decouples validating outputs from consuming inputs. As described in adding custom validation, it's mainly expected that the Field will take care of validating itself once you add it. The pants approach is particularly nice because both the provider and the consumer need to explicitly turn it on, and because it attaches a validation process that is always run to check that the output artifacts (libs, headers) actually work. Spack essentially has all of this, but the validation part isn't shared across packages at all, possibly attracting mild bitrot.

I ranted and raved about this to @becker33 a while ago: pants implements this via this very basic "UnionRule" concept I made up: pantsbuild/pants#7116. I don't think any of the pants rule DSL is relevant to this, but I can explain the mechanics of it further if necessary.

For these explicit dependencies to be visible to clingo, they should be represented in an extension of the spec syntax. This is one proposal for representing that "provides" / "consumes" relationship as a spec (which luckily ended up extremely similar to #15569):

spack/test.py

Lines 60 to 62 in 8777b1d

	'^[%py@=]python@2.6.*[%py@:!3+build]' == '^[%py@=]python@2.6.*%py@:!3+build'
	# TODO^: could also just require all "complex/compiler dependencies" or provides to be enclosed
	# in []! Easier to visually differentiate.

This "explicit resource dependency on a particular package, implicit resource validation" paradigm seems like a possible first step to eventually being able to infer which resources are available via binary analysis (CMake does a pale but useful imitation of binary analysis here).

The concrete proposal here is:
(1) Set up backwards-compatible "resource dependency" registration mechanisms.
- Create a shared base class/mixin which has hook methods to register resource dependencies with provides(...).
- (for example) All resources must be associated with a statically-known type (not a name), which is validated at runtime.
- Each resource class (like the Field concept above) overrides some hook methods to add themselves to the environment correctly.
(2) Write adapters for "virtual dependencies" and "compiler dependencies" to the new "resource dependency".
- Expose any validation methods in compiler.py as resource dependencies via (1).
- Convert all virtual dependencies into explicit resource dependencies (the resource validation can just be a no-op at first).
(3) Remove the distinction between virtual and compiler dependencies.
- Remove #15569's [^virtual=x@y] to instead add unambiguous optional "provides" and "requires" sections, both of which require being enclosed by a [] scope:

# 15569:
s1 = Spec('a [^rpc=x@y]').normalize()
# 15569 has no way for a to declare a resource dependency yet, since I just made it up.
# Something relegated to package.py right now is:
class A:
   provides('rpc')
class B:
    depends_on('[^rpc=x@y]', type=('build', 'link', 'run')


# Proposed refactor:
s2 = Spec('a%rpc=x@y').normalize()
assert s2 == s1
# In this case, the single element of the sub-spec can be parsed
# as either a "provides" or "requires" section -- WOLOG we can pick "requires". 
assert s2 == Spec('a ^[%rpc=x@y]').normalize()

# The `type` multi-valued variant is added to all "resource dependency" subclasses.
# This is equivalent to `class B`'s dependency above:
s3 = Spec('b%rpc=x@y type=build,link,run').normalize()
assert s3 == Spec('b%[x@y type=build,link,run]').normalize()

Here's an excerpt from the prototype syntax I made up earlier again:

spack/test.py

Lines 231 to 247 in 8777b1d

	THIS-PROVIDES-SPEC = '[' '!'? [PROVIDERS-SPEC] ']'
	# This did not use GROUPING-OPERATOR(), because we *require* that "provides" specifications
	# are wrapped with [], but "requires" are optional (for backwards compatibility).
	# - e.g. '^[%py@=]python@2.6.*[%cc]' == '^[%py@=]python@2.6.*%cc'
	THIS-REQUIRES-SPEC = ALTERNATION-OPERATOR([PROVIDERS-SPEC])
	# This is called a "node" because it represents all incoming and outgoing relationships.
	SPACK-SPEC-NODE = ALTERNATION-OPERATOR({
	[THIS-PROVIDES-SPEC]? [PACKAGE-NAME] [VERSION]? [VARIANT-SPEC] [THIS-REQUIRES-SPEC]?
	})
	DEPENDENCY = ALTERNATION-OPERATOR({'^' [SPACK-SPEC-NODE]})
	SPEC = ALTERNATION-OPERATOR({
	ALTERNATION-OPERATOR([SPACK-SPEC-NODE]) |
	[SPEC] [DEPENDENCY]
	})

Additional information

Feedback from @alalazo when I mentioned I had been thinking about this:

The prototype branch for compiler as deps is here: https://github.com/alalazo/spack/tree/features/compiler_as_dependencies
9:32
Severely outdated (so feel free to ping me if you want to look at it and something isn't clear). The point that would still stand, imho are:

• var/spack/repos/runtime contains synthetic packages for runtime dependencies
• gcc has a method, imposed_dependencies that contains information on which dependencies should be injected into the DAG by the compiler

These points should be used to guide the discussion above. In particular:

• var/spack/repos/runtime would correspond to e.g. [%cc type=run], i.e. a resource dependency (probably just LibraryList)?
• imposed_dependencies would be the name of the method to access the resource dependencies produced by gcc.

So I agree these points would still stand, as you said.

[cosmicexplorer]

attempting to diagram "Spec v2":

[provides] (pkg@ver)   [requires => same members and grammar as the current compiler section (after %)],
                                     but with [...] groups!
___V_          V        ________V__________
[%py]python@2.6.*[%py@<3 type=build]                  => SPEC-NODE
                              ^
                      desugars to python@2.6:2.6.999

alternatively (same):
[%py]python@2.6.*%py@<3+build


that's a SPEC-NODE, what's a SPEC?
SPEC = {
  SPEC-NODE |
  SPEC '^' SPEC
}

[...] are not just for decoration: these are the objects that can be grouped:
[%py], [%py@<3], [%py@<3+a~b], [%py+a~b], [+a~b], [+a], [~b], [%py type=[build]]
[python], [python@2.6.*], [2.6.*], [2], [6.*], ...

also, everything that can be grouped with [...] can be negated with [!...] !
this would correspond to a `conflicts()` directive.

in .../python/package.py:

class Python(AutotoolsPackage):
    ...
    # "I, the python package, will build a python
    # which will add  artifacts to the spack environment before spack runs all of these phases"
    provides('%py+build+link+run')  # this is actually the default value of those variants when using provides()
    add('[%py+build+link+run]')       # add() adds a parsed constraint to the model -- this is what the above becomes 
    # only one provider of %py (and therefore only one version) can be registered during any phase:
    depends_on('%py@<3+build')   # default here too is +build+link+run if *none* are specified
    #                                                   -- but if any *are* specified, others are assumed to be false (~link~run, in this case)

^note the most recent expression above: if this is the desired behavior (to assume +build => ~link~run), it might be a lot more elegant using a multi-valued variant like [%py type=build,link,run].

in some_other_file.py:

class ResourceDependency(object):
    ...
        variant('type', default=','.join(self.phases), description='...')

one thing i really like about this approach (the "requires" and "provides" registration -- i think @tgamblin described wanting something like this in a github comment using different words) is that it makes it a lot easier to think about a spec as a conjunction of logical expressions (which is already what we're doing with the ASP solver anyway), where one can imagine the requires "imply" the the specified outputs are available.

[cosmicexplorer]

I also think that the existing dependency operator ^ already does exactly this:

If we can extend the spec syntax to allow representing conjunctions of sub-specs (which can be unambiguously normalized, or shown to conflict),

and it works (and it's linear) because specs just get flattened by name -- there is no recursion necessary to merge two Specs

[cosmicexplorer]

Ok, I think I now have a strongly reduced version of the first draft in the OP:

Problems

1. FIGURED OUT: There's no distinction between real and virtual dependencies in the spec syntax. This can be confusing sometimes.
   • Finer selection of virtual providers #15569 introduces new syntax for virtual dependencies, but it appears virtual dependencies are otherwise still indistinguishable from real dependencies to the Spec parser.
   • This can be addressed by the proposal above to merge "virtual" and "compiler" dependencies, and to use % to represent both of these use cases.
     • ^ wouldn't mean "dependency", it would be a binary operator "and", which was a separate requirement from the OP that we can now fulfill.
       • a ^ b is still a valid Spec string, and the feasible ComplexResources are a union of the two's requirements/outputs, so this definition is backwards-compatible!
     • While "virtual" dependencies would be completely folded into ComplexResource, the act of pinning/assigning a virtual from Finer selection of virtual providers #15569 remains.
       • It would be represented with [%virtual=package@version] instead of [^virtual=package@version] as per Finer selection of virtual providers #15569 under this plan!
     • This should be describing a strict superset of the current spec string syntax, because each new syntax entry is enclosed within [], which Spec v1 does not recognize.
   • Implicit in this plan is a process to:
     • create a ComplexResource base class which:
       • has hooks for subclasses to validate outputs from built packages (see OP), or provide a default value (like pants Field classes)
       • has hooks for registering that a package provides a given ComplexResource subclass (may be abstract -- they are keyed by type)
       • has hooks for automatically generating the type=build,link,run multi-valued variant
       • is a Spec represented by: %name@v0.16.0+some_variant~other_variant.
         • uses the most recent git tag as its version (selectable with @...).
     • declare every compiler as a provider of %cc and %cxx!
       • Note that %cc and %cxx are ComplexResource subclasses.
       • gcc@10.2.0 ~> [%cc+gnu+c98 type=build,link,run %cxx+cxx11+cxx14 type=build,link,run]gcc@10.2.0/xxx[{%cc,%cxx} type=build,link] ^ ... (the rest of gcc's deps)
         • "This gcc version provides %cc and %cxx, with all support for modern C/C++. To build it, spack will use the binaries from whatever fulfills the %cc and %cxx ComplexResources, but avoid polluting the runtime PATH with either of those.
     • declare every Package as a provider of %headers and %libs, which wrap HeaderList and LibraryList.
2. FIGURED OUT: Finer selection of virtual providers in a DAG #15443 and Finer selection of virtual providers #15569 completely address the UX issue of "how can I set constraints on what a virtual dependency resolves to at all in a spec string"?
3. FIGURED_OUT: There's no way to represent in a single spec that a dependency needs to be built with a separate compiler.
   • This can be addressed with the proposed "requires"/"consumes" extension to the spec syntax, e.g. [%cc%cxx~link]gcc@10.2.0+binutils[{%cc,%cxx}=gcc@<5] says:
     • a package with the name gcc and version 10.2.0, with its variant +binutils activated
     • which itself was built with a C and C++ compiler (both of which may only use a gcc less than version 5),
     • provides the cc virtual dep in all phases, and the cxx virtual dep in all phases except link.
4. There are (known and intentional) disparities between the Spack CLI spec syntax and the directives accepted in a package.py.
   • (3) is one of them, which we believe can be addressed.
   • See Specifying the compiler in a depends_on directive sets the compiler only on that node #20127 regarding some surprising behavior from package.py.
   • TODO: determine any intrinsic blockers to expanding the cli Spec syntax with analogues for every directive(...)** (which is the goal here)
5. FIGURED OUT: It seems clear that we should look to expand the [^virtual=real@ver] from Finer selection of virtual providers #15569 into considering [] as beginning a non-nestable paired delimiter. See above for details.
6. FIGURED OUT: We should introduce <,<=,>,>=,==,!= from pip requirements.txt as version comparators, and * as a wildcard version component.
7. FIGURED OUT: If every component of the CLI spec, as well as every directive in every package.py file, was converted into its unambiguous deterministic canonical Spec representation, and every Spec s has a 1:1 unambiguous bijective mapping with a specific ASP assertion a, we should be able to link every element of an .unsat_core() back to the precise set of CLI spec components and directives that caused the issue (the input, not the output)
8. FIGURED OUT: Two parts:
   • A provider of a virtual dependency currently makes absolutely zero guarantees about the files it provides to any dependent build/link/run steps.
     • The only validation of outputs is done by the provider class impl itself, or implicitly by dependent builds that start mysteriously failing.
   • The phases "build", "link", and "run" differentiate between how the output is consumed, but they don't at all address the inverse questions of "what types of output can this package provide?" or "what types of output does this package require from a specific dependency?"
     • Variants are often used to signal this information, since they are the only declarative (i.e. visible to clingo) method Spack provides to represent this concept of "things i will build" vs "things i need to build/run". But that meaning is often informal, as package variants are not propagated to all transitive dependencies the way the list of un/satisfied virtual dependencies are with Finer selection of virtual providers #15569.
   • See ComplexResource above!

[cosmicexplorer]

Much more succinct version, in order of expected completion:

1. [#20258] Support pip requirements.txt version comparators and wildcards.

• <=,=>, and == already exist.
• ==9.2.* => >=9.2,<9.3 (reduce to subproblem)
• !=9.2.0 => <9.2.0,>9.2.0 (reduce to subproblem)
• imprecise:
  • <9.2.0 => :9.1.999
  • >9.2.0 => 9.2.0.0.0.1
  • Spack is unable to represent strict inequalities (<,>) without this feature.
• Breakage: none.

2. [TODO] Differentiate virtual dependencies from normal dependencies by pooling them in a new section of Spec syntax.

• This can be addressed with the proposed "requires"/"consumes" extension to the spec syntax, e.g. [%cc%cxx~link]gcc@10.2.0+binutils[{%cc,%cxx}^gcc@<5] says:
  • a package with the name gcc and version 10.2.0, with its variant +binutils activated
  • which itself was built with a C and C++ compiler (both of which may only use a gcc less than version 5),
  • provides the cc virtual dep in all phases (if requested), and the cxx virtual dep in all phases except link (if requested).

Changes to Spec Syntax:

• Proposed: use % to describe dependencies on virtual deps instead of ^!
• While "virtual" dependencies would be completely folded into ComplexResource, the act of pinning/assigning a virtual from Finer selection of virtual providers #15569 remains!
  - It would be represented with [%virtual=package@version] instead of [^virtual=package@version] as per Finer selection of virtual providers #15569 under this plan!
• let %gcc@4 be transformed to %=gcc@4 if gcc is a package name.
  • %=gcc@4 says "for all virtual interfaces that gcc@4 supports (e.g. %cc), add an assertion (z3 or clingo) that pins the virtual %cc,%cxx, etc to use the package gcc@4 (gcc'@4, to be specific).
• ^ wouldn't mean "dependency", it would be a binary operator "and", which was a separate requirement from the OP that we can now fulfill.
  • a ^ b is still a valid Spec string, and the feasible ComplexResources are a union of the two's requirements/outputs, so this definition is backwards-compatible and consistent with spack v1 concepts!
• Finally, note that this should be describing a strict superset of the current spec string language, because each new syntax entry is enclosed within [], which Spec v1 does not recognize.

3. [ALREADY PLANNED] Use virtual dependencies to represent language frameworks e.g. with %cc, %cxx, %py %py-devel.

• "Compiler dependencies" like %gcc should be converted to virtual dependencies for %cc and %cxx wherever possible instead of figured out by manually aligning version constraints.
  • Basically, convert each comment explaining each individual gcc version constraint into a non-gcc-specific variant name.
• This would allow us to request e.g. a c++14-compatible compiler to build a with by specifying a[%cxx+std14].
  - gcc@10.2.0 ~> [%cc+gnu+c98 type=build,link,run %cxx+cxx11+cxx14 type=build,link,run]gcc@10.2.0/xxx[{%cc,%cxx} type=build,link] ^ ... ("..." = "the rest of gcc's deps")
  - "This gcc version provides %cc and %cxx, with all support for modern C/C++. To build it, spack will use the binaries from whatever fulfills the %cc and %cxx ComplexResources, but spack should completely avoid polluting the runtime PATH with either of those."
• Ideally literally every direct %gcc@... instance can be found and replaced with something like [%cc+c98+gnu%cxx+gnu] (may have a long tail here).

[TODO] 4. Ensure every file used in every build phase was explicitly requested from a package.py.

[#20260] Phase 1: Use fakechroot in CI

1. Add fakechroot package (see docs at https://github.com/dex4er/fakechroot/wiki).
2. Add a +fakechroot variant in AutotoolsPackage which uses fakechroot to execute configure, make, and install.
3. Turn on +fakechroot in our CI (this may be a long tail).

[TODO] Phase 2: Replace "virtual deps" ([^...]) and "compiler deps" (%...) with ComplexResource dependencies

• Implicit in this plan is a process to create a ComplexResource base class, which:
  • has hooks for subclasses to validate outputs from built packages (see OP), or provide a default value (like pants Field classes)
  • has hooks for registering that a package provides a given ComplexResource subclass (may be abstract -- they are keyed by type)
  • has hooks for automatically generating the type=build,link,run multi-valued variant for all subclasses
  • is a Spec represented by: %name@v0.16.0+some_variant~other_variant.
    • uses the most recent git tag or git rev-parse HEAD as its version for invalidation (selectable with @...).
• declare every Package as an automatic provider of %headers and %libs virtuals, which wrap HeaderList and LibraryList.
  • Each package.py has the control to opt out of sending those artifacts e.g. in a particular phase now.
  • We can verify that after all the spec changes above are implemented, clingo can still be bootstrapped (see clean up how spack builds clingo #20159) from the spack/centos6 image, and that that cmake build pipeline is entirely communicating via ComplexResources.
    • legacy API functions can be tracked or overridden to be forced to fail to perform this verification.

[alalazo]

Reading through #20256 (comment), adding a few comments for discussion:

There's no way to represent in a single spec that a dependency needs to be built with a separate compiler.

Currently we can do this:

hdf5 %intel ^zlib %gcc

since the compiler is a node attribute. The limitations we have are that:

1. A compiler spec is identified by a name and a version only (e.g. gcc@10.1.0)
2. A compiler spec is always an aggregate of C, C++ and Fortran compilers (e.g. gcc@10.1.0 usually brings in gcc, g++ and gfortran) . We can't treat mixed compiler toolchains, like LLVM for C,C++ and gfortran for Fortran, in a clean way

The general idea going forward is that by:

1. Permitting a finer selection of virtual dependencies from packages
2. Turning compiled languages into virtual dependencies and compilers into build dependencies

we can improve on the two limitations above and we can also start modeling language runtimes.

---

#20127 is a bug with the new concretizer that I need to fix rather sooner than later. The background for that is that our concretizers can currently deal with tree-like DAGs i.e. there needs to be a single root node - we can't at the moment concretize a "forest" with different roots.

When we added Spack environments as a feature there was an increasing request to be able to specify specs in a list and concretize them in a way that unifies all the constraints so that the resulting installations were compatible with each other and could be merged in the same view. The solution we came up with was to emulate this concretization by making all the user specs direct dependencies of a mock root that exists only as an implementation detail. See:

spack/lib/spack/spack/concretize.py

Lines 677 to 686 in 1681c19

	def concretize_specs_together(*abstract_specs):
	"""Given a number of specs as input, tries to concretize them together.
	Args:
	*abstract_specs: abstract specs to be concretized, given either
	as Specs or strings
	Returns:
	List of concretized specs
	"""

This is currently broken for the ASP-based solver if the spec contains a %<compiler> sigil.

[alalazo]

Concerning #15569, the proposal for the extension of the syntax after the merge of #15256 (that introduces the use of anonymous specs like ^dev-path=* to mean "any node that has dev-path set) is currently to:

1. Maintain the spec syntax flat (no nested scopes)
2. Introduce new delimiters (very likely square brackets) to specify edge attributes after the ^ sigil

Right now we have only one type of edge attribute, the dependency type, that is set implicitly during the concretization process and cannot be influenced by the command line. In #15569 the idea is to add another attribute (the virtual being satisfied by the edge) and make edge attributes specifiable from the command line:

hdf5 ^[virtual=mpi] mpich
netlib-scalapack ^[virtual=mpi] intel-parallel-studio ^[virtual=lapack] openblas

This leaves room for later extensions, for instance if we want to move compiler flags from nodes to edges:

hdf5 ^[virtual=c, cflags=-O3] gcc

This syntax requires to write a lot more characters than before, so the other proposal was to reuse the % sigil also for virtual dependencies:

hdf5 %mpi=mpich  <==> hdf5 ^[virtual=mpi] mpich

Essentially I was seeing a dependency expression started by the ^ sigil as specifying the existence of an edge in the DAG connected to a specific node but with the originating node still to be determined by the concretization process. The square brackets delimit the context in which we can specify edge attributes.

---

Concerning the separation in the syntax of virtual dependencies from normal dependencies: we need to take into account packages that are both a provider of something and a normal dependency at the same time. This might happen with packages that are bundle of many different things, like intel-parallel-studio. The use case is probably related to the SDK concept @scheibelp is working on.

[cosmicexplorer]

Summary

Thank you so much for that brilliant overview! It looks like there are several directions we can expand spack's spec syntax right now, and I was super excited to see where our thoughts met. Here are the remaining useful ones I identified from our discussion:

1. Pin a virtual dependency as in Finer selection of virtual providers #15569.

• 1.1 proposes [virtual=mpi] mpich.
• 1.2 and 2.1 propose %mpi=mpich.

2. Attach compiler flags to edges, not just nodes.

• 1.1 proposes [virtual=c, cflags=-O3] gcc, which requires c to be a virtual dependency.
• 2.1-2.3 propose [%c=gcc, cflags=-O3].

3. Represent the bidirectional provides and requires (dependency) relations that may be present in SDKs or other bundled packages.

• 2.6 proposes [%py]python@2.7:!3[%cc=gcc], which declares that the package provides py, and requires gcc to implement cc.

4. Is backwards compatible with yaml files using spack spec v1.

• 2.7 proposes %gcc = [%cc=gcc,%cxx=gcc], which will allow %gcc specs to be handled without fear.

---

Below I define Edge Syntax 1 from your comments, and Edge Syntax 2 from my comments. I am likely exaggerating somewhere but I have tried not to.

I'm going to use numbered list items like the below to denote new expressions to add to the spec syntax!

• 1.1
• 1.2
• ...

Edge Syntax 1: virtual keyword first, trailing comma to separate variants and flags

@alalazo, this is my interpretation of your comment.

This leaves room for later extensions, for instance if we want to move compiler flags from nodes to edges:
hdf5 ^[virtual=c, cflags=-O3] gcc

I really like the notion of moving from nodes to edges! And I hadn't considered using the trailing comma to separate the variant list without having to check and push any tokens back!

Proposed virtual pinning:

• 1.1 Let <virtual-pin> = ^[virtual=<virtual>(',' <variant-or-flag-chain>)] <non-virtual-spec>.

% Type 1: replace ^[...]

This syntax requires to write a lot more characters than before, so the other proposal was to reuse the % sigil also for virtual dependencies:
hdf5 %mpi=mpich <==> hdf5 ^[virtual=mpi] mpich

Proposed alternate virtual pinning:

• 1.2 %mpi=mpich

Result

• 1.1 virtual dependencies can be pinned by adding a spec like: ^[virtual=mpi] mpich.
  • Compiler flags can be added along a virtual dependency edge with ^[virtual=mpi, cflags=-O3] mpich.
• 1.2 this was extremely useful. %mpi=mpich demonstrates that we can decouple pinning a virtual dependency from square brackets [], freeing those up for a more general interpretation!!
  • I use this fact directly below.

@alalazo am I missing any changes you proposed besides 1.1 and 1.2?

Edge Syntax 2: no virtual assignment, real package first, trailing comma to separate

This is my interpretation of my comment.

Your % Type 1 approach uses either ^[a=b] or %a=b to pin the virtual dependency a to a spec b. Here I want to try to differentiate virtual dependencies from normal dependencies, by pooling them in a new section of spec syntax.

Use the same virtual dependency pin syntax as described in % type 1:

• 2.1 Let %a=b pin the virtual dependency a to a spec b (same as % type 1).
• 2.2 Let %a (without =) specify a virtual dependency on a (same as spack right now).
• 2.3 There are now two types of bases for a spec node, virtual and non-virtual.
  • <spec-node> = <virtual-dep-node> | <non-virtual-node>.
  • <spec-node> is a spec.
  • Valid spec nodes:
    1. virtual: <virtual-dep-node> = %<virtual-dep-name>(=<virtual-pin-spec>)?<variant-or-flag-chain>.
       • Nothing should change about dependency resolution, except to implement %a=b virtual dependency pinning which was not possible before Finer selection of virtual providers #15569.
    2. non-virtual: <non-virtual-node> = <package-name>(@<version>)?<variant-or-flag-chain> (same as Edge Syntax 1).
• 2.4 Let <spec-node-chain> = <spec-node> ('[ ,]' <spec-node>)*.
  • <spec-node-chain> is a spec..
  • The comma , is equivalent to installing multiple specs with a space between them.

Two-Way Virtual Deps

Reading this:

Essentially I was seeing a dependency expression started by the ^ sigil as specifying the existence of an edge in the DAG connected to a specific node but with the originating node still to be determined by the concretization process. The square brackets delimit the context in which we can specify edge attributes.

I then started thinking about [] only to carry edge attributes. Very importantly, I recalled that your construction %mpi=mpich demonstrated that we can decouple pinning a virtual dependency from square brackets [], freeing those up for a more general interpretation.

If we use [] for edge attributes without a ^ sigil, we can make it a top-level spec:

• 2.5 Let [<spec>] = <spec>.
  • <spec-node-chain> is a spec..

Concerning the separation in the syntax of virtual dependencies from normal dependencies: we need to take into account packages that are both a provider of something and a normal dependency at the same time. This might happen with packages that are bundle of many different things, like intel-parallel-studio. The use case is probably related to the SDK concept @scheibelp is working on.

So far, a virtual dependency is only ever a dependency or a pin expression, never the provides relation this use case requires!

Add edge attributes onto either side of a node to denote "requires" vs "provides":

• 2.6 Let <node-with-edges> = [<VIRTUAL SPEC NODE CHAIN 1>]?<NON VIRTUAL SPEC>([<VIRTUAL SPEC NODE CHAIN 2>])?.
  • Then, <node-with-edges> is said to require the virtual dep chain <VIRTUAL SPEC NODE CHAIN 2>, and to produce the virtual dep chain <VIRTUAL SPEC NODE CHAIN 1>.
  • <node-with-edges> is a spec..
• e.g.:
  • [%py]python@2.6:!2.7[%cc] = [%py] python@2.6:!2.7 [%cc]:
    1. provides a virtual py which can execute python code.
    2. requires a virtual cc (C compiler) to build.
• 2.7 Detect and convert old-style compiler constraints.
  • python@2.6:!2.7%gcc = python@2.6:!2.7[%cc=gcc,%cxx=gcc]:
    • In order for the new spec parser to be fully backwards compatible with the old, we must be able to handle <NON VIRTUAL SPEC>%<NON VIRTUAL SPEC>.
    • Blocker: Once gcc's package.py is made to provide a virtual cc which knows how to compile C code, we can convert %gcc into [%cc=gcc,%cxx=gcc]

Result

• 2.1-2.3 created a virtual dependency analog (with the base %<virtual-dep-name>(=<virtual>)?) of a "real" spec node (which has <package>@<version>).
• 2.4-2.5 allows joining (with ' ' or ',') and grouping (with [...]) specs.
• 2.6 represents the bidirectional provides and requires relationships each package can declare by allowing a <VIRTUAL SPEC NODE CHAIN> on either side of a <NON VIRTUAL SPEC>.
• 2.7 ensures all of the new-style behavior occurs via square brackets [], which the spec v1 syntax doesn't know about.

Possible Future Spec Directions

• e.g. [%cc@dbfda9f8+99%cxx+11+14+17]gcc@10.2.0+binutils[{%cc+99,%cxx}=gcc@!5] says this node is:
  • a package with the name gcc and version 10.2.0, with its variant +binutils activated (the rest default),
    • (any additional edge attributes that +binutils induces are not shown),
  • which itself was built with a separate C compiler %cc which supports C99,
    • (both of which must be gcc, and the version for %cc and %cxx must be less than 5),
    • :!5 translates to "less than 5" as per support strict version inequalities in specs #20258.
  • if requested, may provide the cc virtual dep in any phase with support for C99, and the cxx virtual dep in any phases but link with support for C++11/14/17.
    • the cc virtual dep is annotated with an implementation version (which is the checksum of the file that defines the virtual dependency).
      • This assumes a virtual dependency is defined in a file! Right now, it's implicit!

New elements in the above spec:

1. virtual dependencies are declared in a specific file.
2. virtual dependencies can use <virtual-dep-name>@<version> to access an implementation version.
3. a virtual dependency will often declare many variants corresponding to language version.
4. a combination operator which pairs every left element with every right element in order.

• e.g. [{%cc,%cxx}=gcc] = [%cc=gcc,%cxx=gcc].
• This {,} operator can be used on package names, virtual names, whole version strings, and version string components.

Possible further elements:
5. recursive [].
6. automatically declare variants configure, build, install, etc.
- All build phases get a boolean variant e.g. +install defaulting to true.
- If the variant is off, do not run during the corresponding build phase.

[becker33]

@cosmicexplorer I don't know that I can fully respond to every point here, but there are a couple things I have strong feelings about.

The first is that I do not think we should use the % sigil for virtual dependencies. I think we will want to differentiate between build deps and link/run deps on the command line, and I think that's a more appropriate co-option of the % sigil that previously referred to compilers. For example, in a cross-compiling environment users may need to specify foo %python+ssl ^python@3.8.6 ("Use python+ssl to satisfy the python build dependency on the FE node, and use python@3.8.6 to satisfy the python link/run dependency on the BE node").

Second, I think language version support should be expressed as a version on the virtual dependency, rather than variants. So a package could request cxx@17:20 or c@99: to specify dependency on ranges of language standards.

[cosmicexplorer]

I think it's very reasonable to avoid using % for virtual dependencies for now. The reason we would want to expand the concept of virtual dependencies would be to obtain something like this ComplexResource object I tried to define above, as compared to the imho very thin concept of virtual dependencies we have right now. This comment in particular defined the object that would motivate using %: #20256 (comment)

• Implicit in this plan is a process to:
  • create a ComplexResource base class which:
    • has hooks for subclasses to validate outputs from built packages (see OP), or provide a default value (like pants Field classes)
    • has hooks for registering that a package provides a given ComplexResource subclass (may be abstract -- they are keyed by type)
    • has hooks for automatically generating the type=build,link,run multi-valued variant
    • is a Spec represented by: %name@v0.16.0+some_variant~other_variant.
      • uses the most recent git tag as its version (selectable with @...).
  • declare every compiler as a provider of %cc and %cxx!
    • Note that %cc and %cxx are ComplexResource subclasses.
    • gcc@10.2.0 ~> [%cc+gnu+c98 type=build,link,run %cxx+cxx11+cxx14 type=build,link,run]gcc@10.2.0/xxx[{%cc,%cxx} type=build,link] ^ ... (the rest of gcc's deps)
      • "This gcc version provides %cc and %cxx, with all support for modern C/C++. To build it, spack will use the binaries from whatever fulfills the %cc and %cxx ComplexResources, but avoid polluting the runtime PATH with either of those.
  • declare every Package as a provider of %headers and %libs, which wrap HeaderList and LibraryList.

We would have to agree to all of that first, or something else totally different.

[cosmicexplorer]

The issue I saw with not using % was tangential, in that specs currently have no way to determine whether something is a virtual or non-virtual provider just by parsing the spec. I believe @alalazo's solution in #15569 with ^[virtual=c, ...] dependency could still be adaptable to the following to define a syntax for virtual dependencies:

1. ^[virtual=c, ...] dep => pin the virtual package named c to the spec represented by dep.
2. ^[virtual=c, ...] => add a dependency to a virtual package named c.

This is so far just assuming we want #15569, none of the ComplexResource stuff above.

@becker33: I'm going to move the ComplexResource stuff to a new issue since it's clearly separate. I think the above could work, and I think it's a good thing to try to represent virtual dependencies differently than literal ones. In particular, it gets towards the goal in the actual OP, of representing all dependency information in a spec string. Quoted:

A Spec is a Conjunction of Assertions <=> A Package Definition is Serializable

Consider:
- If we can successfully add enough room in the spec syntax to represent all types of relevant constraints in an unambiguous, user-friendly way which is the same across both the CLI and package.py files,
- If we can extend the spec syntax to allow representing conjunctions of sub-specs (which can be unambiguously normalized, or shown to conflict),

That means we will have created a spec syntax which has a bijection with the exact information we provide to clingo. We may then be able to avoid creating any tables at all within spack (see (7)) while building up clingo inputs, instead computing effects on-demand. This is likely to make spec resolution easier to test and easier to implement in a different type of solver. It is also likely to make the task of pretty-printing .unsat_core()s much easier, if we can unambiguously map each constraint provided to the ASP solver back to a single unique depends_on(...) directive, or perhaps to a single CLI argument.

[cosmicexplorer]

I also should have made it clear that % wouldn't have been used until we were able to convert all compilers to be virtual dependencies first. I was under the impression (maybe incorrectly) from @alalazo's comments that it could make sense to do that. In my comments just above, it was recorded as:

declare every compiler as a provider of %cc and %cxx!

I believe this could be done right now as a proof of concept. Do we think compilers and virtual dependencies should eventually be merged? would be a good question to ask.

I think that virtual dependencies have no definition except "has a name" right now. If that's not considered as much of a problem to others, I take no issue with that.

[becker33]

Compilers need to be virtual dependencies for the compilers-as-dependencies work that @alalazo has already started on, but there is a difference between a virtual build dependency (pkgconfig), a virtual build dependency that imposes a link dependency on its dependents (c, cxx, fortran), and a virtual link dependency (mpi, blas, lapack). I think we need to be able to independently express those two concerns.

[cosmicexplorer]

Ok, thank you so much for that context!

there is a difference between a virtual build dependency (pkgconfig), a virtual build dependency that imposes a link dependency on its dependents (c, cxx, fortran), and a virtual link dependency (mpi, blas, lapack). I think we need to be able to independently express those two concerns.

I am pretty sure I agree with that. Here's how I was planning to do that:

1. Enable virtual dependencies ("requires") and virtual "providers".
   • No concept of build step yet.
   • From a proposal for how to represent virtual dependencies in the spec syntax #20256 (comment):
     • Left side is provides.
     • Right side is requires.

[%py]python@2.6:!2.7[%cc] = [%py] python@2.6:!2.7 [%cc]

• provides a virtual py which can execute python code.
• requires a virtual cc (C compiler) to build.

2. Specify build steps for individual virtual deps (requires) as well as downstream provides clauses (of virtual packages).
   • Relies on the concept from 1, and the parsing of those virtual dependencies and providers already having been done.
   • From the "future spec changes" at the bottom of a proposal for how to represent virtual dependencies in the spec syntax #20256 (comment):

Automatically declare variants configure, build, install, etc.

• All build phases get a boolean variant e.g. +install defaulting to true.
• If the variant is off, do not run during the corresponding build phase.

• With an example, from a proposal for how to represent virtual dependencies in the spec syntax #20256 (comment):

This can be addressed with the proposed "requires"/"consumes" extension to the spec syntax, e.g. [%cc%cxx~link]gcc@10.2.0+binutils[{%cc,%cxx}^gcc@<5] says:

• a package with the name gcc and version 10.2.0, with its variant +binutils activated
• which itself was built with a C and C++ compiler (both of which may only use a gcc less than version 5),
• provides the cc virtual dep in all phases (if requested), and the cxx virtual dep in all phases except link (if requested).

I believe this satisfies the conditions. Am I mistaken?

1. Virtual build dependency: mypackage ^pkgconfig => mypackage[%pkgconfig].
2. c, cxx, fortran =>
   • Declare a virtual build dependency: mypackage[%cc,%cxx,%fortran].
   • Declare that a link dependency is imposed by the implementor mycompiler of %cc: [%cc+link,%cxx+link]mycompiler.
     • With the alternation syntax {,}, [{%cc,%cxx}+link]mycompiler declares that mycompiler implements %cc and %cxx and imposes a link dependency on those phases.
3. Virtual link dependency: mypackage ^ mpi => mypackage[%mpi+link].
   • Although my intent was that if we declare mpich as [%mpi+link]mpich, that the +link can be omitted in the "caller" which requires the virtual dep.

[cosmicexplorer]

With #20258, #20260, and #20523 all broken out, I have renamed this issue to represent its narrowed focus. I would say that my comment immediately above: #20256 (comment) should be the entire amount that hasn't been addressed or moved to another issue. cc @tgamblin