How to handle packages that don't install to the traditional filesystem hierarchy

[adamjstewart]

In an ideal world, every package would install to the following directories:

bin
include
lib

Unfortunately, we don't live in an ideal world. Some packages like PGI and Lmod install into subdirectories:

linux86-64/17.4/bin
linux86-64/17.4/include
linux86-64/17.4/lib

others install into multiple directories:

include
include/foo
lib
lib/foo

So far we have inconsistent means of getting around this problem. Sometimes we can force the package to install to the normal directories. Sometimes we can use setup_environment to get the module file working, but this does nothing for the compiler wrappers that expect the libraries to be in lib. #4031 had to add lib/bamtools to the CMake RPATH to get it to build properly. Unfortunately this won't work for other build systems. And what about the packages that depend on these? Right now, prefix is immutable and there is no way to change prefix.lib to point to linux86-64/17.4/lib.

For the PGI/Lmod layout, being able to remap prefix globally would be useful. For the case of multiple directories containing libraries and header files, we need a different way to handle this. Right now, the compiler wrappers work through an environment variable, SPACK_DEPENDENCIES, which contains a list of the installation prefixes of each dependency. We could change this to separate environment variables for lib directories and include directories. Then packages that need to could append to these environment variables.

Thoughts? Are there any other scenarios I haven't thought of?

[tgamblin]

My general recommendation for packages like PGI is to symlink the contents of linux86-64 to the top level so it looks like a normal prefix (is that a typo or did they really make the arch descriptor into a portmanteau?). We do this for TAU and other packages that try to manage their own architecture-specific subdirectories.

In the second case, I consider that to be a reasonable install. Many libraries separate out their headers and libs into subdirectories, and it's part of the contract with developers. i.e., you're supposed to #include <foo/foo.h>, not just foo.h. The interface varies by package, and there's not much we can do about that.

There are cases where the same package is frequently installed two different ways (e.g. libelf is installed different on the system and in a single prefix) and you have to adjust the build. I guess bamtools is in that category too. For those I think we should have a simple way for packages to tell Spack about extra include paths that should be deafult, e.g.:

    # in some package
    def extra_rpaths(self):
        return [ <list of paths> ]

    def extra_includes(self):
        return [ <list of paths> ]

Spack would, as you say, put the above extra RPATHs and includes into the compiler wrappers. This isn't hard to implement -- we just haven't settled on an interface yet.

@mplegendre took a stab at this in #2645 -- we held off on it until @alalazo's #1875 was merged. Maybe it's time to revive that.

[davydden]

others install into multiple directories:

as long as it's include/foo/some-header.h, then that's actually good (not just reasonable 😄) to prevent possible conflicts among packages that use common names like utilities.h by asking users to do #include <foo/utilities.h> and #include <bar/utilities.h>, in addition to using namespaces in those packages, like namespace foo {...} and namespace bar {...}.

We could change this to separate environment variables for lib directories and include directories.
Then packages that need to could append to these environment variables.

I think setup_build_environment() (or whatever its name) should be refactored from collecting prefixes of link dependencies into collecting <prefix/path/to/lib/folder> via LIbraryList class. At some point we may need to have an interface to add to that list when a given package implements various virtual packages (i.e. BLAS is in prefix/blas/lib and ScaLAPACK is in prefix/scalapack/lib). But I am not sure there are any packages right now which would require that. I would say that we should be well covered if those few packages with a single custom lib location implement libs() method.

Similarly for include flags we should use the new class (analog of LIbraryList) introduced by @adamjstewart .

[citibeth]

In the PGI and Lmod case, why don't we look at what RedHat, Ubuntu, etc. do? I expect they somehow patch the package to install in the standard places. Adding linux86-64/17.4 to the path is the kind of thing that packages try to do, but that package managers do better.

[adamjstewart]

This discussion came up in #4300, so I'm copying the relevant bits here.

Packages like intel-mkl install dozens of libraries all over the place, none of which are in prefix.lib or prefix.lib64. We could symlink the important libraries to prefix.lib, but I really don't want to do that because:

1. It won't work for external packages
2. It's a lot of work to do this manually

To get around point 2, we could symlink all libraries to prefix.lib, but that also won't work because intel-mkl has multiple copies of the same library: #4300 (comment)

@davydden and @alalazo suggested using spec['name'].libs.ld_flags instead of -L<prefix>/lib -L<prefix>/lib64 and spec['name'].headers.cpp_flags instead of -I<prefix>/include. However, in its current implementation, this only searches for libraries like lib<name>.so. If we replaced this with lib*.so, it might actually work. Any packages that install normally would Just Work. Any packages that install libraries in weird locations would involve overriding libs and/or headers to get them to work. Thoughts?

[alalazo]

@adamjstewart We can probably do something like (pseudo-code):

try:
    ld_flags = spec[name].libs.ld_flags
except SomeSpecificError:
    tty.warn('Package {foo} still use the old method. Consider updating it.')    
    ld_flags = ... # Whatever we are currently doing

This will permit to transition smoothly from using prefix.lib to spec[name].libs. What do you think?

[davydden]

Any packages that install libraries in weird locations would involve overriding libs and/or headers to get them to work. Thoughts?

probably if packages install into weird locations they don't work currently either. So i would not be worried about this case.

What is trickier are packages which install into foo-prefix/lib but they don't have a single library libfoo.so like trilinos which in my installation has 193 (yes almost two hundred) libraries. So if we could extend LibraryList class to know about locations only and lot actual libs, this should be enough info for compiler wrappers.

As for the usage case, no sane project which link against trillinos would require users to specify those 193 libs manually as linking flags. Most of the time, config system searches for a few (5-10) libs that are actually required for a project.

p.s. another example is oce (108 libs).

[adamjstewart]

We can probably do something like (pseudo-code):

try:
    ld_flags = spec[name].libs.ld_flags
except SomeSpecificError:
    tty.warn('Package {foo} still use the old method. Consider updating it.')    
    ld_flags = ... # Whatever we are currently doing

This will permit to transition smoothly from using prefix.lib to spec[name].libs. What do you think?

@alalazo I don't even think this is necessary. As far as I know, spec['name'].libs doesn't return an error at the moment, does it? It probably should though... With my design (searching for all libraries, not just libname), if find_libraries doesn't find anything, then prefix.lib and prefix.lib64 won't have any libraries, so no need for a backup.

Any packages that install libraries in weird locations would involve overriding libs and/or headers to get them to work. Thoughts?

probably if packages install into weird locations they don't work currently either. So i would not be worried about this case.

@davydden Oh, I'm not saying this is a problem, I'm saying this is a feature! They currently don't work, but if we switch to using spec['name'].libs, we could make them work.

What is trickier are packages which install into foo-prefix/lib but they don't have a single library ... So if we could extend LibraryList class to know about locations only and lot actual libs, this should be enough info for compiler wrappers.

I don't think we need to extend LibraryList to understand locations without libraries, we just need to extend the default libs handler to search for all libraries, not just lib<name>. Yes, this may end up with a lot of libraries that aren't necessary, but it won't break anything (unless there's a weird bug like the too many RPATHs problem).

We may need to extend HeaderList though. I'm thinking about things like Eigen which install to:

$ ls include/eigen3/Eigen/
Cholesky        IterativeLinearSolvers  QtAlignedMalloc  StdDeque
CholmodSupport  Jacobi                  Sparse           StdList
Core            LU                      SparseCholesky   StdVector
Dense           MetisSupport            SparseCore       SuperLUSupport
Eigen           OrderingMethods         SparseLU         SVD
Eigenvalues     PardisoSupport          SparseQR         UmfPackSupport
Geometry        PaStiXSupport           SPQRSupport
Householder     QR                      src

where none of the header files contain a .h in the name. Many packages include <Eigen/LU>, so we would need to add -I<prefix>/include/eigen3. Not sure how to tell headers to return this directory.

[davydden]

where none of the header files contain a .h

o_O , so those are indeed header files without .h?! They look/named like directories to me.

Not sure how to tell headers to return this directory.

tricky indeed.

[adamjstewart]

I just noticed that intel-mkl uses:

def setup_dependent_environment(self, spack_env, run_env, dependent_spec):  
        # set up MKLROOT for everyone using MKL package                         
        spack_env.set('MKLROOT', self.prefix)                                   
        spack_env.append_path('SPACK_COMPILER_EXTRA_RPATHS',                    
                              join_path(self.prefix.lib, 'intel64'))

Is this the recommended workaround until a better solution has been implemented?