[conan-center] Add a hook to check whether installed shared libs are relocatable on Linux & macOS #376

SpaceIm · 2022-01-24T15:08:12Z

Currently, many recipes don't produce state of the art relocatable shared libs on macOS.

It would be nice to check whether shared libs on macOS properly have @rpath/<shared> in install tree (@rpath token is supported since macOS 10.5 Leopard, so since 2007...).
It must not be <shared> or <absolute/path/to/shared> (many binaries produced by conan-center recipes on macOS have the former actually, it's bad).
see https://cmake.org/cmake/help/latest/prop_tgt/MACOSX_RPATH.html#prop_tgt:MACOSX_RPATH

Moreover, rpath should be empty in installed shared libs.
Usually, in a default CMake configuration, binaries produced in build tree have absolute paths of dependencies (direct & transitive) libs folder in their rpath, so that you can run your executable for free (on macOS, it only works if external shared libs also have @rpath token, so proper relocatable binaries), then it is cleared by CMake during installation.

It also means that all CMake based recipe must:

use conan_basic_setup(KEEP_RPATHS)
recipe should ensure that https://cmake.org/cmake/help/latest/policy/CMP0042.html#policy:CMP0042 is enabled (either by injection CMAKE_POLICY_DEFAULT_CMP0042 NEW, or ensuring cmake_minimum_required(VERSION 3.0) or higher in upstream CMakeLists).
recipe should call install target, not manually copying files (it can be allowed if target os is Windows), so that CMake can:
- eventually replace absolute path set in shared libs of the build tree with @rpath for shared libs in install tree (usually it's already @rpath in the build tree, but not always).
- clear rpath (and it's also important on Linux). To do so, CMAKE_INSTALL_RPATH & CMAKE_INSTALL_RPATH_USE_LINK_PATH should not be manipulated.
If manual copy can't be avoided, something like this at the end of package() may allow to add @rpath token into dylib files :
```
 if tools.is_apple_os(self.settings.os):
     with tools.chdir(os.path.join(self.package_folder, "lib")):
         for dylib in glob.glob("*.dylib"):
             command = "install_name_tool -id {0} @rpath/{1}".format(os.path.basename(dylib), dylib)
             self.run(command)
```

I suspect that many issues consumers have with "all shared" builds (-o *:shared=True) on macOS would be solved. Default behavior of conan_basic_setup on macOS is really harmful.

see conan-io/conan-center-index#9052
also conan-io/conan#1238 or conan-io/conan#10253
https://developer.apple.com/library/archive/documentation/DeveloperTools/Conceptual/DynamicLibraries/100-Articles/RunpathDependentLibraries.html
https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling#mac-os-x-and-the-rpath
https://www.mikeash.com/pyblog/friday-qa-2009-11-06-linking-and-install-names.html

The text was updated successfully, but these errors were encountered:

uilianries · 2022-01-24T18:09:52Z

It sounds interesting

SpaceIm · 2022-01-24T22:46:43Z

Regarding autotools builds, it's worse, path of build machine is hardcoded in installed shared libs on macOS. Maybe this thread in spark package manager would help to fix these recipes: spack/spack#26608

Well actually few autotools based recipes have this fix. I have to admit I've removed this fix in some recipes, and it was a mistake, I thought it was handled by Autotools helper.

Here is the fix before running configure:

        # Fix rpath on macOS
        if tools.is_apple_os(self.settings.os): # this condition is not even needed I think, install_name is specific to apple
            tools.replace_in_file(
                os.path.join(self._source_subfolder, "configure"),
                "-install_name \\$rpath/",
                "-install_name @rpath/",
            )

But there is another issue in autotools based recipes with 1 profile: they also have LC_LOAD_DYLIB pointing to shared libs of build requirements (with 1 profile at least) ! So when I put pkgconf or libtool in build requirements of recipes, and build all shared, I ends up with something like that in my installed shared lib (here libcurl built with autotools):

Load command 15
          cmd LC_LOAD_DYLIB
      cmdsize 144
         name /Users/spaceim/.conan/data/libtool/2.4.6/_/_/package/2d1097d0a07e0fd22fb66460fae7c4f88ab883f3/lib/libltdl.7.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 11.1.0
compatibility version 11.0.0
Load command 16
          cmd LC_LOAD_DYLIB
      cmdsize 152
         name /Users/spaceim/.conan/data/pkgconf/1.7.4/_/_/package/bda713dd3b257827c8d11a06ac9d824038871572/lib/libpkgconf.3.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 3.0.0
compatibility version 3.0.0

It doesn't make sense, ltdl and pkgconf are not dependencies of libcurl. I think it comes from autotools helper which inject libs from build requirements :(
I hope that the new Autotools helper in conan v2 will not inject libs of build requirements.

SpaceIm · 2022-01-25T17:18:43Z

Proposal:

Linux:

for each shared lib (not the symlinks) in the install tree of the recipe, check that objdump -x <libname> | grep RUNPATH doesn't return anything (no more hardcoded paths to dependencies which were added to shared libs of the build tree).
we could also check that all lines returned by objdump -x <libname> | grep NEEDED don't contain absolute nor relative paths.

macOS:

for each shared lib (not the symlinks) in the install tree of the recipe:
- run otool -L <libname>:
  - check that it can find @rpath/<libname> (don't be too strict about libname because it might be soversion, while the concrete file may have the lib version in its name, or nothing at all).
  - check that for each shared dependency libN, if <libN> is found in above command, it is of the form @rpath/<libN> (again, don't be too strict) => It requires to visit all cpp_info.libs of the dependency tree. Maybe it can be sufficient to check that there is no <conan_data_path>.
- check that otool -l <libname> | grep LC_RPATH doesn't return anything (no more hardcoded paths to dependencies which were added to shared libs of the build tree).

test the hook:

build 2 toy recipes A & B. A depends on B. A & B shared (A link to shared B, it's important).
recipes A & B built with CMake, and cmake_minimum_required(VERSION 3.0) (3.0 is important for macOS, otherwise CMP0042 must be injected).
recipe A relies on defaut behavior of CMake (https://gitlab.kitware.com/cmake/community/-/wikis/doc/cmake/RPATH-handling#default-rpath-settings), but with few variations:
- CMAKE_MACOSX_RPATH set to ON (conan_basic_setup(KEEP_RPATHS)), and files installed through a CMake install target (basically the default behavior) => test should succeed on Linux & macOS
- CMAKE_MACOSX_RPATH set to OFF (conan_basic_setup()), and files installed through a CMake install target => test should succeed on Linux & fail on macOS.
- CMAKE_MACOSX_RPATH set to ON (conan_basic_setup(KEEP_RPATHS)), and files installed manually (self.copy) => test should fail on Linux & macOS
- CMAKE_MACOSX_RPATH set to OFF (conan_basic_setup()), and files installed manually (self.copy) => test should fail on Linux & macOS.

…table closes #376

SpaceIm changed the title ~~Add a hook to check whether shared libs listed in cpp_info.libs are relocatable on Linux & macOS~~ Add a hook to check whether shared libs & executables are relocatable on Linux & macOS Jan 24, 2022

SpaceIm mentioned this issue Jan 24, 2022

sqlite3: relocatable binaries on macos conan-io/conan-center-index#9060

Merged

4 tasks

SpaceIm changed the title ~~Add a hook to check whether shared libs & executables are relocatable on Linux & macOS~~ Add a hook to check whether shared libs are relocatable on Linux & macOS Jan 24, 2022

SpaceIm changed the title ~~Add a hook to check whether shared libs are relocatable on Linux & macOS~~ [conan-center] Add a hook to check whether shared libs are relocatable on Linux & macOS Jan 24, 2022

SpaceIm changed the title ~~[conan-center] Add a hook to check whether shared libs are relocatable on Linux & macOS~~ [conan-center] Add a hook to check whether installed shared libs are relocatable on Linux & macOS Jan 24, 2022

SpaceIm mentioned this issue Jan 24, 2022

grpc: fix cross-build + provide all CMake executable imported targets + add shared option + modernize conan-io/conan-center-index#9044

Merged

4 tasks

SSE4 assigned uilianries Jan 24, 2022

This was referenced Jul 5, 2022

xerces-c: fixup install name on MacOS conan-io/conan-center-index#11548

Merged

mongo-c-driver: fix static build of 1.22.0 on windows conan-io/conan-center-index#11611

Merged

SpaceIm mentioned this issue Jul 18, 2022

re2: add 20220601 + relocatable shared lib on macOS conan-io/conan-center-index#11777

Merged

4 tasks

SpaceIm mentioned this issue Jan 15, 2023

[conan-center] New KB-H077 - test whether shared libs on macOS are relocatable #477

Merged

uilianries added a commit that referenced this issue Jan 20, 2023

Merge pull request #477 from SpaceIm/feat/kb-h077-apple-shared-reloca…

8914d76

…table closes #376

uilianries closed this as completed in #477 Jan 20, 2023

uilianries mentioned this issue Jan 20, 2023

[docs] Add hooks description for KB-H076 and KB-H077 conan-io/conan-center-index#15382

Merged

3 tasks

SpaceIm mentioned this issue Apr 2, 2023

vulkan-validationlayers: add 1.3.239.0 + modernize more for conan v2 conan-io/conan-center-index#16131

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[conan-center] Add a hook to check whether installed shared libs are relocatable on Linux & macOS #376

[conan-center] Add a hook to check whether installed shared libs are relocatable on Linux & macOS #376

SpaceIm commented Jan 24, 2022 •

edited

uilianries commented Jan 24, 2022

SpaceIm commented Jan 24, 2022 •

edited

SpaceIm commented Jan 25, 2022 •

edited

[conan-center] Add a hook to check whether installed shared libs are relocatable on Linux & macOS #376

[conan-center] Add a hook to check whether installed shared libs are relocatable on Linux & macOS #376

Comments

SpaceIm commented Jan 24, 2022 • edited

uilianries commented Jan 24, 2022

SpaceIm commented Jan 24, 2022 • edited

SpaceIm commented Jan 25, 2022 • edited

SpaceIm commented Jan 24, 2022 •

edited

SpaceIm commented Jan 24, 2022 •

edited

SpaceIm commented Jan 25, 2022 •

edited