macOS healthcheck checks dylib install ID as if it was a dependency

[mx-psi]

Description

When building for macOS, Omnibus includes a healthcheck of dylibs via otool:

omnibus/lib/omnibus/health_check.rb

Lines 289 to 306 in 27c37fc

	def health_check_otool
	current_library = nil
	bad_libs = {}
	good_libs = {}
	read_shared_libs("find #{project.install_dir}/ -type f | egrep '\.(dylib|bundle)$'", "xargs otool -L") do |line|
	case line
	when /^(.+):$/
	current_library = Regexp.last_match[1]
	when /^\s+(.+) \(.+\)$/
	linked = Regexp.last_match[1]
	name = File.basename(linked)
	bad_libs, good_libs = check_for_bad_library(bad_libs, good_libs, current_library, name, linked)
	end
	end
	[bad_libs, good_libs]
	end

As an example (from python-lz4/python-lz4#244), let's assume that the otool -L output is as follows:

❯ otool -L lz4-4.0.0/lz4/.dylibs/liblz4.1.9.3.dylib
lz4-4.0.0/lz4/.dylibs/liblz4.1.9.3.dylib:
	/DLC/lz4/.dylibs/liblz4.1.9.3.dylib (compatibility version 1.0.0, current version 1.9.3)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.0.0)

This roughly does the following:

1. run otool -L output on all dylibs
2. for each dylib, take the library name as the first line from the output (which has a :), in this case lz4-4.0.0/lz4/.dylibs/liblz4.1.9.3.dylib
3. the rest of lines are taken as dependencies of this library, and are passed to check_for_bad_library, which roughly checks that the dependencies refer to paths inside the project (modulo some whitelisting), excluding the library name taken on step (2). In our example, it would check for /DLC/lz4/.dylibs/liblz4.1.9.3.dylib and /usr/lib/libSystem.B.dylib.

This is incorrect in that not all lines of otool -L output are actually dependencies of the library. In particular, the first line can be the install name ID, which usually but not necessarily matches the library path, as explained in matthew-brett/delocate#150 (comment) (continuing with the example above, /DLC/lz4/.dylibs/liblz4.1.9.3.dylib would be the install name ID).

This causes projects that bundle dylibs with install IDs not matching their relative paths to incorrectly fail the healthcheck. The solution is to first run otool -D and exclude this from the libraries checked by check_for_bad_library (see example on delocate library)

Omnibus Version

The code currently on main (as of commit 27c37fc) should reproduce this issue.

Platform Version

This affects macOS builds only.

Replication Case

This can be reproduced with the Python lz4 pip package, version 4.0.0 (again, see python-lz4/python-lz4#244 for many more details). The project where we found this uses a fork of Omnibus and it's hard for me to create MWE, but I hope the detailed information above is enough to reproduce!

Build Output

Healthcheck failure logs

This comes from here, which depending at which point in the future you read, it may be gone from Github.

 The health check failed! Please see above for important information.
            [HealthCheck] E | 2022-03-28T21:44:23+00:00 | The following libraries have unsafe or unmet dependencies:

    --> /opt/datadog-agent//embedded/lib/python3.8/site-packages/lz4/.dylibs/liblz4.1.9.3.dylib
/usr/local/lib/ruby/gems/2.7.0/bundler/gems/omnibus-ruby-012c4078aa84/lib/omnibus/health_check.rb:346:in `block in run!'
  /usr/local/lib/ruby/gems/2.7.0/bundler/gems/omnibus-ruby-012c4078aa84/lib/omnibus/instrumentation.rb:23:in `measure'

            [HealthCheck] E | 2022-03-28T21:44:23+00:00 | The following binaries have unsafe or unmet dependencies:
  /usr/local/lib/ruby/gems/2.7.0/bundler/gems/omnibus-ruby-012c4078aa84/lib/omnibus/health_check.rb:246:in `run!'

  /usr/local/lib/ruby/gems/2.7.0/bundler/gems/omnibus-ruby-012c4078aa84/lib/omnibus/health_check.rb:214:in `run!'
            [HealthCheck] E | 2022-03-28T21:44:23+00:00 | The following libraries cannot be guaranteed to be on target systems:
  /usr/local/lib/ruby/gems/2.7.0/bundler/gems/omnibus-ruby-012c4078aa84/lib/omnibus/project.rb:1416:in `build'
    --> /DLC/lz4/.dylibs/liblz4.1.9.3.dylib
  /usr/local/lib/ruby/gems/2.7.0/bundler/gems/omnibus-ruby-012c4078aa84/lib/omnibus/cli.rb:91:in `build'
  /usr/local/lib/ruby/gems/2.7.0/gems/thor-0.20.3/lib/thor/command.rb:27:in `run'
  /usr/local/lib/ruby/gems/2.7.0/gems/thor-0.20.3/lib/thor/invocation.rb:126:in `invoke_command'
  /usr/local/lib/ruby/gems/2.7.0/gems/thor-0.20.3/lib/thor.rb:387:in `dispatch'
  /usr/local/lib/ruby/gems/2.7.0/bundler/gems/omnibus-ruby-012c4078aa84/lib/omnibus/cli/base.rb:33:in `dispatch'
  /usr/local/lib/ruby/gems/2.7.0/gems/thor-0.20.3/lib/thor/base.rb:466:in `start'
  /usr/local/lib/ruby/gems/2.7.0/bundler/gems/omnibus-ruby-012c4078aa84/lib/omnibus/cli.rb:42:in `execute!'
  /usr/local/lib/ruby/gems/2.7.0/bundler/gems/omnibus-ruby-012c4078aa84/bin/omnibus:16:in `<top (required)>'
  /usr/local/lib/ruby/gems/2.7.0/bin/omnibus:25:in `load'
  /usr/local/lib/ruby/gems/2.7.0/bin/omnibus:25:in `<top (required)>'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/cli/exec.rb:58:in `load'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/cli/exec.rb:58:in `kernel_load'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/cli/exec.rb:23:in `run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/cli.rb:483:in `exec'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/cli.rb:31:in `dispatch'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/cli.rb:25:in `start'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/exe/bundle:48:in `block in <top (required)>'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/lib/bundler/friendly_errors.rb:103:in `with_friendly_errors'
  /usr/local/lib/ruby/gems/2.7.0/gems/bundler-2.3.9/exe/bundle:36:in `<top (required)>'
  /usr/local/lib/ruby/gems/2.7.0/bin/bundle:23:in `load'
  /usr/local/lib/ruby/gems/2.7.0/bin/bundle:23:in `<main>'

            [HealthCheck] E | 2022-03-28T21:44:23+00:00 | The precise failures were:
    --> /opt/datadog-agent//embedded/lib/python3.8/site-packages/lz4/.dylibs/liblz4.1.9.3.dylib
    DEPENDS ON: liblz4.1.9.3.dylib
      COUNT: 1
      PROVIDED BY: /DLC/lz4/.dylibs/liblz4.1.9.3.dylib
      FAILED BECAUSE: Unsafe dependency

            [HealthCheck] I | 2022-03-28T21:44:23+00:00 | Health check time: 1.7175s

[mx-psi]

On our fork we fixed this in DataDog/pull/168