Agent download failure when using hackney_telemetry

[arjan]

Describe the bug

Appsignal extension fails to install properly because the .tar.gz file with the agent does not download.

To Reproduce

Steps to reproduce the behavior:

• Using AppSignal for Elixir package version 1.3.x (tried with 2.x as well, same error)
• Latest Hackney, 1.18.1

mix deps.compile
==> appsignal
Downloading agent release from https://appsignal-agent-releases.global.ssl.fastly.net/20f7d0d/appsignal-x86_64-linux-all-static.tar.gz
AppSignal installation failed: {:error, {:error, :checkout_failure}}

Found a hackney issue that seems related: benoitc/hackney#689

Elixir / OTP versions:

Host information
  Architecture: x86_64-pc-linux-gnu
  Elixir version: 1.14.1
  OTP version: 25
  Operating System: linux
  root user: no
  Container: no

[arjan]

After putting some logging in hackney_pool it looks like this error: benoitc/hackney#615,
however, Hackney is started before the download so not entirely clear yet what is happening.

[arjan]

Figured out what the issue is: we have a custom hackney metrics module configured, hackney_telemetry:

config :hackney, mod_metrics: :hackney_telemetry

When Hackney sends requests, it assumes it can access this telemetry module, but of course this OTP application has not been started on compile time when Appsignal uses hackney.

Not sure what the best solution is here, I guess the telemetry module should be temporarily disabled while appsignal is using hackney during compile?

For now I patch the appsignal install script like this, which also seems to do the trick:

sed -e 's/:application.ensure_all_started(:hackney)/:application.ensure_all_started(:hackney);:application.ensure_all_started(:hackney_telemetry)/' -i deps/appsignal/mix_helpers.exs

[jeffkreeftmeijer]

Hey @arjan! 👋

Thanks for letting us know about this. It seems we’re in a loop here indeed, where :hackney_telemetry isn’t loaded because we’re in compile time, but still used by :hackney, which is explicitly ensured in the mix helpers file.

We could check the hackney configuration and ensure :hackney_telemetry, but I’m not very fond of that idea. Another option would be to drop hackney from mix_helpers, but that seems to be overkill.

For now, I’ll update the title of this issue so its easier to find for other users running into this. I’m of course open to suggestions for fixing this.

[arjan]

Maybe before starting hackney, you could unset the :mod_metrics key?

:application.unset_env(:hackney, :mod_metrics)

As this is only for the agent downloading during compile time, it should not affect anything else..

[unflxw]

Thanks for the suggestion @arjan! We may apply the fix you propose, as this has become a common stumbling block for our users.

For anyone currently affected by this, the problem is partly a configuration issue: a workaround would be to move the line in your config that enables the metrics (config :hackney, mod_metrics: :hackney_telemetry) from config/config.exs to config/runtime.exs.

[unflxw]

Version 2.4.3 of the AppSignal for Elixir integration has been released, which fixes this issue.