checkout_timeout after enough http errors

[IceDragon200]

UPDATE 2020-06-17: Just realized the title was incorrect
UPDATE 2020-06-10: It appears to be a hackney issue, replacing HTTPoison.get with hackney.get yields the same problem

Sorry if the title isn't descriptive enough, but the jist of it is:

When HTTPoison (or hackney) has enough errors the pool gets completely exhausted and all subsequent requests return checkout_timeout

Test:

for _ <- 0..30 do 
  IO.inspect HTTPoison.get("http://localhost:3233")
  :ok # the IO.inspect above is to log the errors as they return
end

You may need to run the above at least twice before it starts happening

Example of the result:

{:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}
{:error, %HTTPoison.Error{id: nil, reason: :econnrefused}}
... # truncated for brevity
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}
{:error, %HTTPoison.Error{id: nil, reason: :checkout_timeout}}

All subsequent request will have a checkout_timeout

I'm not sure if this an HTTPoison or hackney issue at the moment

This test was done in a completely new project (no additional configuration) with only httpoison installed (and it's dependencies)

Versions:

httpoison 1.6.2
hackney 1.16.0

[nateless]

We get the same error, it looks like processes aren't killed by timeout and pool is exhausted:

iex(30)> :hackney_pool.get_stats :default [name: :default, max: 50, in_use_count: 50, free_count: 0, queue_count: 0]

We are on Erlang/OTP 22 [erts-10.7.1], Elixir 1.10.3 (compiled with Erlang/OTP 21), httposion 1.7

[neoamos]

Looks like it could have been introduced with version 1.16 of hackney. Several changes were made to the pool in this version:
benoitc/hackney@8bb529e

[wrren]

We're seeing this issue in one of our production apps and have downgraded hackney to version 1.15.2 through an override.

[atomkirk]

and now we're all in a pickle because we can't upgrade to OTP 23 without hackney 1.16…

[overture8]

Has this issue been resolved? I'm getting it using version 1.7.0 in production. Should I just downgrade? Thanks!

[nateless]

@overture8 no, its not. the issue is with hackney (benoitc/hackney#643). I would recommend catching : checkout_timeout and restart the pool.

[whossname]

code for natelesses suggestion for anyone else who isn't too familiar with hackney's API

for _ <- 0..100 do 
  result = HTTPoison.get("http://localhost:3233")

  if  result == {:error, %HTTPoison.Error{reason: :checkout_timeout}} do
    IO.inspect("fixing timeout")
    IO.inspect(result)
    :hackney_pool.stop_pool(:default)
  end

  result
end

[IceDragon200]

Just an update for those who don't really track hackney's issues

benoitc/hackney#661

That PR should resolve the issue, if you're willing to test it out, let everyone know over there if it works for you, the more eyes we have on the issue, the faster we can get it merged and pushed upstream (I think)

[damonvjanis]

It looks like this was included in the 1.17 release, so if you update hackney it will fix the issue.

https://github.com/benoitc/hackney/blob/master/NEWS.md

[bukenbp8]

When using pools, one :timeout produces :checkout_timeout on all following requests (more than 200). Pool size is default (as I understand is 50).

If I set pool: false everything works as expected. It would be nice to use pools for performance reasons...

httpoison 1.8
hackney 1.17