malformed header (Excon::Error::ResponseParse) in version 0.108.0

[bird-dog]

Struggled with the version 0.107.0 error this morning for a bit, but now getting the following when running in chef

      Error Log & Stacktrace

      docker_registry[***] (***::install-docker line 26) had an error: Excon::Error::Socket: malformed header (Excon::Error::ResponseParse)


Excon::Error::Socket
----------------------------------------------------
malformed header (Excon::Error::ResponseParse)


Cookbook Trace:
----------------------------------------------------
/var/chef/cache/cookbooks/docker/libraries/docker_registry.rb:30:in `block in <class:DockerRegistry>'

Backtrace
----------------------------------------------------
      /opt/chef/embedded/lib/ruby/gems/2.5.0/gems/excon-0.108.0/lib/excon/response.rb:202:in `parse_headers'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/excon-0.108.0/lib/excon/response.rb:105:in `parse'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/excon-0.108.0/lib/excon/middlewares/response_parser.rb:7:in `response_call'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/docker-api-1.34.2/lib/excon/middlewares/hijack.rb:45:in `response_call'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/excon-0.108.0/lib/excon/connection.rb:467:in `response'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/excon-0.108.0/lib/excon/connection.rb:298:in `request'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/docker-api-1.34.2/lib/docker/connection.rb:40:in `request'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/docker-api-1.34.2/lib/docker/connection.rb:65:in `block (2 levels) in <class:Connection>'
/var/chef/cache/cookbooks/docker/libraries/docker_registry.rb:30:in `block in <class:DockerRegistry>'
(eval):2:in `block in action_login'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:236:in `instance_eval'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:236:in `compile_and_converge_action'
(eval):2:in `action_login'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/provider.rb:182:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource.rb:578:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:70:in `run_action'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block (2 levels) in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `each'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:98:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:94:in `block in execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:114:in `call_iterator_block'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:85:in `step'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:103:in `iterate'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/stepable_iterator.rb:55:in `each_with_index'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/resource_collection/resource_list.rb:92:in `execute_each_resource'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/runner.rb:97:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:720:in `block in converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `catch'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:715:in `converge'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:754:in `converge_and_save'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/client.rb:286:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:303:in `run_with_graceful_exit_option'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:279:in `block in run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/local_mode.rb:44:in `with_server_connectivity'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:261:in `run_chef_client'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application/client.rb:444:in `run_application'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/lib/chef/application.rb:66:in `run'
/opt/chef/embedded/lib/ruby/gems/2.5.0/gems/chef-14.10.9/bin/chef-client:25:in `<top (required)>'
/usr/bin/chef-client:74:in `load'
/usr/bin/chef-client:74:in `<main>'

This is obviously a very old version of ruby we are using

System Info:
------------
chef_version=14.10.9
platform=ubuntu
platform_version=22.04
ruby=ruby 2.5.3p105 (2018-10-18 revision 65156) [x86_64-linux]
program_name=/usr/bin/chef-shell
executable=/opt/chef/bin/chef-shell

I have reverted to version 0.106.0 for now and that seems to be working fine.

[jmelowry]

I'm encountering this error w/ 0.108.0:

Excon::Error::Socket
--------------------
malformed header (Excon::Error::ResponseParse)

Cookbook Trace:
---------------
/var/chef/cache/cookbooks/docker/libraries/docker_image.rb:149:in `block in pull_image'
/var/chef/cache/cookbooks/docker/libraries/docker_base.rb:29:in `with_retries'
/var/chef/cache/cookbooks/docker/libraries/docker_image.rb:147:in `pull_image'
/var/chef/cache/cookbooks/docker/libraries/docker_image.rb:51:in `block in <class:DockerImage>'

[geemus]

Thanks for the heads up. I'm pretty occupied with family stuff over the weekend, but will try to dig in a bit more early next week. In the mean time, I'd recommend locking to v0.106.0 until we can figure out what part of the performance changes might be causing this.

[isay89]

Hello.
I have the same issue with v0.108.0

  Excon::Error::Socket
  --------------------
  malformed header (Excon::Error::ResponseParse)

  Cookbook Trace:
  ---------------
  /var/chef/cache/cookbooks/docker/libraries/docker_image.rb:163:in `block in pull_image'
  /var/chef/cache/cookbooks/docker/libraries/docker_base.rb:29:in `with_retries'
  /var/chef/cache/cookbooks/docker/libraries/docker_image.rb:161:in `pull_image'
  /var/chef/cache/cookbooks/docker/libraries/docker_image.rb:61:in `block in <class:DockerImage>'

but with v0.106.0 everything works fine

[geemus]

Thanks for the extra details. I think for my purposes the backtrace is much more helpful than the cookbook trace though, so if you have those that might be more informative.

Looking at the provided backtrace from the first one, I noticed something that is unusual (from my perspective anyway), which is the "hijack" middleware from docker-api. It looks like that tries to do some of it's own response parsing, distinct from what would normally happen, and that this is where the error is coming from. It's the first time I've seen this. I think you may need to touch base with the folks on that to see if they have more insight, as even if I could figure out fully what they intended, I might not be able to fix it on my side. You can see the code in question here: https://github.com/upserve/docker-api/blob/master/lib/excon/middlewares/hijack.rb.

Also, at least at present my test suite seems to pass fine in v0.108.0, so perhaps they would have something we could use to reproduce this. In any event, I think the next step is probably to reach out on that repo. Sorry I can't be more directly helpful, it's tricky with unfamiliar code and without a good repro case though.

[raghukumarc]

Hello @bird-dog & @isay89 , Would you be able to help me lock the excon version to 0.106.0. I tried using chef_gem resource and forced a 0.106.0 install of excon. Is this the right way to lock or is there a better way to do this?

[bird-dog]

Hello @bird-dog & @isay89 , Would you be able to help me lock the excon version to 0.106.0. I tried using chef_gem resource and forced a 0.106.0 install of excon. Is this the right way to lock or is there a better way to do this?

You will want to add
gem 'excon', '= 0.106.0'
to your metadata.rb

metadata.rb gems are installed at compile time.

If the machine already has a newer version of excon installed you will need to uninstall it with chef_gem
chef_gem 'excon' do
version "0.017.0"
action :remove
end
This is because cookbooks, like the docker cookbook, may just use the newest installed version of excon

This is all from memory so there may be some syntax errors, but hopefully that will get you

[raghukumarc]

Hello @bird-dog & @isay89 , Would you be able to help me lock the excon version to 0.106.0. I tried using chef_gem resource and forced a 0.106.0 install of excon. Is this the right way to lock or is there a better way to do this?

You will want to add gem 'excon', '= 0.106.0' to your metadata.rb

metadata.rb gems are installed at compile time.

If the machine already has a newer version of excon installed you will need to uninstall it with chef_gem chef_gem 'excon' do version "0.017.0" action :remove end This is because cookbooks, like the docker cookbook, may just use the newest installed version of excon

This is all from memory so there may be some syntax errors, but hopefully that will get you

@bird-dog should the chef_gem run with compile_time true option also?

[ilias-devops]

Hi all!
I had the same issue on chef infra client v.15.1.36.
I upgraded to chef infra client v.15.17.4 and updated cookbook to install docker-api v2.2.0.
This helped to solve the issue.

[geemus]

@bird-dog thanks for sharing your experience and helping others on this, especially since this particular use case is outside my experience.

@ilias-devops thanks for the report, hopefully that might also be a fix for other folks that were running into this (since that seems preferable to version locks).

[bird-dog]

Hello @bird-dog & @isay89 , Would you be able to help me lock the excon version to 0.106.0. I tried using chef_gem resource and forced a 0.106.0 install of excon. Is this the right way to lock or is there a better way to do this?

You will want to add gem 'excon', '= 0.106.0' to your metadata.rb
metadata.rb gems are installed at compile time.
If the machine already has a newer version of excon installed you will need to uninstall it with chef_gem chef_gem 'excon' do version "0.017.0" action :remove end This is because cookbooks, like the docker cookbook, may just use the newest installed version of excon
This is all from memory so there may be some syntax errors, but hopefully that will get you

@bird-dog should the chef_gem run with compile_time true option also?

Not necessary, but I don't believe it would hurt anything.