503 from Docker Registry #60

jellybob · 2014-11-15T10:47:22Z

I'm having some problems getting an image from the public registry deployed with Centurion. I'm running rev aaaf36f and getting 503 errors from the registry when Centurion tries to list the available tags, but when I curl the same URI I get a successful response.

Command: bundle exec centurion -p rabbitmq -e local -a list

Config:

namespace :environment do
  task :common do
    set :image, 'registry.hub.docker.com/dockerfile/rabbitmq'

    host_port 5672, container_port: 5672
    host_port 15672, container_port: 15672
  end

  task :local => :common do
    set_current_environment(:local)

    host 'localhost'
  end
end

Output:

** Invoke environment:local (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:local
** Invoke list (first_time)
** Execute list
** Invoke list:tags (first_time)
** Execute list:tags
GET: "https://registry.hub.docker.com/v1/repositories/dockerfile/rabbitmq/tags"
E, [2014-11-15T10:46:23.198254 #32543] ERROR -- : Couldn't communicate with Registry: #<Excon::Response:0x007f1d46cfdf68 @data={:body=>"<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n\n", :headers=>{"Cache-Control"=>"no-cache", "Connection"=>"close", "Content-Type"=>"text/html"}, :status=>503, :reason_phrase=>"Service Unavailable", :remote_ip=>"162.242.195.83", :local_port=>52296, :local_address=>"10.20.0.11"}, @body="<html><body><h1>503 Service Unavailable</h1>\nNo server is available to handle this request.\n</body></html>\n\n", @headers={"Cache-Control"=>"no-cache", "Connection"=>"close", "Content-Type"=>"text/html"}, @status=503, @remote_ip="162.242.195.83", @local_port=52296, @local_address="10.20.0.11">

** Invoke list:running_containers (first_time)
** Execute list:running_containers
I, [2014-11-15T10:46:23.198515 #32543]  INFO -- : ----- Connecting to Docker on localhost -----

Curl:

$ curl -vvv https://registry.hub.docker.com/v1/repositories/dockerfile/rabbitmq/tags

* Hostname was NOT found in DNS cache
*   Trying 162.242.195.83...
* Connected to registry.hub.docker.com (162.242.195.83) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using ECDHE-RSA-AES128-GCM-SHA256
* Server certificate:
*    subject: OU=GT51013891; OU=See www.rapidssl.com/resources/cps (c)14; OU=Domain Control Validated - RapidSSL(R); CN=*.hub.docker.com
*    start date: 2014-10-25 15:58:21 GMT
*    expire date: 2015-05-23 20:41:27 GMT
*    subjectAltName: registry.hub.docker.com matched
*    issuer: C=US; O=GeoTrust Inc.; CN=RapidSSL SHA256 CA - G3
*    SSL certificate verify ok.
> GET /v1/repositories/dockerfile/rabbitmq/tags HTTP/1.1
> User-Agent: curl/7.35.0
> Host: registry.hub.docker.com
> Accept: */*
> 
< HTTP/1.1 200 OK
* Server nginx is not blacklisted
< Server: nginx
< Date: Sat, 15 Nov 2014 10:41:21 GMT
< Content-Type: application/json
< Transfer-Encoding: chunked
< Connection: close
< Vary: Cookie
< X-Frame-Options: SAMEORIGIN
< Strict-Transport-Security: max-age=31536000
< 
* Closing connection 0
* SSLv3, TLS alert, Client hello (1):

The text was updated successfully, but these errors were encountered:

md5 · 2014-11-15T15:49:49Z

This looks like a transient error on the Docker Hub side, not an issue with centurion.

jellybob · 2014-11-15T17:15:43Z

I'm not convinced - every request I make with curl works, and every request
made by Centurion doesn't.
On 15 Nov 2014 15:49, "Mike Dillon" notifications@github.com wrote:

This looks like a transient error on the Docker Hub side, not an issue
with centurion.

—
Reply to this email directly or view it on GitHub
#60 (comment).

md5 · 2014-11-16T02:03:44Z

You're right. I was able to reproduce it. I first ran your centurion command with EXCON_DEBUG=1 to see what it was sending in the HTTP payload. My first thought was that it was SSL cipher related, but then I noticed a Host: registry.hub.docker.com:443. So I tried this:

curl -i -vvv -H 'Host: registry.hub.docker.com:443' https://registry.hub.docker.com/v1/repositories/dockerfile/rabbitmq/tags

That gave me the 503 error. Since the Host header is allowed to have a port number in it, this actually seems to be a Docker Registry bug.

relistan · 2014-11-18T00:34:50Z

Hey guys, thanks for the excellent debugging. I'll take a look at what we can tell Excon to do there. Interestingly, we deploy from Docker Hub all the time and haven't seen this up to now. Perhaps a new change.

relistan · 2014-11-18T01:09:21Z

Give 969fccd or newer a try, should fix the issue with a workaround that sets the Host header manually.

md5 · 2014-11-18T03:09:45Z

I've submitted a support request to Docker:
https://support.docker.com/hc/en-us/requests/2246

jellybob · 2014-11-18T09:03:40Z

master had a syntax error in lib/tasks/deploy.rake (See #61 for a fix), but even after adding that in I now get this error:

$ bundle exec centurion -p rabbitmq -e local
** Invoke environment:local (first_time)
** Invoke environment:common (first_time)
** Execute environment:common
** Execute environment:local
** Invoke list (first_time)
** Execute list
** Invoke list:tags (first_time)
** Execute list:tags
GET: "https://registry.hub.docker.com/v1/repositories/dockerfile/rabbitmq/tags"
E, [2014-11-18T09:01:47.676494 #5905] ERROR -- : Couldn't communicate with Registry: undefined method `[]' for nil:NilClass

** Invoke list:running_containers (first_time)
** Execute list:running_containers
I, [2014-11-18T09:01:47.677115 #5905]  INFO -- : ----- Connecting to Docker on localhost -----

jellybob · 2014-11-18T09:07:28Z

Digging into this further it looks like Centurion::DockerRegistry#is_official_registry? isn't picking up that the API response needs normalizing into the correct format. I'll see if I can work out why.

jellybob · 2014-11-18T09:13:35Z

My (now somewhat inaccurately named) pull request now includes an additional changeset which removes the repository name check and just checks if the base_uri matches the official repository. That seems to fix this for me, but might cause problems elsewhere, since I couldn't see any explanation of why that check was there.

relistan · 2014-11-20T17:29:59Z

So I'm not sure what changed, but it appears that the correct fix for the Host header is here: excon/excon#357 . We can tell Excon to omit_default_port. In the meantime it appears that the PR I merged has broken one of the tests that was making sure that image names can contain machine names. I'm going to spend some time on that before closing this issue.

Thanks @md5 for opening the issue with Docker Hub and both you and @jellybob for the debugging.

relistan · 2014-11-20T22:56:51Z

This is all sorted out and the tests are fixed. Merged and released in Centurion v1.3.0. Thanks!

relistan closed this as completed Nov 20, 2014

andyshinn mentioned this issue Dec 15, 2014

How to specify image from Docker Hub? #76

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

503 from Docker Registry #60

503 from Docker Registry #60

jellybob commented Nov 15, 2014

md5 commented Nov 15, 2014

jellybob commented Nov 15, 2014

md5 commented Nov 16, 2014

relistan commented Nov 18, 2014

relistan commented Nov 18, 2014

md5 commented Nov 18, 2014

jellybob commented Nov 18, 2014

jellybob commented Nov 18, 2014

jellybob commented Nov 18, 2014

relistan commented Nov 20, 2014

relistan commented Nov 20, 2014

503 from Docker Registry #60

503 from Docker Registry #60

Comments

jellybob commented Nov 15, 2014

md5 commented Nov 15, 2014

jellybob commented Nov 15, 2014

md5 commented Nov 16, 2014

relistan commented Nov 18, 2014

relistan commented Nov 18, 2014

md5 commented Nov 18, 2014

jellybob commented Nov 18, 2014

jellybob commented Nov 18, 2014

jellybob commented Nov 18, 2014

relistan commented Nov 20, 2014

relistan commented Nov 20, 2014