Add ability to handle registry downtimes via a 503 HTTP error code

[kencochrane]

Right now if the registry wants to have a scheduled downtime for routine maintenance there isn't a good way to notify someone using the docker engine. Two things we can do to make this better.

1. Allow some why to display a message to the user ahead of time, letting them know when the maintenance will be, giving them a warning before it happens.
2. When the registry goes offline, it will start serving 503 HTTP status codes, where ever we interact with the registry API's the engine should be able to handle this case, and display the message that comes back with the response to the user, letting them know what is going on. Right now this just give them an error, with no message and it isn't very helpful.

1 is less important, but I figured if we are going to talk about this, lets see what the best possible solution would be, and strive for that.

Open for other suggestions on how to make this better.

[dmp42]

Is this registry.hub or registry? Now, this is confusing :-)

[kencochrane]

@dmp42 I think it should work for either.

[dmp42]

registry never ever goes down :-)

[erikh]

I’m working on #2 currently, but without the message component. Basically says, “your registry appears to be down” when encountering any 5xx code. Is that a workable solution for now, or do you require the message as well?

On Nov 12, 2014, at 7:12 AM, Ken Cochrane notifications@github.com wrote:

Right now if the registry wants to have a scheduled downtime for routine maintenance there isn't a good way to notify someone using the docker engine. Two things we can do to make this better.

Allow some why to display a message to the user ahead of time, letting them know when the maintenance will be, giving them a warning before it happens.

When the registry goes offline, it will start serving 503 HTTP status codes, where ever we interact with the registry API's the engine should be able to handle this case, and display the message that comes back with the response to the user, letting them know what is going on. Right now this just give them an error, with no message and it isn't very helpful.

1 is less important, but I figured if we are going to talk about this, lets see what the best possible solution would be, and strive for that.

Open for other suggestions on how to make this better.

—
Reply to this email directly or view it on GitHub #9120.

[rogaha]

+1

[kencochrane]

@erikh It would be nice to have the message as well, so that we can give you more then the generic message. For example "We are down for routine maintenance, be back in 5 minutes". Maybe default to the generic message if there isn't one in the body of the reply?

[thaJeztah]

I think that mentioning the status page in the message would be a nice addition; http://status.docker.com

[erikh]

I’ve interpreted that this has less to do with the hub, and more to do with registries, which may be private.

-Erik

On Nov 14, 2014, at 1:47 PM, Sebastiaan van Stijn notifications@github.com wrote:

I think that mentioning the status page in the message would be a nice addition; http://status.docker.com http://status.docker.com/
—
Reply to this email directly or view it on GitHub #9120 (comment).

[thaJeztah]

Ah, yes. Visiting that page won't be very informative for your private registry :)

[kencochrane]

Ideally, if the message was just pulled from the response from the server, then the registry sending the 503 can put what ever it wants for a message. On the hub, we could link to status.docker.com, and when it is a private registry, they could point to what ever they want. But I don't think putting the status.docker.com address in there by default is a good idea.

Lets keep it as generic and simple as possible so it works for all cases.

[jessfraz]

um whose issue is this @dmcgowan @dmp42 registry? or hub?

[thaJeztah]

I'm thinking;

• Docker needs to be able to display the error message (see Docker client does not handle HTTP 401 responses properly #11016)
• A feature request on https://github.com/docker/distribution to allow setting custom error messages
• Docker Hub Registry creates an internal ticket to track progress on those
• This ticket can be closed, because there's nothing Docker can do, apart from the first bullet.

[dmp42]

Let's make this actionable: when docker gets a 50x error code from a v2 registry, it needs to output the (body) message to the user.

[thaJeztah]

@dmp42 agreed; and I think that's what #11016 is about, so should this be closed as dup?

[dmp42]

Related - but I would keep both.

[ewindisch]

Relatedly, we're looking for a similar feature for EOL tags on official images so we can inform users that that image is no longer supported and no long receives security updates.

[kencochrane]

I think the key thing to take out of this, is that it would be nice to be able to display the message to the client if it isn't an expected status code (200, 201, etc)

[shin-]

FWIW, having a similar issue with a new feature we want to enable on the Hub with server-side error messages. HTTP status other than 200 will result in an error message akin to FATA[0002] HTTP code: 400.

[kencochrane]

@jfrazelle How about this solution.

If there is a status code that is higher then 400, then look to see if it is a JSON response, and if so, take the "message" value and display that to the user. If neither JSON, or a message attribute in the JSON, then display what ever you do now. It is important to note, that there might be other attributes besides the message, but feel free to ignore the other attributes.

Example are some example messages

Message #1

{
 "message": "There was an error, please try again"
}

Message #2

{
  "message": "The repo myname/myrepo has been moved, please try myname/myotherrepo",
  "location": "myname/myotherrepo"
}

Message #3

{
 "message": "We are currently down for maintenance please try again in a few minutes for more information please go to https://status.docker.com"
}

@shin- @dmp42

[jessfraz]

I see no problem with this as long as it's okay with registry folks :) also ping @crosbymichael

[thaJeztah]

Example are some example messages

IIRC, the new registry will return more structure errors (ie, a "code" and a "descriptive text"). Would be nice to take the same approach here?

[stevvooe]

Do we need anything more complicated than a 503 status with a Retry-After header?

If a 503 is encountered, just display something like this:

service unavailable, please try again in <Retry-After> seconds, see status.docker.com for details

I don't see any reason for implementing a more complex protocol. It just doesn't provide enough benefit for the amount of time required.

This is already in the V2 specification: https://github.com/docker/distribution/blob/master/doc/spec/api.md#errors-1.

[jessfraz]

@stevvooe but for depreciating images we need something

[thaJeztah]

@stevvoe agreed it doesn't need to be complicated. My response was to the example, and my concern was introducing something new that would differ from the V2 response. If V2 returns just a header, that's fine with me :)

[kencochrane]

@stevvooe yes, sorry, I keep doing that.. argh.

[jessfraz]

writing this patch tonight sorry got sidetracked should be simple

[jessfraz]

Maybe I am confused but I went to implement and it looks like we already have this https://github.com/docker/docker/blob/master/utils/utils.go#L256 and it is called serveral times in registry/session.go

[jessfraz]

which gets the body anyways https://github.com/docker/docker/blob/master/registry/session.go#L288 so are we looking for something on top of that, because if hub now sends something in the body it will show up in the response

[dmp42]

This is not simple, and requires understanding v1/v2 and describing precisely what the expected behavior should be.
@jfrazelle we can discuss this tomorrow if you want.

[jessfraz]

Cool sounds great thanks

On Sunday, March 22, 2015, Olivier Gambier notifications@github.com wrote:

This is not simple, and requires understanding v1/v2 and describing
precisely what the expected behavior should be.
@jfrazelle https://github.com/jfrazelle we can discuss this tomorrow if
you want.

—
Reply to this email directly or view it on GitHub
#9120 (comment).

[LK4D4]

@aaronlehmann @stevvooe is this still need to be done?

[stevvooe]

@LK4D4 I'd be surprised that this doesn't work. We already have error codes that can propagate to client. Is there anything else we need to do?

cc @dmcgowan @RichardScothern

[RichardScothern]

This is supported in the registry with a health check. 503 and a descriptive error are returned, but the daemon.

A docker pull will produce:

Error: Status 503 trying to pull repository hello-world: "{\"errors\":[{\"code\":\"UNAVAILABLE\",\"message\":\"service unavailable\",\"detail\":\"health check failed: please see /debug/health\"}]}\n"

a docker push is less descriptive:

The push refers to a repository [192.168.203.114:5000/hello-world]
ERRO[0141] Attempting next endpoint for push after error: Get https://192.168.203.114:5000/v2/: http: server gave HTTP response to HTTPS client 
a02596fdd012: Preparing 
a02596fdd012: Retrying in 4 seconds