503 Service Temporarily Unavailable

[brandonweiss]

My nginx-proxy container died because it tried to restart and there was a networking error while fetching the latest version. When I manually booted it back up, I noticed all my sites were returning a 503 Service Temporarily Unavailable. I haven't changed anything in months... did something change recently in nginx-proxy?

[wader]

Know the reason it restarted? what versions of things? can you give more config info logs etc?

[ginkel]

Can you have a look at your (generated) default.conf? Could be that all upstream servers are empty turning this into a duplicate of #438.

[brandonweiss]

@wader I'm pretty sure it's due to a CoreOS update/reboot. Here's a snippet I got: https://gist.github.com/brandonweiss/49476b9b71f2501861f14569e639b63e

So it reboots, and then tries to start nginx, but then either there's a blip somewhere in DigitalOcean or a blip in the Docker service, and then I guess I didn't set it up to retry or something.

[brandonweiss]

@ginkel I can't seem to figure out where the generated file is?

[brandonweiss]

@ginkel Oh, I found it in the issue you linked, I think, but I don't appear to have that file at all...

[wader]

@brandonweiss this is with latest version fo nginx-proxy? can you check if this is due to nginx-proxy/docker-gen#186? you can check cgroup format with docker exec <nginx-proxy-container-id> cat /proc/self/cgroups

[brandonweiss]

Hmm.

core@mallory ~ $ docker exec 2eb0f468a822 cat /proc/self/cgroups
cat: /proc/self/cgroups: No such file or directory

[wader]

Sorry cat /proc/self/cgroup (without "s")

[brandonweiss]

core@mallory ~ $ docker exec 2eb0f468a822 cat /proc/self/cgroup 
9:freezer:/system.slice/docker-2eb0f468a8220e3da9fd681220336e5f13cb85abd3f5194209c6a0bb53392c58.scope
8:perf_event:/system.slice/docker-2eb0f468a8220e3da9fd681220336e5f13cb85abd3f5194209c6a0bb53392c58.scope
7:memory:/system.slice/docker-2eb0f468a8220e3da9fd681220336e5f13cb85abd3f5194209c6a0bb53392c58.scope
6:devices:/system.slice/docker-2eb0f468a8220e3da9fd681220336e5f13cb85abd3f5194209c6a0bb53392c58.scope
5:cpuset:/system.slice/docker-2eb0f468a8220e3da9fd681220336e5f13cb85abd3f5194209c6a0bb53392c58.scope
4:cpu,cpuacct:/system.slice/docker-2eb0f468a8220e3da9fd681220336e5f13cb85abd3f5194209c6a0bb53392c58.scope
3:net_cls,net_prio:/system.slice/docker-2eb0f468a8220e3da9fd681220336e5f13cb85abd3f5194209c6a0bb53392c58.scope
2:blkio:/system.slice/docker-2eb0f468a8220e3da9fd681220336e5f13cb85abd3f5194209c6a0bb53392c58.scope
1:name=systemd:/system.slice/docker-2eb0f468a8220e3da9fd681220336e5f13cb85abd3f5194209c6a0bb53392c58.scope

[wader]

Thanks, that looks like nginx-proxy/docker-gen#186

[brandonweiss]

Ah, thanks for helping me debug it. If it's a dupe of #438 let me know and I'll close it!

[wader]

No problem. Not a dupe, keep open. Probably need to update docker-gen version for nginx-proxy to fix this.

[wader]

@brandonweiss @jwilder pushed a new version of docker-gen fixing this. Can you verify if it works?

[powo]

looks good:

root@b434b8a5c966:~# docker-gen -version
0.7.0
root@b434b8a5c966:~# docker-gen <(echo '{{ json .Docker.CurrentContainerID }}')
""

root@b434b8a5c966:~# wget https://github.com/jwilder/docker-gen/releases/download/0.7.1/docker-gen-linux-amd64-0.7.1.tar.gz
root@b434b8a5c966:~# tar -xzvf docker-gen-linux-amd64-0.7.1.tar.gz
root@b434b8a5c966:~# ./docker-gen -version
0.7.1
root@b434b8a5c966:~# ./docker-gen <(echo '{{ json .Docker.CurrentContainerID }}')
"b434b8a5c966fb091350ddbbdc8c28060cc1f586ac1a39739ea4f0a57f835b0d"

[wader]

@powo 👍

[brandonweiss]

I haven't been able to verify it because I had to divert traffic off my box and don't want to divert it back yet, but if @powo confirmed it works I think it's safe to close.