Docker SWARM labels on nodes are not recognized after update to docker v17.05.0-ce with docker stack deploy and yaml #33338

al-sabr · 2017-05-22T18:21:29Z

Description

This seems to be a BUG

After upgrading from 17.03 to 17.05 the same yaml file I used is not deployed as expected by docker.
I have 5 nodes on 10 in my swarm which have the following label constaints on each node:

node.labels.arch==arm64
node.labels.isDBReplicate==true
node.labels.hasHDD==true

The last service from the YAML gets deployed on the node number 8 only.
When I update that service to 5 replicas the 5 replicas are deployed on the same node which should not happen. Docker should find the 4 other nodes with the same node labels and spawn one service on each node.

Steps to reproduce the issue:

Copy thi YAML file

version: "3"

services:

  agency: 
    image: arangodb/arangodb
    environment:
      - ARANGO_NO_AUTH=1
    command: arangod --server.endpoint tcp://0.0.0.0:8529 --agency.my-address tcp://{{.Service.Name}}:8529 --server.authentication false --agency.size 1 --agency.activate true --agency.supervision true --log.file /var/log/arangodb3/arangod.log
    volumes:
      - datas:/var/log/arangodb3
    networks:
      - myingress-net
    deploy:
        placement:
            constraints:
                - node.labels.isAgency==true
                - node.labels.arch==arm64
                - node.labels.hasHDD==true
                
  coordinator:
    image: arangodb/arangodb
    environment:
      - ARANGO_NO_AUTH=1
    command: arangod --server.authentication=false --server.endpoint tcp://0.0.0.0:8529 --cluster.my-address tcp://{{.Service.Name}}:8529 --cluster.my-local-info {{.Service.Name}} --cluster.my-role COORDINATOR --cluster.agency-endpoint tcp://agency:8529 --log.file /var/log/arangodb3/arangod.log
    volumes:
      - datas:/var/lib/arangodb3
      - datas:/var/lib/arangodb3-apps
      - datas:/var/log/arangodb3
    networks:
      - myingress-net
    deploy:
        placement:
            constraints:
                - node.labels.arch==arm64
                - node.labels.isCoordinator==true
                - node.labels.hasHDD==true
    ports: ['8529:8529']
    depends_on:
      - agency
      
  cluster:
    image: arangodb/arangodb
    environment:
      - ARANGO_NO_AUTH=1
    command: arangod --server.authentication=false --server.endpoint tcp://0.0.0.0:8529 --cluster.my-address tcp://cluster{{.Task.Slot}}:8529 --cluster.my-local-info cluster{{.Task.Slot}} --cluster.my-role PRIMARY --cluster.agency-endpoint tcp://agency:8529 --log.file /var/log/arangodb3/arangod.log
    volumes:
      - datas:/var/lib/arangodb3
      - datas:/var/log/arangodb3
    networks:
      - myingress-net
    deploy:
        placement:
            constraints:
                - node.labels.arch==arm64
                - node.labels.isDBReplicate==true
                - node.labels.hasHDD==true    
    depends_on:
      - agency
      - coordinator

networks:
    myingress-net:
        external: true
            
volumes:
  datas:
    driver: local
    driver_opts:
        type: volume 
        mountpoint: /mnt/virtual/docker/volumes/arangodb3

set 5 nodes in the cluster with node label constraints like:

node.labels.arch==arm64
node.labels.isDBReplicate==true
node.labels.hasHDD==true

server8
[
    {
        "ID": "jwyju7toczd0p5exe8yqi8k5g",
        "Version": {
            "Index": 2903
        },
        "CreatedAt": "2017-05-21T22:10:42.676208648Z",
        "UpdatedAt": "2017-05-22T09:22:26.81705998Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true"
            },
            "Role": "worker",
            "Availability": "active"

server9
[
    {
        "ID": "90dnan6z170sfq99i87lalugt",
        "Version": {
            "Index": 2902
        },
        "CreatedAt": "2017-05-21T22:10:42.593800182Z",
        "UpdatedAt": "2017-05-22T09:20:54.078629647Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true"
            },
            "Role": "worker",
            "Availability": "active"

server10
[
    {
        "ID": "31xv25iscsrvgbkt1sua7c1k0",
        "Version": {
            "Index": 2901
        },
        "CreatedAt": "2017-05-21T22:10:42.691917737Z",
        "UpdatedAt": "2017-05-22T09:19:54.598566476Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true"
            },
            "Role": "worker",
            "Availability": "active"

server11
[
    {
        "ID": "w1s3l7mpkhic3gzt4nc3r75tq",
        "Version": {
            "Index": 2900
        },
        "CreatedAt": "2017-05-21T22:10:42.573110065Z",
        "UpdatedAt": "2017-05-22T09:13:40.239399757Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true"
            },
            "Role": "worker",
            "Availability": "active"

server12
[
    {
        "ID": "thx1jzja5qx87i7e20sm5wz68",
        "Version": {
            "Index": 2849
        },
        "CreatedAt": "2017-05-21T22:10:42.539686875Z",
        "UpdatedAt": "2017-05-22T08:59:06.582647575Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true"
            },
            "Role": "worker",
            "Availability": "active"

docker stack deploy -c docker-compose.yml arangodb3
docker service scale arangodb3_cluster =5

Describe the results you received:

screenshot

Describe the results you expected:

The scaling should not happen on the same node with 5 instance of the same service but rather the 4 other nodes individually

Additional information you deem important (e.g. issue happens only occasionally):

This problem appeared when I upgraded docker 17.03-ce to 17.05-ce

Output of docker version:
Manager host

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:28:23 2017
 OS/Arch:      linux/arm

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:28:23 2017
 OS/Arch:      linux/arm
 Experimental: false

Output of docker info:
Manager host

Containers: 4
 Running: 4
 Paused: 0
 Stopped: 0
Images: 379
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 520
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: x81ofqv53z0n66vvjrrs8i38s
 Is Manager: true
 ClusterID: vg6cba4lt2zpdhywscrdyevex
 Managers: 1
 Nodes: 12
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 192.168.1.3
 Manager Addresses:
  192.168.1.3:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
Kernel Version: 3.10.104
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 940.9MiB
Name: bambuserver1
ID: 7GHE:CHRG:TDC4:UOTO:3JWM:2ZYU:CHBN:AMIE:W45Y:I5G7:AMSK:ETMY
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Manager host

uname -a
Linux server1 3.10.104 #1 SMP PREEMPT Fri Apr 14 22:52:56 CEST 2017 armv7l GNU/Linux

Worker nodes

uname -a
Linux bambuserver12 3.14.79-109 #1 SMP PREEMPT Thu Mar 16 20:05:25 BRT 2017 aarch64 GNU/Linux

The text was updated successfully, but these errors were encountered:

al-sabr · 2017-05-22T19:38:39Z

I tried with a super simple whoami service on the 5 nodes and it seems to work on different nodes with the scaling

version: "3"

services:

  whoami: 
    image: admiralobvious/whoami-aarch64
    networks:
      - myingress-net
    deploy:
        placement:
            constraints:
                - node.labels.hasHDD==true
                - node.labels.arch==arm64
                - node.labels.isDBReplicate==true
                

networks:
    myingress-net:
        external: true

al-sabr · 2017-05-22T19:48:48Z

Can it be that the command poses problem with the template variables ?????

command: arangod --server.authentication=false --server.endpoint tcp://0.0.0.0:8529 --cluster.my-address tcp://cluster{{.Task.Slot}}:8529 --cluster.my-local-info cluster{{.Task.Slot}} --cluster.my-role PRIMARY --cluster.agency-endpoint tcp://agency:8529 --log.file /var/log/arangodb3/arangod.log

dongluochen · 2017-05-22T22:18:46Z

@gdeverlant I'm not aware of constraint change between 17.03 and 17.05. There could be different reasons. A few things to check are node resource availability (e.g., your task request certain amount of memory that the node doesn't have), node plugin availability, node network/volume availability (the request network doesn't exist on the node), etc.

I noticed in screenshot that some tasks were failing. What's the reason of failing? You can use docker inspect $task_id to view the error. When you do docker service scale arangodb3_cluster=5, do the tasks select server8 right away, or they select other nodes but fail, eventually reschedule to server8? You can use docker service ps arangodb3_cluster to inspect the task history.

If it's not clear, you may simplify your service to test what specs in the YAML start to fail the task spreading result.

al-sabr · 2017-05-23T08:51:56Z

The failings where because of address already in use:

al-sabr · 2017-05-23T09:30:33Z

You cannot explain why a simple cluster of whoami seems to work but a litte bit more complex scenario is failing. Don't worry I don't have any memory issues all devices ahve 2GB ram free unused nothing else is running.

al-sabr · 2017-05-23T15:57:34Z

Ok it is said on docker swarm documentation that ports should be open on each node for swarm to communicate with each node:

https://docs.docker.com/engine/swarm/swarm-tutorial/#open-protocols-and-ports-between-the-hosts

This is my iptables for the 10 nodes

[1] 17:55:32 [SUCCESS] server9:22
Chain INPUT (policy ACCEPT 3 packets, 748 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2       55 49979 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3       23  3131 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789
[2] 17:55:32 [SUCCESS] server5:22
Chain INPUT (policy ACCEPT 406 packets, 64475 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2      140 80210 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3      777  119K ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789
[3] 17:55:33 [SUCCESS] server3:22
Chain INPUT (policy ACCEPT 336 packets, 45995 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2      160  107K ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3      727  110K ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789
[4] 17:55:33 [SUCCESS] server6:22
Chain INPUT (policy ACCEPT 139 packets, 17265 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2      101 87843 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3      272 34837 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789
[5] 17:55:33 [SUCCESS] server10:22
Chain INPUT (policy ACCEPT 293 packets, 39531 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2      157  108K ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3      634 82871 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789
[6] 17:55:33 [SUCCESS] server7:22
Chain INPUT (policy ACCEPT 337 packets, 48218 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2      146 88658 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3      767  122K ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789
[7] 17:55:33 [SUCCESS] server12:22
Chain INPUT (policy ACCEPT 190 packets, 26170 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2      105 87116 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3      382 50518 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789
[8] 17:55:33 [SUCCESS] server4:22
Chain INPUT (policy ACCEPT 342 packets, 49214 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2      131 87617 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3      743  110K ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789
[9] 17:55:33 [SUCCESS] server8:22
Chain INPUT (policy ACCEPT 300 packets, 40488 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2      155  102K ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3      691 97132 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789
[10] 17:55:33 [SUCCESS] server11:22
Chain INPUT (policy ACCEPT 353 packets, 44331 bytes)
num   pkts bytes target     prot opt in     out     source               destination
1        0     0 ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:2377
2      185  153K ACCEPT     tcp  --  eth0   any     anywhere             anywhere             tcp dpt:7946
3      748  118K ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:7946
4        0     0 ACCEPT     udp  --  eth0   any     anywhere             anywhere             udp dpt:4789

dongluochen · 2017-05-23T17:14:48Z

starting container failed address already in use is also reported by #32548 on Docker version 17.03.0-ce. @mavenugo, any known issue on IP address assignment?

@gdeverlant When you do docker service scale arangodb3_cluster=5, do the tasks select server8 right away, or they select other nodes but fail, eventually reschedule to server8? You can use docker service ps arangodb3_cluster to inspect the task history.

al-sabr · 2017-05-23T21:16:57Z

I think that we need to plan a Team Viewer session so that you really believe that is a bug.

dongluochen · 2017-05-23T21:53:15Z

@gdeverlant I don't doubt there is a problem. I'm trying to see what the scheduler's decision on scheduling the tasks. Your input is helpful for us to narrow down the problem.

al-sabr · 2017-05-23T22:32:10Z

It seems also that the templates are not working in docker stack deploy and docker-compose.yml

https://github.com/moby/moby/blob/master/docs/reference/commandline/service_create.md#create-services-using-templates

al-sabr · 2017-05-23T22:32:57Z

This is the error log of the servers :

2017-05-23T22:18:46Z [1] INFO ArangoDB 3.2.devel [linux] 64bit, using VPack 0.1.30, ICU 58.1, V8 5.7.0.0, OpenSSL 1.0.1t  3 May 2016
2017-05-23T22:18:46Z [1] INFO using storage engine mmfiles
2017-05-23T22:18:46Z [1] INFO Starting up with role COORDINATOR
2017-05-23T22:19:46Z [1] INFO {cluster} Fresh start. Persisting new UUID CRDN-8285eaf9-a9ab-47a5-9892-8855733d23a2
2017-05-23T22:19:46Z [1] INFO Waiting for DBservers to show up...
2017-05-23T22:19:46Z [1] INFO Found 2 DBservers.
2017-05-23T22:19:46Z [1] INFO {syscall} file-descriptors (nofiles) hard limit is 1048576, soft limit is 1048576
2017-05-23T22:19:50Z [1] INFO Cluster feature is turned on. Agency version: {"server":"arango","version":"3.2.devel","license":"community"}, Agency endpoints: http+tcp://agency:8529, server id: 'CRDN-8285eaf9-a9ab-47a5-9892-8855733d23a2', internal address: tcp://coordinator:8529, role: COORDINATOR
2017-05-23T22:19:50Z [1] INFO using heartbeat interval value '1000 ms' from agency
2017-05-23T22:19:51Z [1] INFO using endpoint 'http+tcp://0.0.0.0:8529' for non-encrypted requests
2017-05-23T22:19:57Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:19:57Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:19:57Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:19:57Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:19:58Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:19:58Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:20:00Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:20:00Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:20:03Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:20:03Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:20:10Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:20:10Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:20:20Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:20:20Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:20:30Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:20:30Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:20:41Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:20:41Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:20:51Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:20:51Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:21:02Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:02Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:21:12Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:12Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:21:22Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:22Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:21:32Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:32Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:21:42Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:42Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:21:53Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:53Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s5:/_db/_system/_api/collection/s5/count
2017-05-23T22:21:53Z [1] WARNING {queries} slow query: 'FOR doc IN @@collection FILTER doc.`user` == "root" LIMIT 0, 1  RETURN doc', took: 116.288091
2017-05-23T22:21:53Z [1] ERROR In database '_system': Executing task #4 (addDefaultUserSystem: add default root user for system database) failed with exception: ArangoError 1478: could not determine number of documents in collection (while optimizing plan) ArangoError: could not determine number of documents in collection (while optimizing plan)
2017-05-23T22:21:53Z [1] ERROR     at ArangoStatement.execute (/usr/share/arangodb3/js/server/modules/@arangodb/arango-statement.js:81:16)
2017-05-23T22:21:53Z [1] ERROR     at ArangoDatabase._query (/usr/share/arangodb3/js/server/modules/@arangodb/arango-database.js:80:45)
2017-05-23T22:21:53Z [1] ERROR     at SimpleQueryByExample.execute (/usr/share/arangodb3/js/server/modules/@arangodb/simple-query.js:137:42)
2017-05-23T22:21:53Z [1] ERROR     at SimpleQueryByExample.SimpleQuery.toArray (/usr/share/arangodb3/js/common/modules/@arangodb/simple-query-common.js:340:8)
2017-05-23T22:21:53Z [1] ERROR     at ArangoCollection.firstExample (/usr/share/arangodb3/js/server/modules/@arangodb/arango-collection.js:292:71)
2017-05-23T22:21:53Z [1] ERROR     at Object.exports.save (/usr/share/arangodb3/js/server/modules/@arangodb/users.js:136:22)
2017-05-23T22:21:53Z [1] ERROR     at Object.task (/usr/share/arangodb3/js/server/upgrade-database.js:518:21)
2017-05-23T22:21:53Z [1] ERROR     at runTasks (/usr/share/arangodb3/js/server/upgrade-database.js:274:27)
2017-05-23T22:21:53Z [1] ERROR     at upgradeDatabase (/usr/share/arangodb3/js/server/upgrade-database.js:346:16)
2017-05-23T22:21:53Z [1] ERROR     at upgrade (/usr/share/arangodb3/js/server/upgrade-database.js:787:12)
2017-05-23T22:21:53Z [1] ERROR In database '_system': Executing task #4 (addDefaultUserSystem: add default root user for system database) failed. Aborting init procedure.
2017-05-23T22:21:53Z [1] ERROR In database '_system': Please fix the problem and try starting the server again.
2017-05-23T22:21:53Z [1] ERROR upgrade-database.js for cluster script failed!
2017-05-23T22:21:56Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:56Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:21:56Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:56Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:21:57Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:57Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:21:59Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:59Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:22:02Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:22:02Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:22:10Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:22:10Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:22:20Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:22:20Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:22:31Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:22:31Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:22:41Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:22:41Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:22:51Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:22:51Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:23:01Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:23:01Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:23:12Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:23:12Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:23:22Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:23:22Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:23:32Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:23:32Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:23:43Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:23:43Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:23:53Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:23:53Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:23:53Z [1] WARNING {queries} slow query: 'FOR doc IN @@collection  RETURN doc', took: 117.358452
2017-05-23T22:23:53Z [1] ERROR ArangoError: could not determine number of documents in collection (while optimizing plan)
2017-05-23T22:23:53Z [1] ERROR     at ArangoStatement.execute (/usr/share/arangodb3/js/server/modules/@arangodb/arango-statement.js:81:16)
2017-05-23T22:23:53Z [1] ERROR     at ArangoDatabase._query (/usr/share/arangodb3/js/server/modules/@arangodb/arango-database.js:80:45)
2017-05-23T22:23:53Z [1] ERROR     at SimpleQueryAll.execute (/usr/share/arangodb3/js/server/modules/@arangodb/simple-query.js:96:42)
2017-05-23T22:23:53Z [1] ERROR     at SimpleQueryAll.SimpleQuery.hasNext (/usr/share/arangodb3/js/common/modules/@arangodb/simple-query-common.js:388:8)
2017-05-23T22:23:53Z [1] ERROR     at refillCaches (/usr/share/arangodb3/js/server/modules/@arangodb/foxx/manager.js:266:17)
2017-05-23T22:23:53Z [1] ERROR     at Object.initializeFoxx (/usr/share/arangodb3/js/server/modules/@arangodb/foxx/manager.js:1493:3)
2017-05-23T22:23:53Z [1] ERROR     at Object.foxxes (/usr/share/arangodb3/js/server/bootstrap/foxxes.js:64:47)
2017-05-23T22:23:53Z [1] ERROR     at server/bootstrap/cluster-bootstrap.js:57:54
2017-05-23T22:23:53Z [1] ERROR     at server/bootstrap/cluster-bootstrap.js:61:2
2017-05-23T22:23:54Z [1] ERROR JavaScript exception in file '/usr/share/arangodb3/js/server/modules/@arangodb/foxx/queues/index.js' at 108,7: TypeError: Cannot read property 'save' of undefined
2017-05-23T22:23:54Z [1] ERROR !      throw err;
2017-05-23T22:23:54Z [1] ERROR !      ^
2017-05-23T22:23:54Z [1] FATAL {v8} error during execution of JavaScript file 'server/bootstrap/coordinator.js'
2017-05-23T22:26:32Z [1] INFO ArangoDB 3.2.devel [linux] 64bit, using VPack 0.1.30, ICU 58.1, V8 5.7.0.0, OpenSSL 1.0.1t  3 May 2016
2017-05-23T22:26:32Z [1] INFO using storage engine mmfiles
2017-05-23T22:26:32Z [1] INFO Starting up with role COORDINATOR

as you can see the server cannot start because the template variables are not parsed by docker :

2017-05-23T22:21:56Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'
2017-05-23T22:21:56Z [1] ERROR {cluster} ClusterComm::performRequests: got BACKEND_UNAVAILABLE or TIMEOUT from shard:s8:/_db/_system/_api/collection/s8/count
2017-05-23T22:21:56Z [1] ERROR {cluster} cannot create connection to server 'PRMR-91196b2a-0adb-42d5-95b6-5e96d3df32f4' at endpoint 'tcp://cluster{{.Task.Slot}}:8529'

al-sabr · 2017-05-23T22:34:34Z

You can see the output : 'tcp://cluster{{.Task.Slot}}:8529'

dongluochen · 2017-05-23T23:45:37Z

It seems also that the templates are not working in docker stack deploy and docker-compose.yml

@gdeverlant Let's focus on one problem per issue. You may open a separate issue for template.

For your original issue where tasks are not distributed evenly, what's the output from docker service ps arangodb3_cluster when you change scale of service?

al-sabr · 2017-05-24T00:43:50Z

This is what I get :
from 1 to 5 scaled

ID                  NAME                  IMAGE                                       NODE                DESIRED STATE       CURRENT STATE           ERROR                       PORTS
sjuu2u7988bo        arangodb3_cluster.1   server1:5000/meer/arangodb-arm64:3.2   server12       Shutdown            Failed 2 minutes ago    "task: non-zero exit (1)"
sb27hbhvl8b2        arangodb3_cluster.2   server1:5000/meer/arangodb-arm64:3.2   server9        Shutdown            Failed 2 minutes ago    "task: non-zero exit (1)"
d33kj3puvjep        arangodb3_cluster.3   server1:5000/meer/arangodb-arm64:3.2   server8        Running             Running 4 minutes ago
3hxg5msupk6y        arangodb3_cluster.4   server1:5000/meer/arangodb-arm64:3.2   server9        Running             Running 4 minutes ago
ace2077g3duh        arangodb3_cluster.5   server1:5000/meer/arangodb-arm64:3.2   server12       Running             Running 4 minutes ago

dongluochen · 2017-05-24T01:16:29Z

From the output of docker service ps $service, 3 nodes server8, server9, and server12 are selected by scheduler. Are server10 and server11 healthy and available? You can get node status from docker node ls. I'd expect scheduler also selects them for the tasks.

al-sabr · 2017-05-24T08:52:58Z

Correct! the 2 other servers should be server11 and server10 and not twice server9 and server12. The scheduler is not able to find the 2 other nodes with the same label constraints.

docker node inspect server11 server10 server12 server9 server8
[
    {
        "ID": "kk9rz2y2thwwoma608sy1onwl",
        "Version": {
            "Index": 17576
        },
        "CreatedAt": "2017-05-23T21:56:47.647203726Z",
        "UpdatedAt": "2017-05-24T08:42:46.513346271Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true"
            },
            "Role": "worker",
            "Availability": "active"
        },
        "Description": {
            "Hostname": "server11",
            "Platform": {
                "Architecture": "aarch64",
                "OS": "linux"
            },
            "Resources": {
                "NanoCPUs": 4000000000,
                "MemoryBytes": 2070323200
            },
            "Engine": {
                "EngineVersion": "17.05.0-ce",
                "Plugins": [
                    {
                        "Type": "Network",
                        "Name": "bridge"
                    },
                    {
                        "Type": "Network",
                        "Name": "host"
                    },
                    {
                        "Type": "Network",
                        "Name": "macvlan"
                    },
                    {
                        "Type": "Network",
                        "Name": "null"
                    },
                    {
                        "Type": "Network",
                        "Name": "overlay"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local"
                    }
                ]
            }
        },
        "Status": {
            "State": "ready",
            "Addr": "192.168.1.13"
        }
    },
    {
        "ID": "k2zvcwg2s551a17qr9265s80w",
        "Version": {
            "Index": 17576
        },
        "CreatedAt": "2017-05-23T21:54:33.74745149Z",
        "UpdatedAt": "2017-05-24T08:42:46.51251327Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true",
                "test": "test"
            },
            "Role": "worker",
            "Availability": "active"
        },
        "Description": {
            "Hostname": "server10",
            "Platform": {
                "Architecture": "aarch64",
                "OS": "linux"
            },
            "Resources": {
                "NanoCPUs": 4000000000,
                "MemoryBytes": 2070323200
            },
            "Engine": {
                "EngineVersion": "17.05.0-ce",
                "Plugins": [
                    {
                        "Type": "Network",
                        "Name": "bridge"
                    },
                    {
                        "Type": "Network",
                        "Name": "host"
                    },
                    {
                        "Type": "Network",
                        "Name": "macvlan"
                    },
                    {
                        "Type": "Network",
                        "Name": "null"
                    },
                    {
                        "Type": "Network",
                        "Name": "overlay"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local"
                    }
                ]
            }
        },
        "Status": {
            "State": "ready",
            "Addr": "192.168.1.12"
        }
    },
    {
        "ID": "aik6pgft5yf42axbjez8o4l8b",
        "Version": {
            "Index": 17578
        },
        "CreatedAt": "2017-05-23T10:01:51.189942926Z",
        "UpdatedAt": "2017-05-24T08:42:46.752815543Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true"
            },
            "Role": "worker",
            "Availability": "active"
        },
        "Description": {
            "Hostname": "server12",
            "Platform": {
                "Architecture": "aarch64",
                "OS": "linux"
            },
            "Resources": {
                "NanoCPUs": 4000000000,
                "MemoryBytes": 2070323200
            },
            "Engine": {
                "EngineVersion": "17.05.0-ce",
                "Plugins": [
                    {
                        "Type": "Network",
                        "Name": "bridge"
                    },
                    {
                        "Type": "Network",
                        "Name": "host"
                    },
                    {
                        "Type": "Network",
                        "Name": "macvlan"
                    },
                    {
                        "Type": "Network",
                        "Name": "null"
                    },
                    {
                        "Type": "Network",
                        "Name": "overlay"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local-persist"
                    }
                ]
            }
        },
        "Status": {
            "State": "ready",
            "Addr": "192.168.1.14"
        }
    },
    {
        "ID": "g1ck7k5mn3ju0h2ctczzmrbwd",
        "Version": {
            "Index": 17578
        },
        "CreatedAt": "2017-05-23T10:01:51.136193635Z",
        "UpdatedAt": "2017-05-24T08:42:46.753827545Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true"
            },
            "Role": "worker",
            "Availability": "active"
        },
        "Description": {
            "Hostname": "server9",
            "Platform": {
                "Architecture": "aarch64",
                "OS": "linux"
            },
            "Resources": {
                "NanoCPUs": 4000000000,
                "MemoryBytes": 2070323200
            },
            "Engine": {
                "EngineVersion": "17.05.0-ce",
                "Plugins": [
                    {
                        "Type": "Network",
                        "Name": "bridge"
                    },
                    {
                        "Type": "Network",
                        "Name": "host"
                    },
                    {
                        "Type": "Network",
                        "Name": "macvlan"
                    },
                    {
                        "Type": "Network",
                        "Name": "null"
                    },
                    {
                        "Type": "Network",
                        "Name": "overlay"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local-persist"
                    }
                ]
            }
        },
        "Status": {
            "State": "ready",
            "Addr": "192.168.1.11"
        }
    },
    {
        "ID": "xuuhvppw43ak40e1x9pv4sv6q",
        "Version": {
            "Index": 17579
        },
        "CreatedAt": "2017-05-23T21:52:30.941077229Z",
        "UpdatedAt": "2017-05-24T08:42:47.275547136Z",
        "Spec": {
            "Labels": {
                "arch": "arm64",
                "hasHDD": "true",
                "isDBReplicate": "true"
            },
            "Role": "worker",
            "Availability": "active"
        },
        "Description": {
            "Hostname": "server8",
            "Platform": {
                "Architecture": "aarch64",
                "OS": "linux"
            },
            "Resources": {
                "NanoCPUs": 4000000000,
                "MemoryBytes": 2070323200
            },
            "Engine": {
                "EngineVersion": "17.05.0-ce",
                "Plugins": [
                    {
                        "Type": "Network",
                        "Name": "bridge"
                    },
                    {
                        "Type": "Network",
                        "Name": "host"
                    },
                    {
                        "Type": "Network",
                        "Name": "macvlan"
                    },
                    {
                        "Type": "Network",
                        "Name": "null"
                    },
                    {
                        "Type": "Network",
                        "Name": "overlay"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local"
                    },
                    {
                        "Type": "Volume",
                        "Name": "local-persist"
                    }
                ]
            }
        },
        "Status": {
            "State": "ready",
            "Addr": "192.168.1.10"
        }
    }
]

al-sabr · 2017-05-24T09:33:16Z

It works and I think I have found the solution

some of the nodes didn't have the local-persist plugin driver running. Now that I have installed it it works like a charm the scaling.

al-sabr · 2017-05-24T09:38:33Z

Do you think that you can have a look at the other problem with template?

link : #33364

GordonTheTurtle added area/swarm version/17.05 labels May 22, 2017

al-sabr mentioned this issue May 24, 2017

template variables not parsed in docker-compose.yml with docker swarm on docker v17.05.0-ce #33364

Closed

al-sabr closed this as completed May 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker SWARM labels on nodes are not recognized after update to docker v17.05.0-ce with docker stack deploy and yaml #33338

Docker SWARM labels on nodes are not recognized after update to docker v17.05.0-ce with docker stack deploy and yaml #33338

al-sabr commented May 22, 2017 •

edited

al-sabr commented May 22, 2017 •

edited

al-sabr commented May 22, 2017 •

edited

dongluochen commented May 22, 2017 •

edited

al-sabr commented May 23, 2017

al-sabr commented May 23, 2017 •

edited

al-sabr commented May 23, 2017 •

edited

dongluochen commented May 23, 2017

al-sabr commented May 23, 2017

dongluochen commented May 23, 2017

al-sabr commented May 23, 2017

al-sabr commented May 23, 2017 •

edited

al-sabr commented May 23, 2017

dongluochen commented May 23, 2017

al-sabr commented May 24, 2017 •

edited

dongluochen commented May 24, 2017

al-sabr commented May 24, 2017 •

edited

al-sabr commented May 24, 2017

al-sabr commented May 24, 2017

Docker SWARM labels on nodes are not recognized after update to docker v17.05.0-ce with docker stack deploy and yaml #33338

Docker SWARM labels on nodes are not recognized after update to docker v17.05.0-ce with docker stack deploy and yaml #33338

Comments

al-sabr commented May 22, 2017 • edited

al-sabr commented May 22, 2017 • edited

al-sabr commented May 22, 2017 • edited

dongluochen commented May 22, 2017 • edited

al-sabr commented May 23, 2017

al-sabr commented May 23, 2017 • edited

al-sabr commented May 23, 2017 • edited

dongluochen commented May 23, 2017

al-sabr commented May 23, 2017

dongluochen commented May 23, 2017

al-sabr commented May 23, 2017

al-sabr commented May 23, 2017 • edited

al-sabr commented May 23, 2017

dongluochen commented May 23, 2017

al-sabr commented May 24, 2017 • edited

dongluochen commented May 24, 2017

al-sabr commented May 24, 2017 • edited

al-sabr commented May 24, 2017

al-sabr commented May 24, 2017

al-sabr commented May 22, 2017 •

edited

al-sabr commented May 22, 2017 •

edited

al-sabr commented May 22, 2017 •

edited

dongluochen commented May 22, 2017 •

edited

al-sabr commented May 23, 2017 •

edited

al-sabr commented May 23, 2017 •

edited

al-sabr commented May 23, 2017 •

edited

al-sabr commented May 24, 2017 •

edited

al-sabr commented May 24, 2017 •

edited