swarm init segmentation fault

[Snorch]

Description
While running "docker swarm init" one of the processes of docker daemon gets SIGSEGV.

Steps to reproduce the issue:

1. Run docker daemon with gdb: "gdb --args dockerd -D -s devicemapper".
2. (gdb) run
3. In other terminal session do: "docker swarm init --advertise-addr $IP"

Describe the results you received:
Docker daemon gets segfault:

DEBU[0406] Assigning addresses for endpoint ingress-endpoint's interface on network ingress 
DEBU[0406] RequestAddress(LocalDefault/10.255.0.0/16, 10.255.0.3, map[]) 
DEBU[0406] Assigning addresses for endpoint ingress-endpoint's interface on network ingress 
Detaching after fork from child process 11405.

Thread 9 "dockerd" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff1a02700 (LWP 11055)]
0x000000000066bf12 in net.networkNumberAndMask ()

Describe the results you expected:
Work fine without segfault.

Output of docker version:

docker version
Client:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.2
 API version:  1.24
 Go version:   go1.6.3
 Git commit:   bb80604
 Built:        
 OS/Arch:      linux/amd64

Output of docker info:

Backing Filesystem: xfs
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 16.91 MB
 Data Space Total: 107.4 GB
 Data Space Available: 95 GB
 Metadata Space Used: 585.7 kB
 Metadata Space Total: 2.147 GB
 Metadata Space Available: 2.147 GB
 Thin Pool Minimum Free Space: 10.74 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.122 (2016-04-09)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge null host
Swarm: active
 NodeID: 29ql6fig3b3xu21k8ysckisyg
 Is Manager: true
 ClusterID: 06j5qk4aln8uqo6c11gnt6pzb
 Managers: 1
 Nodes: 1
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.30.29.163
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.7.7-200.snorch.fc24.x86_64
Operating System: Fedora 24 (Workstation Edition)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.54 GiB
Name: dhcp-10-30-29-163.sw.ru
ID: FOBI:53SZ:3XT7:5E4Q:JJND:QPOY:PQAT:GPCK:IVQ6:L2AO:IZPY:QWKU
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 34
 Goroutines: 119
 System Time: 2016-10-21T15:16:15.941817613+03:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
Fedora 24, docker-1.12.2

gdb with symbols:

#0  0x000000000066dcb2 in net.networkNumberAndMask (n=0x0, ip=..., m=...) at /usr/local/go/src/net/ip.go:433
#1  0x000000000066e002 in net.(*IPNet).String (n=0x0, ~r0="") at /usr/local/go/src/net/ip.go:483
#2  0x00000000004b3e1a in fmt.(*pp).handleMethods (p=0xc8204369c0, verb=118, depth=0, handled=true) at /usr/local/go/src/fmt/print.go:730
#3  0x00000000004b43a9 in fmt.(*pp).printArg (p=0xc8204369c0, arg=..., verb=118, depth=0, wasString=false)
    at /usr/local/go/src/fmt/print.go:806
#4  0x00000000004bb70d in fmt.(*pp).doPrintf (p= []uint8, format="error setting interface %q IPv6 to %v", a= []interface {} = {...})
    at /usr/local/go/src/fmt/print.go:1238
#5  0x00000000004aeeef in fmt.Sprintf (format="error setting interface %q IPv6 to %v", a= []interface {} = {...}, ~r2="")
    at /usr/local/go/src/fmt/print.go:203
#6  0x0000000000d50a52 in github.com/docker/libnetwork/osl.configureInterface (nlh=0xc820b4d8a0, iface=..., i=0xc820436d00, ~r3=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/osl/interface_linux.go:333
#7  0x0000000000d4f25a in github.com/docker/libnetwork/osl.(*networkNamespace).AddInterface (n=0xc8200f7080, srcName="ov-000100-eko1d", 
    dstPrefix="br", options= []github.com/docker/libnetwork/osl.IfaceOption = {...}, ~r3=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/osl/interface_linux.go:297
#8  0x0000000000d82538 in github.com/docker/libnetwork/drivers/overlay.(*network).setupSubnetSandbox (n=0xc820158820, s=0xc820148a00, 
    brName="ov-000100-eko1d", vxlanName="vx-000100-eko1d", ~r3=...)
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_network.go:473
#9  0x0000000000d82e5c in github.com/docker/libnetwork/drivers/overlay.(*network).initSubnetSandbox (n=0xc820158820, s=0xc820148a00, 
    restore=false, ~r2=...) at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_network.go:507
#10 0x0000000000d99756 in github.com/docker/libnetwork/drivers/overlay.(*network).joinSubnetSandbox.func1 ()
    at /go/src/github.com/docker/docker/vendor/src/github.com/docker/libnetwork/drivers/overlay/ov_network.go:229
#11 0x000000000070d584 in sync.(*Once).Do (o=0xc8209139e0, f={void (void)} 0xc820b45658) at /usr/local/go/src/sync/once.go:44
#12 0x0000000000d7fd0a in github.com/docker/libnetwork/drivers/overlay.(*network).joinSubnetSandbox (n=0xc820158820, s=0xc820148a00,

[justincormack]

cc @aboch PTAL

[aboch]

The issue is not related to swarm init. Any operation which triggers the IP address assigment of an interface inside the sandbox would cause this. For example, this will cause it for me: docker run busybox.

[aboch]

Also, that code path does not get triggered for an IPv4 endpoint when the daemon is run outside of gdb. This makes me think there are issues/limitations running the daemon with gdb.

Not sure how updated this is, but in https://golang.org/doc/gdb I read something which seems to support that theory:

GDB does not understand Go programs well. The stack management, threading, and runtime contain
aspects that differ enough from the execution model GDB expects that they can confuse the debugger,
even when the program is compiled with gccgo. As a consequence, although GDB can be useful in
some situations, it is not a reliable debugger for Go programs, particularly heavily concurrent ones.
Moreover, it is not a priority for the Go project to address these issues, which are difficult.

[aboch]

Ok, it looks like this exact problem was already reported in #14173 and @mrjana explained the behavior in #14173 (comment) as expected limitation.

It was decided not to pursue any further action.
I think this PR should be closed as duplicate of #14173.

[mrjana]

From what I remember gdb does not have first class support for go runtime. Add to the fact that it is perfectly valid in go to pass slices, maps, pointers which are nil and go fmt package does the appropriate thing to make those arguments printed to something appropriate(like <nil>).

@Snorch Have you tried delve debugger which is a debugger of choice for go programs? I think this issue can be closed now. Feel free to reopen if you see any issues outside of gdb.

[Snorch]

Thanks for a hint about delve will try it.

Recovering from segfault is a bit surprizing for me =)