Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipework Re-assigning IP to existing container kills host / Segfault #7627

Closed
46labs opened this issue Aug 19, 2014 · 3 comments
Closed

Pipework Re-assigning IP to existing container kills host / Segfault #7627

46labs opened this issue Aug 19, 2014 · 3 comments

Comments

@46labs
Copy link

46labs commented Aug 19, 2014

Environment
Ubuntu 14.04 LTS (GNU/Linux 3.13.0-30-generic x86_64)
Docker 1.1.1
Latest Pipework

Method to replicate issue:

Bring up a container with an ip using pipework:

ie. root@docker:~# pipework eth0 -i eth1 0b3 1.1.1.1/24@1.1.1.254

Then run the same command again after the container is up:

root@docker:~# pipework eth0 -i eth1 0b3 1.1.1.1/24@1.1.1.254
Segmentation fault

It segfaults the container, leaves docker in a disk wait mode and hoses the physical host. The only method to get the host to do anything is to hard boot it. It will not respond to a restart inside of linux.

@46labs
Copy link
Author

46labs commented Aug 19, 2014

Execute pipework twice on the SAME INTERFACE with new IPs or same and segfaults and throws a kernel error.

ifconfig inside the container will hang the container and throw docker into disk wait mode. Impossible to restart docker process. Shutdown even hangs. This requires a power cycle to resolve.

Note: Running pipework twice on DIFFERENT interfaces works just fine as long as you don't rerun pipework on them once they are up.

How to reproduce:
root@docker:# pipework em1 -i eth0 d9a 10.0.0.17/16@10.0.0.1
root@docker:
# pipework em1 -i eth0 d9a 10.0.0.17/16@10.0.0.1
Segmentation fault

Dmesg output after incident:
[ 526.807415] ------------[ cut here ]------------
[ 526.807430] WARNING: CPU: 10 PID: 4798 at /build/buildd/linux-3.13.0/fs/sysfs/dir.c:486 sysfs_warn_dup+0x86/0xa0()
[ 526.807433] sysfs: cannot create duplicate filename '/devices/pci0000:00/0000:00:01.0/0000:02:00.0/net/em1/upper_ph4100eth0'
[ 526.807436] Modules linked in: macvlan ipmi_devintf xt_addrtype xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge aufs iptable_filter ip_tables x_tables gpio_ich coretemp radeon kvm_intel kvm ttm serio_raw drm_kms_helper drm 8021q i7core_edac lpc_ich garp hpwdt hpilo i2c_algo_bit edac_core ipmi_si stp mac_hid acpi_power_meter mrp llc lp parport tcp_htcp hid_generic bnx2x psmouse usbhid hid hpsa mdio libcrc32c
[ 526.807499] CPU: 10 PID: 4798 Comm: ip Tainted: G I 3.13.0-30-generic #55-Ubuntu
[ 526.807502] Hardware name: HP ProLiant BL460c G6, BIOS I24 01/14/2010
[ 526.807505] 0000000000000009 ffff8805fe9db610 ffffffff8171a324 ffff8805fe9db658
[ 526.807513] ffff8805fe9db648 ffffffff810676bd ffff88060e5ac000 ffff88060e5ac000
[ 526.807519] ffff8800360d37e0 0000000000000000 0000000000000001 ffff8805fe9db6a8
[ 526.807526] Call Trace:
[ 526.807539] [] dump_stack+0x45/0x56
[ 526.807547] [] warn_slowpath_common+0x7d/0xa0
[ 526.807552] [] warn_slowpath_fmt+0x4c/0x50
[ 526.807557] [] sysfs_warn_dup+0x86/0xa0
[ 526.807562] [] sysfs_add_one+0x40/0x50
[ 526.807568] [] sysfs_do_create_link_sd.isra.2+0xbf/0x210
[ 526.807573] [] sysfs_create_link+0x25/0x50
[ 526.807581] [] __netdev_adjacent_dev_insert+0x1d9/0x270
[ 526.807588] [] ? addrconf_leave_anycast+0x90/0xc0
[ 526.807594] [] __netdev_adjacent_dev_link_lists+0x2d/0x80
[ 526.807600] [] __netdev_adjacent_dev_link_neighbour+0x71/0xa0
[ 526.807606] [] __netdev_upper_dev_link+0xf0/0x460
[ 526.807614] [] ? raw_notifier_call_chain+0x16/0x20
[ 526.807620] [] ? dev_queue_xmit_nit+0x220/0x220
[ 526.807625] [] netdev_upper_dev_link+0x12/0x20
[ 526.807633] [] macvlan_common_newlink+0x197/0x3a0 [macvlan]
[ 526.807639] [] ? netif_rx+0x190/0x190
[ 526.807646] [] macvlan_newlink+0x1c/0x20 [macvlan]
[ 526.807653] [] rtnl_newlink+0x4f5/0x5d0
[ 526.807658] [] ? rtnl_newlink+0x13e/0x5d0
[ 526.807665] [] rtnetlink_rcv_msg+0x99/0x260
[ 526.807673] [] ? __alloc_skb+0x7e/0x2b0
[ 526.807678] [] ? rtnetlink_rcv+0x30/0x30
[ 526.807688] [] netlink_rcv_skb+0xa9/0xc0
[ 526.807693] [] rtnetlink_rcv+0x28/0x30
[ 526.807699] [] netlink_unicast+0xd5/0x1b0
[ 526.807704] [] netlink_sendmsg+0x2ff/0x740
[ 526.807712] [] sock_sendmsg+0x8b/0xc0
[ 526.807722] [] ? __mem_cgroup_commit_charge+0x156/0x3d0
[ 526.807729] [] ? move_addr_to_kernel.part.16+0x1e/0x60
[ 526.807735] [] ? move_addr_to_kernel+0x21/0x30
[ 526.807741] [] ___sys_sendmsg+0x3c3/0x3d0
[ 526.807749] [] ? __do_page_fault+0x204/0x560
[ 526.807758] [] ? acct_account_cputime+0x1c/0x20
[ 526.807765] [] ? account_user_time+0x8b/0xa0
[ 526.807769] [] __sys_sendmsg+0x42/0x80
[ 526.807773] [] SyS_sendmsg+0x12/0x20
[ 526.807778] [] tracesys+0xe1/0xe6
[ 526.807780] ---[ end trace 178f5d6eac59ae85 ]---
[ 526.807802] ------------[ cut here ]------------
[ 526.807884] kernel BUG at /build/buildd/linux-3.13.0/net/core/dev.c:6385!
[ 526.807988] invalid opcode: 0000 [#1] SMP
[ 526.808064] Modules linked in: macvlan ipmi_devintf xt_addrtype xt_conntrack ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack bridge aufs iptable_filter ip_tables x_tables gpio_ich coretemp radeon kvm_intel kvm ttm serio_raw drm_kms_helper drm 8021q i7core_edac lpc_ich garp hpwdt hpilo i2c_algo_bit edac_core ipmi_si stp mac_hid acpi_power_meter mrp llc lp parport tcp_htcp hid_generic bnx2x psmouse usbhid hid hpsa mdio libcrc32c
[ 526.809074] CPU: 10 PID: 4798 Comm: ip Tainted: G W I 3.13.0-30-generic #55-Ubuntu
[ 526.809198] Hardware name: HP ProLiant BL460c G6, BIOS I24 01/14/2010
[ 526.809299] task: ffff8805ffc0c7d0 ti: ffff8805fe9da000 task.ti: ffff8805fe9da000
[ 526.809413] RIP: 0010:[] [] free_netdev+0xff/0x110
[ 526.809544] RSP: 0018:ffff8805fe9db8d8 EFLAGS: 00010293
[ 526.809632] RAX: 0000000000000001 RBX: ffff88060e5ae018 RCX: 0000000000000101
[ 526.809771] RDX: 0000000000000001 RSI: 000000000be00bde RDI: 0000000000000286
[ 526.809878] RBP: ffff8805fe9db8f0 R08: fffffee0ffffffc0 R09: ffffffc0ffffffc0
[ 526.809984] R10: 0001d778fffffee0 R11: 0001d778fffffee0 R12: ffff88060e5ae060
[ 526.810091] R13: ffff88060e5ae000 R14: ffff8805fe9db900 R15: ffffffffa0350000
[ 526.810199] FS: 00007f12e4321740(0000) GS:ffff88061faa0000(0000) knlGS:0000000000000000
[ 526.810321] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 526.810407] CR2: 00007fffb6cc63a0 CR3: 000000060b81f000 CR4: 00000000000007e0
[ 526.810514] Stack:
[ 526.810548] 00000000ffffffef ffffffff81cda040 ffff8805fe9db920 ffff8805fe9dbae8
[ 526.810686] ffffffff81630e38 0000000000000000 ffff88060b377c54 0000000000000000
[ 526.810864] ffffffff816309fe ffffffff81cda040 ffff88060b377c10 ffff88060e5ae000
[ 526.811002] Call Trace:
[ 526.811046] [] rtnl_newlink+0x578/0x5d0
[ 526.811130] [] ? rtnl_newlink+0x13e/0x5d0
[ 526.811219] [] rtnetlink_rcv_msg+0x99/0x260
[ 526.811311] [] ? __alloc_skb+0x7e/0x2b0
[ 526.811396] [] ? rtnetlink_rcv+0x30/0x30
[ 526.811482] [] netlink_rcv_skb+0xa9/0xc0
[ 526.811568] [] rtnetlink_rcv+0x28/0x30
[ 526.811653] [] netlink_unicast+0xd5/0x1b0
[ 526.811740] [] netlink_sendmsg+0x2ff/0x740
[ 526.811845] [] sock_sendmsg+0x8b/0xc0
[ 526.811961] [] ? __mem_cgroup_commit_charge+0x156/0x3d0
[ 526.812068] [] ? move_addr_to_kernel.part.16+0x1e/0x60
[ 526.812172] [] ? move_addr_to_kernel+0x21/0x30
[ 526.812267] [] ___sys_sendmsg+0x3c3/0x3d0
[ 526.812355] [] ? __do_page_fault+0x204/0x560
[ 526.812448] [] ? acct_account_cputime+0x1c/0x20
[ 526.812543] [] ? account_user_time+0x8b/0xa0
[ 526.812636] [] __sys_sendmsg+0x42/0x80
[ 526.812720] [] SyS_sendmsg+0x12/0x20
[ 526.812803] [] tracesys+0xe1/0xe6
[ 526.816823] Code: b0 e6 ff 5b 41 5c 41 5d 5d c3 66 90 e8 bb 50 b6 ff e9 53 ff ff ff 66 0f 1f 44 00 00 4c 89 ef e8 b8 fe ff ff 5b 41 5c 41 5d 5d c3 <0f> 0b 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66
[ 526.825483] RIP [] free_netdev+0xff/0x110
[ 526.829643] RSP
[ 526.833724] ---[ end trace 178f5d6eac59ae86 ]---

@46labs
Copy link
Author

46labs commented Aug 19, 2014

root@docker:~# lspci
00:00.0 Host bridge: Intel Corporation 5500 I/O Hub to ESI Port (rev 13)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)
00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 13)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)
00:08.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 (rev 13)
00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)
00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)
00:0d.0 Host bridge: Intel Corporation Device 343a (rev 13)
00:0d.1 Host bridge: Intel Corporation Device 343b (rev 13)
00:0d.2 Host bridge: Intel Corporation Device 343c (rev 13)
00:0d.3 Host bridge: Intel Corporation Device 343d (rev 13)
00:0d.4 Host bridge: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 (rev 13)
00:0d.5 Host bridge: Intel Corporation 7500/5520/5500 Physical Layer Port 1 (rev 13)
00:0d.6 Host bridge: Intel Corporation Device 341a (rev 13)
00:0e.0 Host bridge: Intel Corporation Device 341c (rev 13)
00:0e.1 Host bridge: Intel Corporation Device 341d (rev 13)
00:0e.2 Host bridge: Intel Corporation Device 341e (rev 13)
00:0e.3 Host bridge: Intel Corporation Device 341f (rev 13)
00:0e.4 Host bridge: Intel Corporation Device 3439 (rev 13)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)
00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller
01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02)
01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)
01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03)
01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller
01:04.6 IPMI SMIC interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface
02:00.0 Ethernet controller: Broadcom Corporation NetXtreme II BCM57711E 10-Gigabit PCIe
02:00.1 Ethernet controller: Broadcom Corporation NetXtreme II BCM57711E 10-Gigabit PCIe
0c:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)
3e:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05)
3e:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05)
3e:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05)
3e:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05)
3e:02.4 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 1 (rev 05)
3e:02.5 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 1 (rev 05)
3e:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05)
3e:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05)
3e:03.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller RAS Registers (rev 05)
3e:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05)
3e:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05)
3e:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05)
3e:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05)
3e:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05)
3e:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05)
3e:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05)
3e:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05)
3e:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05)
3e:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05)
3e:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05)
3e:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05)
3e:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05)
3f:00.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture Generic Non-Core Registers (rev 05)
3f:00.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QuickPath Architecture System Address Decoder (rev 05)
3f:02.0 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 0 (rev 05)
3f:02.1 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 0 (rev 05)
3f:02.4 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Link 1 (rev 05)
3f:02.5 Host bridge: Intel Corporation Xeon 5500/Core i7 QPI Physical 1 (rev 05)
3f:03.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller (rev 05)
3f:03.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Target Address Decoder (rev 05)
3f:03.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller RAS Registers (rev 05)
3f:03.4 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Test Registers (rev 05)
3f:04.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Control Registers (rev 05)
3f:04.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Address Registers (rev 05)
3f:04.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Rank Registers (rev 05)
3f:04.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 0 Thermal Control Registers (rev 05)
3f:05.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Control Registers (rev 05)
3f:05.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Address Registers (rev 05)
3f:05.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Rank Registers (rev 05)
3f:05.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 1 Thermal Control Registers (rev 05)
3f:06.0 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Control Registers (rev 05)
3f:06.1 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Address Registers (rev 05)
3f:06.2 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Rank Registers (rev 05)
3f:06.3 Host bridge: Intel Corporation Xeon 5500/Core i7 Integrated Memory Controller Channel 2 Thermal Control Registers (rev 05)

@cpuguy83
Copy link
Member

This doesn't really belong here as it's a pipework thing.
Please post to jeptazzo/pipework.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant