kernel got stuck (concurrent containers)

[rkononov]

Aws ec2 r3.xlarge instance (also could be reproduced on m1.small), HVM based,
CoreOS alpha channel (could be reproduced on ubuntu). 3.15.8+ #2 SMP Thu Aug 28 08:03:21 UTC 2014 x86_64 Intel(R) Xeon(R) CPU E5-2670 v2 @ 2.50GHz GenuineIntel GNU/Linux

Run following bash file(in 30-40 concurrent processes):

#!/bin/bash
touch kill.me
for i in {0..10000}
do
container_name=$RANDOM-$i
echo $container_name
if [ ! -f kill.me ]; then
    echo "Exiting..."
    exit 0
fi
sudo docker run  --name=$container_name  -m='300M' -c 51 -i -a=stdout -a=stderr ruby ruby -e 'File.open("big.file","w") {|f| f.write("0" * 99999) }'
sudo docker stop -t=5 $container_name
sudo docker rm -f $container_name
echo item: $i
done

Callstack:

Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel: INFO: task kworker/u30:8:16904 blocked for more than 120 seconds.
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:       Not tainted 3.15.8+ #2
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel: kworker/u30:8   D ffff8803e0433ec0     0 16904      2 0x00000000
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel: Workqueue: netns cleanup_net
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  ffff8803532a7c48 0000000000000046 ffff8803491fcb60 0000000000013ec0
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  ffff8803532a7fd8 0000000000013ec0 ffff8803cb341920 ffff8803491fcb60
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  ffff880377de3240 ffffffff81aac820 ffff880377de3240 ffff8803491fcb60
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel: Call Trace:
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] schedule+0x29/0x70
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] schedule_preempt_disabled+0xe/0x10
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] __mutex_lock_slowpath+0x1bc/0x1d0
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] mutex_lock+0x23/0x37
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] rtnl_lock+0x15/0x20
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] default_device_exit_batch+0x72/0x160
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] ? finish_wait+0x80/0x80
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] ops_exit_list.isra.1+0x53/0x60
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] cleanup_net+0x110/0x230
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] process_one_work+0x187/0x440
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] worker_thread+0x121/0x3e0
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] ? manage_workers.isra.21+0x300/0x300
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] kthread+0xc9/0xe0
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] ? flush_kthread_worker+0x80/0x80
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] ret_from_fork+0x7c/0xb0
Sep 01 05:33:04 ip-10-41-166-70.ec2.internal kernel:  [] ? flush_kthread_worker+0x80/0x80

[jessfraz]

@rkononov are you still seeing this on the latest version of docker

[rkononov]

@jfrazelle seems like issue has been resolved

[jessfraz]

thanks @rkononov