Add new partitioner script, which can do job on first boot #1239

avnik · 2016-03-07T17:36:46Z

Installs cleanly on a fresh build of most recent master branch
Upgrades cleanly from the most recent release
Updates documentation relevant to the changes
Rebases cleanly onto the latest master

Also refactor mantl-storage-setup a bit

avnik · 2016-03-31T13:57:31Z

Add docker-io's docker support from @stevendborrelli

stevendborrelli · 2016-03-31T14:45:19Z

roles/docker/files/docker.repo

@@ -0,0 +1,6 @@
+[dockerrepo]


There are 2 docker repos in the PR.

Correct, I'll remove mine

This has been done.

stevendborrelli · 2016-03-31T15:21:42Z

Documentation needs to be updated before this can go in.

Still need to be included/linked from main doc

langston-barrett · 2016-04-06T19:14:45Z

We should update the Packer image after this is merged.

langston-barrett · 2016-04-06T19:29:24Z

roles/lvm/files/mantl-storage-setup.rst

+Order of execution
+------------------
+
+All configuration rules reside in ``/etc/mantl/filesystems.d``, and processed


Maybe we could include an example configuration? Or include roles/lvm/templates/mantl-volume-group.conf.j2 is an example?

langston-barrett · 2016-04-06T20:08:35Z

@avnik outlined these testing criteria:

clean install
fresh install in both overlay+xfs and devicemapper
upgrade from 1.0.3 with devicemapper
downgrade docker-io's docker to RH one
upgrade from current master in both overlay+xfs and devicemapper

ryane · 2016-04-12T12:58:32Z

on a clean install on AWS, I get the following error on the ansible run:

TASK: [docker | enable docker] ************************************************
failed: [resching-aws-worker-04] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

failed: [resching-aws-worker-03] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

failed: [resching-aws-worker-02] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

failed: [resching-aws-worker-01] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

failed: [resching-aws-control-01] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

failed: [resching-aws-edge-01] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

failed: [resching-aws-control-02] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

failed: [resching-aws-edge-02] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.

failed: [resching-aws-control-03] => {"failed": true}
msg: Job for docker.service failed because a configured resource limit was exceeded. See "systemctl status docker.service" and "journalctl -xe" for details.


FATAL: all hosts have already failed -- aborting

systemctl status docker.service:

[centos@ip-10-1-1-9 ~]$ sudo systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─10-options.conf, 12-network-options.conf, 14-storage-options.conf, 20-ExecStart.conf
   Active: failed (Result: resources)
     Docs: https://docs.docker.com

Apr 12 12:05:27 ip-10-1-1-9.ec2.internal systemd[1]: Failed to load environment files: No such file or directory
Apr 12 12:05:27 ip-10-1-1-9.ec2.internal systemd[1]: docker.service failed to run 'start' task: No such file or directory
Apr 12 12:05:27 ip-10-1-1-9.ec2.internal systemd[1]: Failed to start Docker Application Container Engine.
Apr 12 12:05:27 ip-10-1-1-9.ec2.internal systemd[1]: Unit docker.service entered failed state.
Apr 12 12:05:27 ip-10-1-1-9.ec2.internal systemd[1]: docker.service failed.
Apr 12 12:05:27 ip-10-1-1-9.ec2.internal systemd[1]: Starting Docker Application Container Engine...

any idea? @avnik @stevendborrelli

ryane · 2016-04-12T13:40:07Z

same thing on an upgrade from 1.0.3

This package cleans up docker containers, images and volues by adding systemd timers tht run commands like docker rm

avnik · 2016-04-18T15:42:21Z

@ryane I guess it because we not update package with mantl-storage-setup, I'll do this atm (because I think code of partitioner enough stable).
Pass debug_storage_setup=True to ansible-playbook is another option

ryane · 2016-04-27T22:02:50Z

roles/docker/defaults/main.yml

+  - "--dns {{ private_ipv4 }}"
+  - "--log-driver={{ docker_log_driver }}"
+  - "{% if docker_selinux_enabled %}--selinux-enabled {% endif %}"
+  - "{% if kube_build is defined %}--dns-search {{ cluster_name }}{% endif %}"


is kube_build ever set anywhere?

ryane · 2016-04-27T22:57:39Z

successful clean build on GCE. However, I am seeing some unexpected docker errors. I restarted the nginx-consul service on a node and received the following:

Apr 27 22:54:11 resching-gce-control-03 systemd[1]: Stopping nginx-consul...
Apr 27 22:54:11 resching-gce-control-03 docker[2012]: nginx-consul
Apr 27 22:54:11 resching-gce-control-03 docker[7983]: Error response from daemon: Unable to remove filesystem for 566be3da7914736c38e32db59f2a04599c8fe76191fee610d7a67e71908f9847: remove /var/lib/docker/conta
iners/566be3da7914736c38e32db59f2a04599c8fe76191fee610d7a67e71908f9847/shm: device or resource busy
Apr 27 22:54:11 resching-gce-control-03 systemd[1]: nginx-consul.service: main process exited, code=exited, status=137/n/a
Apr 27 22:54:11 resching-gce-control-03 systemd[1]: Unit nginx-consul.service entered failed state.
Apr 27 22:54:11 resching-gce-control-03 systemd[1]: nginx-consul.service failed.
Apr 27 22:54:11 resching-gce-control-03 systemd[1]: Starting nginx-consul...
Apr 27 22:54:11 resching-gce-control-03 docker[2024]: Error response from daemon: Unable to remove filesystem for 566be3da7914736c38e32db59f2a04599c8fe76191fee610d7a67e71908f9847: remove /var/lib/docker/conta
iners/566be3da7914736c38e32db59f2a04599c8fe76191fee610d7a67e71908f9847/shm: device or resource busy
Apr 27 22:54:11 resching-gce-control-03 docker[2028]: 1.2: Pulling from ciscocloud/nginx-consul
...
Apr 27 22:55:33 resching-gce-control-03 docker[2875]: Status: Image is up to date for ciscocloud/nginx-consul:1.2
Apr 27 22:55:33 resching-gce-control-03 systemd[1]: Started nginx-consul.
Apr 27 22:55:33 resching-gce-control-03 docker[2880]: /usr/bin/docker: Error response from daemon: Conflict. The name "/nginx-consul" is already in use by container 566be3da7914736c38e32db59f2a04599c8fe76191fee610d7a67e71908f9847. You have to remove (or rename) that container to be able to reuse that name..
Apr 27 22:55:33 resching-gce-control-03 docker[2880]: See '/usr/bin/docker run --help'.

The container is marked as Dead and cannot be removed:

566be3da7914        ciscocloud/nginx-consul:1.2                           "/scripts/launch.sh"     31 minutes ago      Dead                                                                                                                          nginx-consul

$ sudo docker rm 566be3da7914
Error response from daemon: Unable to remove filesystem for 566be3da7914736c38e32db59f2a04599c8fe76191fee610d7a67e71908f9847: remove /var/lib/docker/containers/566be3da7914736c38e32db59f2a04599c8fe76191fee610d7a67e71908f9847/shm: device or resource busy

ryane · 2016-04-28T15:01:01Z

the issue doesn't seem to be directly related to the code in this pr but just may be a bug with the current version of docker. There are some similar issues open:

After discussing, we will merge this and workaround the immediate problem in #1390. We will open new issues for any other that may crop up.

avnik added enhancement help wanted need feedback labels Mar 7, 2016

avnik force-pushed the feature/custom-partitioner branch from 27d5b9e to e2d302b Compare March 10, 2016 18:18

avnik changed the title ~~DON'T MERGE: Add new partitioner script, which can do job on first boot~~ Add new partitioner script, which can do job on first boot Mar 10, 2016

avnik force-pushed the feature/custom-partitioner branch from e2d302b to 03075d0 Compare March 11, 2016 14:09

stevendborrelli added this to the 1.1 milestone Mar 14, 2016

stevendborrelli and others added 23 commits March 15, 2016 14:36

Add docker options template

c721871

Update docker options to be a list

3f3db8c

Add network docker options systemd drop-in

55dd10b

Add drop-in for docker ExecStart

12c4164

Add docker yum repo file

f6a7ff6

Update to use mantl-storage-setup

412d8fa

Add docker storage template

a920f7f

Add mantl volume configuration

2359ba4

Add docker storage setup

a2b1d20

Refactor storage to use mantl-storage-setup

9bd8f07

Update comments

3609656

Save VG name as fact

67a4cc2

Add storage options to docker

6289790

Add docker storage drop in

af00466

Merge branch 'master' into feature/docker-refactor

924b941

Add new partitioner script, which can do job on first boot

f692f4d

Fix boot order

16e205c

Add rudimentary config writting support

b68eb8f

Also refactor mantl-storage-setup a bit

Migrate to docker.io's docker package

f901abc

Simplify code

d13691e

Devicemapper support

6356491

Fixed pool name for devicemapper

9406ded

Upgrade path for docker/docker-engine

6d9e983

rename variables from DOCKER_ to MANTL_

8c380f3

avnik force-pushed the feature/custom-partitioner branch from e91aefa to 8c380f3 Compare March 31, 2016 13:50

stevendborrelli reviewed Mar 31, 2016
View reviewed changes

Document new options

3e7a50d

stevendborrelli mentioned this pull request Apr 6, 2016

Support docker 1.9 overlay networking #781

Closed

avnik added 2 commits April 6, 2016 17:39

Add documentation for new partitioner

d114201

Still need to be included/linked from main doc

Note about docker_source variable

f71a577

remove duplicating docker-io repo

a42735d

langston-barrett reviewed Apr 6, 2016
View reviewed changes

Add Docker gc package

fa19252

This package cleans up docker containers, images and volues by adding systemd timers tht run commands like docker rm

avnik added 4 commits April 19, 2016 12:56

Use same paths as packaged mantl-storage-setup

1f795f2

docker: Install docker-gc after lvm stuff

5271aa6

docker: Install package after systemd drop-ins added

afa8619

Explicit process with mantl-storage-setup

af414dd

ryane reviewed Apr 27, 2016
View reviewed changes

ryane added the readytotest label Apr 28, 2016

ryane merged commit 0fd30d7 into master Apr 28, 2016

ryane deleted the feature/custom-partitioner branch April 28, 2016 15:01

ryane mentioned this pull request Apr 28, 2016

force remove containers in systemd services #1393

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new partitioner script, which can do job on first boot #1239

Add new partitioner script, which can do job on first boot #1239

avnik commented Mar 7, 2016 •

edited by ryane

avnik commented Mar 31, 2016

stevendborrelli Mar 31, 2016

avnik Mar 31, 2016

langston-barrett Apr 6, 2016

stevendborrelli commented Mar 31, 2016

langston-barrett commented Apr 6, 2016

langston-barrett Apr 6, 2016

langston-barrett commented Apr 6, 2016 •

edited by ryane

ryane commented Apr 12, 2016

ryane commented Apr 12, 2016

avnik commented Apr 18, 2016

ryane Apr 27, 2016

ryane commented Apr 27, 2016

ryane commented Apr 28, 2016

Add new partitioner script, which can do job on first boot #1239

Add new partitioner script, which can do job on first boot #1239

Conversation

avnik commented Mar 7, 2016 • edited by ryane

avnik commented Mar 31, 2016

stevendborrelli Mar 31, 2016

Choose a reason for hiding this comment

avnik Mar 31, 2016

Choose a reason for hiding this comment

langston-barrett Apr 6, 2016

Choose a reason for hiding this comment

stevendborrelli commented Mar 31, 2016

langston-barrett commented Apr 6, 2016

langston-barrett Apr 6, 2016

Choose a reason for hiding this comment

langston-barrett commented Apr 6, 2016 • edited by ryane

ryane commented Apr 12, 2016

ryane commented Apr 12, 2016

avnik commented Apr 18, 2016

ryane Apr 27, 2016

Choose a reason for hiding this comment

ryane commented Apr 27, 2016

ryane commented Apr 28, 2016

avnik commented Mar 7, 2016 •

edited by ryane

langston-barrett commented Apr 6, 2016 •

edited by ryane