Kirby

Code Coverage Tool for Ansible

Description

It is usual to measure the code coverage for your source code written in python, Java, and so on. On the other hand, we usually do not measure the coverage for an Ansible playbook. Kirby is the tool to support this.

Here is the example. This is the playbook to be tested. There are 2 tasks.

~/src/kirby_demo% cat create_2dirs.yml
---
- hosts: localhost
  gather_facts: no
  tasks:
    - name: create dir1
      file: path=./dir1 state=directory

    - name: create dir2
      file: path=./dir2 state=directory

Here is the Serverspec test. There is 1 test for the first task (create dir1).

~/src/kirby_demo% cat spec/localhost/sample_spec.rb 
require 'spec_helper'

describe file('./dir1') do
  it { should be_directory }
end

Now, run the playbook. Kirby shows you the code coverage.

~/src/kirby_demo% ansible-playbook create_2dirs.yml -i "localhost," -c local

PLAY [localhost] ************************************************************** 

TASK: [create dir1] *********************************************************** 
changed: [localhost]
tested by: 
- rspec ./spec/localhost/sample_spec.rb:4 # File "./dir1" should be directory

TASK: [create dir2] *********************************************************** 
changed: [localhost]
tested by: 

PLAY RECAP ******************************************************************** 
*** Kirby Results ***
Coverage  : 50% (1 of 2 tasks are tested)
Not tested:
 - create dir2
*** Kirby End *******
localhost                  : ok=2    changed=2    unreachable=0    failed=0   

It tells us the coverage (50%) and the task not tested (create dir2).

Installation

Download

Make a callback_plugins directory in your playbook directory, and put kirby.py inside of it.

mkdir callback_plugins
cd callback_plugins
wget https://raw.githubusercontent.com/ks888/kirby/master/callback_plugins/kirby.py

Setup

Make a kirby.cfg file in your playbook directory, and write the contents below.

[defaults]
enable = yes

serverspec_dir = ./
serverspec_cmd = bundle exec rake spec

• serverspec_dir is a directory to run serverspec.
• serverspec_cmd is a command to run serverspec.

Usage

Run

Run ansible-playbook command as usual.

• Hint: For the best results, your target system should be clean, that is, a playbook is not executed against the target system yet.

Check Results

Results are like this:

~/src/kirby_demo% ansible-playbook create_2dirs.yml -i "localhost," -c local

PLAY [localhost] ************************************************************** 

TASK: [create dir1] *********************************************************** 
changed: [localhost]
tested by: 
- rspec ./spec/localhost/sample_spec.rb:4 # File "./dir1" should be directory

TASK: [create dir2] *********************************************************** 
changed: [localhost]
tested by: 

PLAY RECAP ******************************************************************** 
*** Kirby Results ***
Coverage  : 50% (1 of 2 tasks are tested)
Not tested:
 - create dir2
*** Kirby End *******
localhost                  : ok=2    changed=2    unreachable=0    failed=0   

• When a task's result is changed, Kirby determines whether the task is tested, and shows you the result.

  • If the next line of tested by: is empty, the task was not tested (create dir2 task is this).
  • If not empty, the task was tested (create dir1 task is this).

• When a task's result is not changed, Kirby removes the task from the coverage, and shows nothing.

• At last, Kirby shows you a summary. It includes the coverage and the list of not tested tasks.

Improve Coverage

If a task is not tested, let's write a serverspec test for the task.

However, you may think the test is not necessary for some tasks, such as:

• run ls using command module.
• download a package in preparation for install.

There are 2 options for this:

1. use changed_when

   If a changed task does not change the target system actually, use changed_when like this:

   - command: ls
     changed_when: False

   Kirby removes the task from the coverage when a task's result is not changed.

2. use coverage_skip

   If you sure a test is not necessary, includes coverage_skip in a task's name like this:

   - name: create dir2 [coverage_skip]
     file: path=./dir2 state=directory

   Kirby removes the task from the coverage when a task's name includes coverage_skip.

FAQ

• How does Kirby calculate a coverage?
• It takes longer time to run a playbook.
• How can I disable Kirby?

Contributing

Contributions are very welcome, including bug reports, idea sharing, feature requests, and English correction of documents. Feel free to open an issue or a pull request, or contact me on twitter.