Skip to content

Tools for generating yaml documents from templates using bosh ops files and variable interpolation

License

Notifications You must be signed in to change notification settings

cjnosal/manifer

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

76 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

manifer

Manifer uses yaml processors and variable interpolation to generate yaml documents from a starting template and a set of reusable yaml snippets.

To reduce the size and complexity of yaml documents, named sets of ops files or yq scripts can be organized into a 'library'. The 'template' is now only responsible for simplified high level definitions. Running manifer will combine the library and template to compose the final document.

glossary

  • template: an arbitrary yaml file to be modified
  • snippet: a file used to modify the template (e.g. BOSH ops files or yq scripts)
  • scenario: a named collection of snippets that should be used together
  • library: manifer's yaml file format that defines a collection of scenarios and dependencies between scenarios
  • interpolate: replace variable placeholders with values from the CLI, files, or environment variables
  • process: apply structural changes to a template as defined by a snippet
  • compose: process and interpolate all snippets defined in a set of scenarios

getting started

1a) generate a library from your collection of opsfiles and yq scripts
manifer import -r -p ./ops-dir -o ./new-lib.yml
or
1b) generate a library from your yaml file
manifer generate -t ./template.yml -y opsfile -o ./new-lib.yml -d ./opsdir
2) view the generated scenarios
manifer list -l new-lib.yml
3) add scenarios for your common use cases, which can define variables or invoke other scenarios
manifer add -l new-lib.yml -n use_case -d "thing I need frequently" -s "dependency_name" -- -v foo=bar -o extra-op.yml
4) inspect the scenario you created
manifer inspect -l new-lib.yml -s use_case
5) use your new scenario to modify a template
manifer compose -l new-lib.yml -t base.yml -s use_case

global flags and environment variables

For convenience there are different ways to specify which libraries to read.
In order of precedence:

  1. a local flag
    manifer list -l mylib.yml
  2. a global flag
    manifer -l mylib.yml list
  3. specific libraries via MANIFER_LIBS and the system path separator
    MANIFER_LIBS=mylib.yml:myotherlib.yml manifer list
  4. directories to search via MANIFER_LIB_PATH and the system path separator
    MANIFER_LIB_PATH=./mylibs/:./sharedlibs manifer list

subcommands

import

./manifer import [--recursive] --path <import path> --out <library path>:
  create a library from a directory of snippets.

Usage:
  manifer import [flags]

Flags:
  -h, --help               help for import
  -o, --out string         Path to save generated library file
  -p, --path string        Directory or snippet to import
  -r, --recursive          Import snippets from subdirectories

Global Flags:
  -l, --library strings   Path to library file

generate

./manifer generate --template <yaml path> --out <library path> [--directory <snippet path>]:
  create a library based on the structure of a yaml file.

Usage:
  manifer generate [flags]

Flags:
  -d, --directory string   Directory to save generated snippets (default out/snippets)
  -h, --help               help for generate
  -o, --out string         Path to save generated library file
  -y, --processor string   Yaml backend for this library (opsfile or yq)
  -t, --template string    Template to generate from

Global Flags:
  -l, --library strings   Path to library file

add

./manifer add --library <library path> --name <scenario name> [--description <text>] [--scenario <dependency>...] [-- passthrough flags ...]:
  add a new scenario to a library.

Usage:
  manifer add [flags]

Flags:
  -d, --description string   Informative description of the new scenario
  -h, --help                 help for add
  -n, --name string          Name to identify the new scenario
  -s, --scenario strings     Dependency of the new scenario

Global Flags:
  -l, --library strings   Path to library file

list

./manifer list [--all] (--library <library path>...):
  list scenarios in selected libraries.

Usage:
  manifer list [flags]

Flags:
  -a, --allScenarios   Include all referenced libraries
  -h, --help           help for list
  -j, --json           Print output in json format

Global Flags:
  -l, --library strings   Path to library file

search

./manifer search (--library <library path>...) (query...):
  search scenarios in selected libraries by name and description.

Usage:
  manifer search [flags]

Flags:
  -h, --help   help for search
  -j, --json   Print output in json format

Global Flags:
  -l, --library strings   Path to library file

inspect

./manifer inspect (--library <library path>...) [--tree|--plan] (-s <scenario name>...) [-- passthrough flags ...]:
  inspect scenarios as a dependency tree or execution plan.

Usage:
  manifer inspect [flags]

Flags:
  -h, --help               help for inspect
  -j, --json               Print output in json format
  -p, --plan               Print execution plan
  -s, --scenario strings   Scenario name in library
  -t, --tree               Print dependency tree (default)

Global Flags:
  -l, --library strings   Path to library file

compose

./manifer compose --template <template path> (--library <library path>...) (--scenario <scenario>...) [--print] [--diff] [-- passthrough flags ...] [\;] :
  compose a yml file from snippets. Use '\;' as a separator when reusing a scenario with different variables.

Usage:
  manifer compose [flags]

Flags:
  -d, --diff               Show diff after each snippet is applied
  -h, --help               help for compose
  -p, --print              Show snippets and arguments being applied
  -s, --scenario strings   Scenario name in library
  -t, --template string    Path to initial template file

Global Flags:
  -l, --library strings   Path to library file

appending additional compositions

Additional compositions can be appended using \; as a separator. For each additional composition:

  • the output of the last composition is used as the template
  • the list of libraries will be preserved
  • new libraries can be referenced
  • new scenarios and passthrough arguments can be specified
  • global variables cleared

This allows the value of a variable to be changed, without having to re-enter file paths for the libraries or template. The following invocations are equivalent:

./manifer compose -t my-template -l my-library -s my-scenario -- -v arg=foo > temp
./manifer compose -t temp -l my-library -s my-scenario -- -v arg=bar > final
./manifer compose -t my-template -l my-library -s my-scenario -- -v arg=foo \; \
  -s my-scenario -- -v arg=bar > final

schemas

template

Any valid yaml document you would like to modify with snippets and implicit bosh variables

e.g. foo-template.yml

foo:
  bar: bizz
  buzz: ((bazz))
  extra: redundant

snippets

Yaml snippets to compose into the template:

e.g. base-case.yml

- path: /foo/bar
  type: replace
  value: ((newbar))

- path: /foo/extra
  type: remove

- path: /foo/((sub))? # note: bosh/go-patch do not natively support variables in opsfile paths
  type: replace
  value:
    new: struct

library

library.go

Library files are used to organize sets of snippets, interpolation variables, and processor options to allow easy dynamic composition of large yaml files.

Libraries consist of:

  • a default processor type for all snippets
    type: opsfile
  • aliases to other libraries
    libraries:
    - alias: common
      path: ./commonlib.yml
    
  • a list of scenarios, consisting of:
    • a unique name
    • a user-friendly description
    • references to other scenarios this scenario depends on:
      • by name, prefixed with . delimited library aliases
      • interpolator variables to apply to the referenced scenario
    • snippets to transform the template yaml:
      • path to the snippet file
      • interpolator variables for this snippet
      • processor options for this snippet
    • interpolator variables to use with all snippets in this scenario and referenced scenarios
    • global interpolator variables to use with all snippets in this composition
    scenarios:
    - name: first
      description: my first scenario
      scenarios:
      - name: second
        interpolator:
          vars:
            foo: bar
      - name: common.setup
      snippets:
      - path: ./opsfile.yml
        interpolator:
          vars:
            bizz: bazz
        processor:
          type: opsfile
          options:
            path: /buzz
    - name: second
      snippets:
      - path: ./secondop.yml
      - path: ./thirdop.yml
      global_interpolator:
        vars_store: ./generated.yml
    

migrating from v1

In v1 libraries interpolator variables were specified as CLI args. In v2 args is replaced by the interpolator struct.

When upgrading from manifer v1=>v2 you can either:

  • move each args element to interpolator.raw_args or
  • replace args with the appropriate interpolator.var* field

interpolator variables

Variables can be defined by adding an interpolator block to a snippet, scenario reference, scenario, or via passthrough flags from the CLI

interpolator:
  vars: {} # map variable names to static values [--var=key=val (-v)]
  var_files: {} # map variable names to files that contain the value [--var-file=key=path]
  vars_files: [] # file paths that contain a map of variable names to static values [--vars-file=path (-l)]
  vars_env: [] # environment variables with the given prefixes [--vars-env=prefix]
  vars_store: "" # a vars-file that can lazily generate random passwords or certificates [--vars-store=path]
  raw_args: [] # insert CLI flags into the scenario definition (for internal use)

See bosh interpolate and variable types for more details

processor options

A snippet can override the library's default processor type, or provide options

opsfile processor

type: opsfile
options:  
  path: "/foo" # return a section of the composed yaml instead of the full document

See the go-patch and ops-file docs for more details

Differentiating features: field matching and index selection

yq processor

type: yq
options:  
  command: # write, read, delete, prefix, merge (default write)
  path: # yaml element to read or delete
  prefix: # key to nest the current yaml structure under
  overwrite: # boolean for merges to replace existing elements
  append: # boolean for merges to append new array elements

See the yq docs for more details

Differentiating features: wildcards, prefix, and merge

Invocation

Running manifer compose --library mainlib.yml --template foo-template.yml --scenario my-use-case should produce:

foo:
  bar: trendy
  buzz: 123
  nested:
    new: struct

interpolation and variables

There are four variable scopes:

  • snippet vars
  • scenario vars
  • scenario global vars
  • CLI global vars

Every snippet is used in two interpolations:

  • the snippet itself is interpolated with snippet vars, scenario vars, and global vars
  • then the interpolated snippet is applied to the template, interpolated with global vars

multiple libraries

Libraries can include other libraries by specifying the path and an alias under libraries:. If a scenario needs to include a scenario from a referenced library the name should be prefixed with <library alias>..

If multiple independant libraries are provided to the CLI all scenario names should be unambiguous.

build

./scripts/build.sh [all]

  • all will build manifer_darwin and manifer_linux

test

./scripts/test.sh [unit|integration|go test flags]

  • -count=1 can be used to disable test caching of integration tests in cmd/manifer

use manifer in your project

lib.Manifer can be imported to list scenarios or compose yaml

package main

import (
  "os"
  "fmt"
  "github.com/cjnosal/manifer/v2/lib"
)

func main() {
  logger := os.Stderr
  output := os.Stdout
  manifer := lib.NewManifer(logger)

  // starting yaml file
  template := "test/data/v2/template.yml"

  // collection of scenarios
  libraries := []string{"test/data/v2/library.yml"}

  // sets of snippets to apply
  scenarios := []string{"placeholder"}

  // arguments to pass through to `bosh interpolate`
  interpolationVars := []string{"-vpath3=/foo", "-vvalue3=tweaks"}

  // list scenario names with descriptions
  scenarioSummary, err := manifer.ListScenarios(libraries, false)
  logger.Write([]byte(fmt.Sprintf("%v\n", scenarioSummary)))
  logger.Write([]byte(fmt.Sprintf("%v\n", err)))

  // apply snippets from the selected scenarios to the provided template
  composedYaml, err := manifer.Compose(template, libraries, scenarios, interpolationVars, false, false)
  output.Write(composedYaml)
  logger.Write([]byte(fmt.Sprintf("%v\n", err)))
}

About

Tools for generating yaml documents from templates using bosh ops files and variable interpolation

Topics

Resources

License

Stars

Watchers

Forks

Packages

No packages published