The ruby GEM consul-templaterb is both a library and an executable that allows to generate files using data from Consul (Discovery and Key/Value Store) easily using ruby's erb templates. It also support launching programs and baby-sitting processes when rendering the files, thus notifying programs when data do change.
It is intended for user accustomed to expressiveness or Ruby templating (ERB), allowing for more flexibility and features than Go templates.
It also allows to use all of ruby language, especially useful for generating files in several formats (JSON, XML) for which text substitutions are hard to get right (escaping, attributes encoding...).
It also focuses on good performance and lightweight usage of bandwidth, especially for very large clusters and watching lots of services.
For complicated rendering of templates and large Consul Clusters, it usually renders faster with a more predictable way the template than the original consul-template.
It provides a very simple API to write your own templates with fully working examples.
It also allow to display a very nice and hi-performance HTML5 UI for Consul, see consul-ui for details.
Hashicorp's Consul Template inspired strongly the creation of this GEM and this executable wants to achieve better results in some use cases, especially for very large Consul clusters with many nodes and servers.
consul-template has more features regarding Consul support (for instance, it has support for Hashicorp's Vault), but consul-templaterb focuses on getting more power with the generation of templates and more performance.
Consul Template uses Go templates which is very limited in its set of features: it is complicated to sort, apply real transformations using code and even interact with the OS (ex: get the current date, format timestamps...).
The sort feature for instances allow you to create predictable output (i.e: meaning that the order of nodes is predictable), thus it might offer better performance since the reload of processes if happening ONLY when the files are binary different. Thus, if using consul-templaterb, you will reload less your haproxy or load-balancer than you would do with consul-template.
Compared to consul-template, consul-templaterb offers the following features:
- Hot-Reload of template files
- Bandwidth limitation per endpoint (will soon support dynamic bandwidth limiter)
- Supports launch and supervision of multiple child processes
- Supports launching commands when files do change on disk (reload commands...)
- Supports all Ruby features (ex: base64, real JSON/XML generation...)
- Information about bandwidth
The executable supports semantics and command line flags and options similar to
HashiCorp's Consul-template, so many flags you might use in consul-template will
work in a similar way. It also supports the same environment variable
CONSUL_HTTP_ADDR to find the Consul Agent to query and CONSUL_HTTP_TOKEN to
get the token.
You might either use the executable directly OR use this GEM as a library by
adding this line to your application's Gemfile:
gem 'consul-templaterb'And then execute:
$ bundle
[...]Or install it yourself as:
$ gem install consul-templaterb
[...]If you simply want to use the executable on your preferred Linux distribution, you have to install first: ruby and ruby dev.
sudo apt-get install ruby ruby-dev && sudo gem install consul-templaterbYou can now use it directly using the binary consul-templaterb in your path.
On Windows, a bug exists with Ruby greater than 2.4.
In order to make it work, use this commands as explained in oneclick/rubyinstaller2#96 (comment)
gem install eventmachine consul-templaterb --platform ruby
Samples are installed with the GEM, you can either download them or simply use the ones installed with the gem. To figure out where the templates are installed:
$ gem contents consul-templaterb|grep samples
[...]Will output the path where the samples are being installed, you can copy the directory somewhere and then issue the command:
$ consul-templaterb samples/*.html.erb
Using samples/checks.html output for samples/checks.html.erb
[...]It will render a full web site you may browse to look in real time the status of your Consul Cluster.
You can now have a look to the API Documentation to modify existing templates or write your owns, it is very easy!
$ consul-templaterb --help
USAGE: consul-templaterb [[options]]
-h, --help Show help
-v, --version Show Version
-g, --no-gzip-compression Disable GZIP compression in HTTP requests
-c, --consul-addr=<address> Address of Consul, eg: http://localhost:8500
-l, --log-level=<log_level> Log level, default=info, any of none|error|info|debug
--consul-token=<token> Use a token to connect to Consul
-V, --vault-addr=<address> Address of Vault, eg: http://localhost:8200
--vault-token=<token> Token used to authenticate against vault.
--[no-]vault-renew Control auto-renewal of the Vault token. Default: activated
--vault-lease-duration-factor=<factor>
Wait at least <factor> * lease time before updating a Vault secret. Default: 0.5
-w, --wait=<min_duration> Wait at least n seconds before each template generation
-r, --retry-delay=<min_duration> Min Retry delay on Error/Missing Consul Index
-k, --hot-reload=<behavior> Control hot reload behaviour, one of :[die (kill daemon on hot reload failure), keep (on error, keep running), disable (hot reload disabled)]
-K, --sig-term=kill_signal Signal to send to next --exec command on kill, default=TERM
-T, --trim-mode=trim_mode ERB Trim mode to use (- by default)
-R, --sig-reload=reload_signal Signal to send to next --exec command on reload (NONE supported), default=HUP
-M, --debug-memory-usage Display messages when RAM grows
-e, --exec=<command> Execute the following command
-d, --debug-network-usage Debug the network usage
-t erb_file:[output]:[command]:[params_file],
--template Add a erb template, its output and optional reload command
-o, --once Do not run the process as a daemonWhen launched with file arguments ending with .erb, the executable will assume
the file is a template and will render the corresponding file without the
.erb extension.
It means that you can call consul-templaterb with *.erb arguments, the shell
will then substitute all files and render it by removing the .erb extension as
if the --template my_file.ext.erb:myfile.ext was used.
In the same way as consul-template, consul-templaterb supports multiple templates and executing
commands when the files do change. The parameter --template <ERB>:<DEST>:[reload_command]:params_file works
in the following way:
- ERB : the ERB file to use as a template
- DEST: the destination file
- reload_command: optional shell command executed whenever the file has been modified
- params_file: JSON or YAML file to load and to use as parameter for template (see param() function to retrieve the values)
The argument can be specified multiple times, ex:
Example of usage:
$ consul-templaterb \\
--template "samples/ha_proxy.cfg.erb:/opt/haproxy/etc/haproxy.cfg:sudo service haproxy reload"
--template "samples/consul_template.erb:consul-summary.txt"With the --exec argument (can be specified multiple times), consul-templaterb will launch
the process specified when all templates have been generated and will send a reload signal
if the content of any of the files do change (the signal will be sent atomically however,
meaning that if 2 results of templates are modified at the same time, the signal will be
sent only once (it is helpful for instance if your app is using several configurations
files that must be consistent all together).
Signals can be customized per process. Two signals are supported with options --sig-reload and
--sig-term. When the option is added, the next --exec options to start a process will use the
given signal. By default, HUP will be sent to reload events (you can use NONE to avoid sending any
reload signal), TERM will be used when leaving consul-templaterb.
This is actually the original reason for the creation of this GEM: on Criteo's large clusters, consul-template generated several hundreds of Mb/s to the Consul-Agent which also generated several hundreds of Mb/s with the Consul servers.
By design, the GEM supports limiting the number of requests per endpoints (see code in
bin/consul-templaterb file). It avoids using too much network to fetch data from Consul
in large Consul Clusters (especially when watching lots of files).
The limitation is static for now, but fair dynamic bandwidth allocation will allow to limit the bandwidth used to get information for all services by capping the global bandwidth used by consul-templaterb.
Have a look into the samples/ directory to browse example files which contains those examples:
- List all nodes on Cluster
- Show all services in Cluster
- Show all Service Checks and their output
- Show all Key/Values nicely
- Show Choregraphies - work on content of K/V with JSON
- Services in XML
- Services in JSON
- Generate HaProxy Configuration
- Export Consul Statistics to Prometheus : count all services, their state, datacenters and nodes and export it to prometheus easily to trigger alerts.
- List all services/Nodes with their statuses for all datacenters
- Show all services/instances not passing on all DCs
If you want to test it quickly, you might try with (assuming your consul agent is listening on
http://localhost:8500):
$ be bin/consul-templaterb -c 'http://localhost:8500' samples/*.html.erb
[...]It will generate a full website in samples/ directory with lots of Consul information ready to use (website updated automagically when values to change).
All templates are validated using the Travis CI, so all should be working for your Consul Configuration.
Please look at the template API to have a list of all functions you might use for your templates. Also have a look to samples/ directory to have full working examples.
We recommend using bundle using bundle install, you can now run bundle exec bin/consul-templaterb.
Help is available running bundle exec bin/consul-templaterb --help
The following example will generate static HTML pages and JSON data for consul-ui:
bundle exec bin/consul-templaterb -c your.consul.agent:8500 samples/consul-ui/*.erb
If you need remote calls, you need an HTTP server. A simple way to have one is using Python's simple HTTP
server. Example for consul-ui:
cd samples/consul-ui
python -m SimpleHTTPServer
To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the
version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version,
push git commits and tags, and push the .gem file to rubygems.org.
Here are the known bugs of the application:
-
render_filemight create an infinite recursion if a template includes itself indirectly. - On Windows, the consul-templaterb cannot work with more than 2048 endpoints watched (see oneclick/rubyinstaller#104 (comment))
Please consult CHANGELOG.md for fixed bugs.
- Hashi's Vault support (EXPERIMENTAL)
- Implement automatic dynamic rate limit
- More samples: apache, nginx, full website displaying consul information...
- Optimize rendering speed at start-up: an iteration is done very second by default, but it would be possible to speed up rendering by iterating with higher frequency until the first write of result has been performed.
- Allow to tune bandwidth using a simple configuration file (while it should not be necessary for 90% of use-cases)
See CONTRIBUTING.md for guidance: Bug reports and pull requests are welcome on GitHub at https://github.com/criteo/consul-templaterb.
This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct. See CODE_OF_CONDUCT.md
The gem is available as open source under the terms of the Apache v2 license. See LICENSE.txt file.