Cannot render pillar SLS with more than 98 if/elif statements

[dseira]

Description of Issue/Question

Having a pillar with the following content:

{% if grains['id'] == 'server0' %}
{% elif grains['id'] == 'server1' %}
{% elif grains['id'] == 'server2' %}
{% elif grains['id'] == 'server3' %}
{% elif grains['id'] == 'server4' %}
{% elif grains['id'] == 'server5' %}
{% elif grains['id'] == 'server6' %}
.......
{% elif grains['id'] == 'server96' %}
{% elif grains['id'] == 'server97' %}
{% elif grains['id'] == 'server98' %}
{% endif %}

Fails with the following error message:

2017-08-22 17:25:57,194 [salt.pillar      ][CRITICAL][22011] Pillar render error: Rendering SLS 'test' failed, render error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 164, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 442, in render_jinja_tmpl
    line, out = _get_jinja_error(trace, context=decoded_context)
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 241, in _get_jinja_error
    line = _get_jinja_error_line(trace)
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 224, in _get_jinja_error_line
    return _get_jinja_error_slug(tb_data)[1]
TypeError: 'NoneType' object has no attribute '__getitem__'

On the other hand, if one of the elif statement line is removed (keeping 98 elif), the sls is compiled without errors.

It seems there is a hard limit on elif statements in the jinja templating because the following pillar doesn't fail:

...
{% elif grains['id'] == 'server95' %}
{% elif grains['id'] == 'server96' %}
{% elif grains['id'] == 'server97' or grains['id'] == 'server98' %}
{% endif %}

Setup

test.sls

{% if grains['id'] == 'server0' %}
{% elif grains['id'] == 'server1' %}
{% elif grains['id'] == 'server2' %}
{% elif grains['id'] == 'server3' %}
{% elif grains['id'] == 'server4' %}
{% elif grains['id'] == 'server5' %}
{% elif grains['id'] == 'server6' %}
.......
{% elif grains['id'] == 'server96' %}
{% elif grains['id'] == 'server97' %}
{% elif grains['id'] == 'server98' %}
{% endif %}

Steps to Reproduce Issue

Create a pillar with the above content and assign it to a minion. After that, try to list the pillar items of a minion:

salt 'minion1' pillar.items

The error will be raise. After that, remove the last elif line (for example):

{% elif grains['id'] == 'server98' %}

The sls is compiled.

Versions Report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Tested in two different salt versions:

Salt Version:
           Salt: 2017.7.1
 
Dependency Versions:
           cffi: 0.8.6
       cherrypy: 3.2.2
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.0
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.6 (default, Oct 26 2016, 20:30:19)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
 
System Versions:
           dist: Ubuntu 14.04 trusty
         locale: UTF-8
        machine: x86_64
        release: 3.13.0-101-generic
         system: Linux
        version: Ubuntu 14.04 trusty

Salt Version:
           Salt: 2016.11.3
 
Dependency Versions:
           cffi: 0.8.6
       cherrypy: 3.2.2
       dateutil: 1.5
          gitdb: Not Installed
      gitpython: 0.1.6
          ioflo: Not Installed
         Jinja2: 2.6
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.0
   msgpack-pure: Not Installed
 msgpack-python: 0.3.0
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
         pygit2: Not Installed
         Python: 2.7.3 (default, Oct 26 2016, 21:01:49)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.5.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
 
System Versions:
           dist: Ubuntu 12.04 precise
        machine: x86_64
        release: 3.13.0-71-generic
         system: Linux
        version: Ubuntu 12.04 precise

Same behaviour in both.

[gtmanfred]

This appears to be a limitation of jinja2.

[root@salt ~]# cat this.py
import os
import jinja2
def render(tpl_path, context):
    path, filename = os.path.split(tpl_path)
    return jinja2.Environment(
        loader=jinja2.FileSystemLoader(path or './')
    ).get_template(filename).render(context)

print(render('./this.jinja2', {'name': 'daniel'}))
[root@salt ~]# python this.py

1
[root@salt ~]# vi this.jinja2
[root@salt ~]# grep -c '1 == 1' this.jinja2
99
[root@salt ~]# python this.py
Traceback (most recent call last):
  File "this.py", line 9, in <module>
    print(render('./this.jinja2', {'name': 'daniel'}))
  File "this.py", line 7, in render
    ).get_template(filename).render(context)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 830, in get_template
    return self._load_template(name, self.make_globals(globals))
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 804, in _load_template
    template = self.loader.load(self, name, globals)
  File "/usr/lib/python2.7/site-packages/jinja2/loaders.py", line 125, in load
    code = environment.compile(source, name, filename)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 588, in compile
    return self._compile(source, filename)
  File "/usr/lib/python2.7/site-packages/jinja2/environment.py", line 551, in _compile
    return compile(source, filename, 'exec')
  File "./this.jinja2", line 501
    pass
       ^
IndentationError: too many levels of indentation

Can you open a bug upstream with them?

Thanks,
Daniel

[dseira]

Done. #759

[gtmanfred]

Thank!
Daniel

[ThiefMaster]

I've just fixed this upstream - pallets/jinja#760. The fix will go into 2.10 and 2.9.7 (not yet released).

However, since you are using Jinja (2.6, a six-year-old version) from OS packages, you'd probably need to ask distros to backport the fix or salt needs to somehow use a more recent Jinja version... Or maybe you can avoid generating that much elif? Generating separate if..endif blocks (and thus not using elif at all) might be the easiest way to fix this on your side without requiring a newer Jinja version.

[gtmanfred]

We provide the oldest supported version in our repos usually, newer versions need to be supplied by the distro or by using pip to install.

[dseira]

I will try it through pip when it is released.

Thanks both,
David