makemessages for Jinja2 fails translation lookup with trim_blocks: true and leading newline in trans block

[vaughnkoch]

Hi, thanks for this useful library!

I'm using django-jinja to run makemessages with Jinja trans blocks to make a .po file for translation.

I see that you've added ext.i18n.trimmed support - can you also add support for trim_blocks and lstrip_blocks? When I use some of these settings, makemessages doesn't strip away the newline, but Jinja does. This causes gettext to fail lookup because the .mo file has the newline baked in.

Example template:

{% trans %}      <--- offending newline which Jinja strips away with trim_blocks: True
Hi Jinja
{% endtrans %}

This template works in comparison:

{% trans %}Hi Jinja{% endtrans %}

See pallets/jinja#1925 for context.

[wizpig64]

Hi @vaughnkoch ,

I've submitted a pull request that enables trim_blocks and lstrip_blocks if you set them in the TEMPLATES settings.

Would you mind trying it out with your project before I release a new version of django-jinja? If there's anything I'm missing or have messed up, I'd rather find out before pushing it to PyPI.

pip install -e git+https://github.com/wizpig64/django-jinja.git@5fd0fce26c510969d7b7216695769feec84ddc12#egg=django_jinja

Thanks and I hope this helps!

[vaughnkoch]

Hi Philip, thanks for the quick turnaround! I tested your updated package, but I'm getting an error when running pytest.

Here's a summary of the behavior:

• With trim_blocks and lstrip_blocks set in settings...OPTIONS, and also trim_blocks=True and lstrip_blocks=True in the custom jinja environment, I get the error shown in the log below.
• With those entries commented out in settings, but enabled in the custom jinja env, I still get the below error.
• With the entries enabled in Django settings, but commented out in the custom jinja env, I don't get any stacktrace errors, but the text isn't correctly translated (the original problem in this github issue).
• With entries commented out in settings AND commented out in the custom jinja env, I get no errors, and the translation works as expected, but the trim_blocks behavior is obviously different (doesn't actually trim so I get a bunch of unwanted newlines).

I think the TLDR from the above is that with this new code, adding trim_blocks / lstrip_blocks to the custom jinja env causes this error. This didn't appear to happen with the published django-jinja library.

Errors:

(api-5Nl-c2RL) ~/redacted $ pytest -k test_redacted

========================================================================== test session starts ==========================================================================
collected 85 items / 84 deselected / 1 selected          
redacted/tests.py [2022-12-31 16:00:01] ERROR reminders.py:189   error: 'trim_blocks'
Traceback (most recent call last):
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/utils.py", line 69, in __getitem__
    return self._engines[alias]
           ~~~~~~~~~~~~~^^^^^^^
KeyError: 'jinja2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/virtualenvs/redacted/reminders.py", line 184, in compose_message
    body = render_to_string_with_translation(template, context).strip()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/redacted/util/i18n/translation.py", line 27, in render_to_string_with_translation
    text = render_to_string(template_name, context, request, using)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/loader.py", line 61, in render_to_string
    template = get_template(template_name, using=using)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/loader.py", line 12, in get_template
    engines = _engine_list(using)
              ^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/loader.py", line 66, in _engine_list
    return engines.all() if using is None else [engines[using]]
           ^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/utils.py", line 94, in all
    return [self[alias] for alias in self]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/utils.py", line 94, in <listcomp>
    return [self[alias] for alias in self]
            ~~~~^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/utils.py", line 85, in __getitem__
    engine = engine_cls(params)
             ^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/src/django-jinja/django_jinja/backend.py", line 190, in __init__
    self.env = environment_cls(**options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/redacted/util/template/jinja_custom_env.py", line 13, in environment
    env = Environment(
          ^^^^^^^^^^^^^^^^^^^^
KeyError: 'trim_blocks'
[2022-12-31 16:00:01] ERROR redacted.py:189 error: 'trim_blocks'
Traceback (most recent call last):
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/utils.py", line 69, in __getitem__
    return self._engines[alias]
           ~~~~~~~~~~~~~^^^^^^^
KeyError: 'jinja2'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/virtualenvs/redacted/reminders.py", line 184, in compose_message
    body = render_to_string_with_translation(template, context).strip()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/redacted/util/i18n/translation.py", line 27, in render_to_string_with_translation
    text = render_to_string(template_name, context, request, using)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/loader.py", line 61, in render_to_string
    template = get_template(template_name, using=using)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/loader.py", line 12, in get_template
    engines = _engine_list(using)
              ^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/loader.py", line 66, in _engine_list
    return engines.all() if using is None else [engines[using]]
           ^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/utils.py", line 94, in all
    return [self[alias] for alias in self]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/utils.py", line 94, in <listcomp>
    return [self[alias] for alias in self]
            ~~~~^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/lib/python3.11/site-packages/django/template/utils.py", line 85, in __getitem__
    engine = engine_cls(params)
             ^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/api-5Nl-c2RL/src/django-jinja/django_jinja/backend.py", line 190, in __init__
    self.env = environment_cls(**options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "~/virtualenvs/redacted/util/template/jinja_custom_env.py", line 13, in environment
    env = Environment(
          ^^^^^^^^^^^^^^^^^^^^
KeyError: 'trim_blocks'

django_jinja from ipdb:

(api-5Nl-c2RL) ~/src $ pym shell
Current arguments: ['manage.py', 'shell']), RUN_MAIN=None

In [1]: import django_jinja

In [2]: django_jinja??
Type:        module
String form: <module 'django_jinja' from '~/virtualenv/src/django-jinja/django_jinja/__init__.py'>
File:        ~/data/virtualenvs/api-5Nl-c2RL/src/django-jinja/django_jinja/__init__.py
Docstring:   <no docstring>

Jinja environment:

def environment(**options):
    env = Environment(
        trim_blocks=True,
        lstrip_blocks=True,
        **options,
    )

    env.add_extension('jinja2.ext.i18n')
    # env.policies['ext.i18n.trimmed'] = True

    env.globals['redacted'] = redacted

    # This function is added by the jinja2.ext.i18n extension
    env.install_gettext_translations(translation, newstyle=True)  # type: ignore

    env.globals.update(
        {
            'static': static,
            'url': reverse,
        }
    )

    return env

settings.py:

{
    # regular django templates...
}
{
    'NAME': 'jinja2',
    'BACKEND': 'django_jinja.backend.Jinja2',
    'DIRS': [
        os.path.join(BASE_DIR, 'redacted', 'templates'),
    ],
    'APP_DIRS': True,
    'OPTIONS': {
        'environment': 'redacted.template.jinja_custom_env.environment',
        'match_extension': '',
        'trim_blocks': True,
        'lstrip_blocks': True,
    },
},

Pipfile:

django-jinja = {editable = true, git = "https://github.com/wizpig64/django-jinja.git@5fd0fce26c510969d7b7216695769feec84ddc12"}

Pipfile.lock:

"django-jinja": {
    "editable": true,
    "git": "https://github.com/wizpig64/django-jinja.git",
    "hashes": [
        "sha256:47c06d3271e6b2f27d3596278af517bfe2e19c1eb36ae1c0b1cc302d7f0259af",
        "sha256:cc4c72246a6e346aa0574e0c56c3e534c1a20ef47b8476f05d7287781f69a0a9"
    ],
    "ref": "5fd0fce26c510969d7b7216695769feec84ddc12"
},

Pipenv graph output:

django-jinja==2.11.0
  - django [required: >=3.2, installed: 4.2]
    - asgiref [required: >=3.6.0,<4, installed: 3.7.2]
    - sqlparse [required: >=0.3.1, installed: 0.4.4]
  - jinja2 [required: >=3, installed: 3.1.2]
    - MarkupSafe [required: >=2.0, installed: 2.1.4]