ImportError: cannot import name 'escape' from 'jinja2'

[bilykigor]

Environment:

• Python version: 3.7
• Jinja version: 3.1.0

[davidism]

Sounds like a library you use is attempting to do from jinja2 import escape, which was previously deprecated and now removed. You'll need to open a bug with that library, they should import directly from MarkupSafe: from markupsafe import escape

Use a tool like pip-tools to pin your dependencies and control when you get updates. Be sure to run your tests with deprecation warnings treated as errors so that you get notified of these types of changes early.

[cculianu]

You really should be doing these types of breaking changes when releasing a new major version, not a minor version. This is odd, really, to see removal of deprecated API across a minor version.

[davidism]

See https://hynek.me/articles/semver-will-not-save-you/, https://snarky.ca/why-i-dont-like-semver/, and https://bernat.tech/posts/version-numbers/, which go in depth on why pinning is the only valid solution to these types of issues, regardless of any project's version scheme.

[mjmeijer]

When the cause is hidden somewhere in your dependency tree, as in my Flask app, you can workaround for the moment by adding explicit dependecy to your requirements.txt

jinja2==3.0.3

[SylvainCorlay]

Jupyter maintainer here. Thanks for the heads up on the nbconvert repository.

Would you considering adding a shim back in a patch release? I think that this will break a lot of downstream use cases.

In general, I think it would be better for complete removal of deprecated APIs to only happen at major releases. It seems that these were deprecated with jinja2 3.0 and completely removed in jinja2 3.1, which is quite a short span of version for downstream packages to adapt to the change.

[davidism]

I don't plan to add any removed code back in. From past experience, it allows users who are not pinning or testing with warnings to defer, because suddenly they don't see the error anymore. Then they raise the same issue when I remove the shim later anyway.

We do not use semver, and have never indicated that we do. See https://hynek.me/articles/semver-will-not-save-you/, https://snarky.ca/why-i-dont-like-semver/, and https://bernat.tech/posts/version-numbers/. 3.0 was released in May last year, I do not consider 10 months a short span of time. From past experience I don't believe that any particular amount of time would have had a different result.

Deprecations have entries in the change log and warnings in the code. Use a tool like pip-tools to pin your dependencies and control when you get updates. Be sure to run your tests with deprecation warnings treated as errors so that you get notified of these types of changes early.

It looks like many of your users use nbconvert during documenation builds. For an example of how to pin for docs builds, see Jinja itself. This allows us to have a reproducible environment, and address upgrades on our schedule. For nbconvert itself, if you test with warnings as errors, you could address these before they become an issue, allowing your users to continue to ignore pinning for a bit longer.

[martinRenou]

Use a tool like pip-tools to pin your dependencies and control when you get updates

I am sorry, this only work if you have one package that depends on Jinja. You can very well end up with multiple dependencies that themselves depend on Jinja, if they all pin the specific Jinja version they need, you end up with an impossible to resolve environment at best (if using a package manager that solves the version constraints) or a completely broken environment at worst. This solution does not scale unfortunately.

[davidism]

I wasn't specific enough. Applications should pin exact dependencies. Libraries should pin minimum versions so that the resolver can work alongside any application pins (and pin exact versions for test/docs tools environments, see Jinja itself for an example of this). See this article for more discussion: https://iscinumpy.dev/post/bound-version-constraints/

If you don't want to commit to pinning though because you believe it won't work, then I also outlined the other solution: applications and libraries must test with warnings treated as errors. Then they can proactively address the warnings, either by setting a new minimum version, or by opting in to supporting both versions with a try/import/except or similar technique. Of course, this works best if you combine both approaches, pinning to control when you upgrade and address warnings.

[leandropls]

I can see why Quart/Flask/whatever library won't pin to a Jinja2 version. And I usually don't pin to any specific version of dependencies of libraries I use because it turns it into a maintenance hell (figuring out new dependency versions when we upgrade library versions).

But this was pretty messy... all of the sudden libraries that depended on Jinja2 got broken without any change in version or their code. And that cascaded to apps that depended on those libraries pinned to versions known to work. I agree with @cculianu that this could have been better handled by not breaking API's in minor version updates.

On the app developer's side, I'm thinking that maybe this can be prevented in a similar way the JavaScript community uses package.json vs package-lock.json.

Edit:
@davidism , thanks for the tip on pip-tool, I'll sure try it out.

[philippjfr]

Edit: now seeing that this was 10 months in the making I wonder why the deprecation warning never showed up for me. We indeed need to set our ci to fail on warnings, but I never even remember seeing the warning.

I certainly feel for any maintainer with tons of downstream users and who has to put up with long deprecation cycle. That said, in this case, for such a crucial library the deprecation cycle definitely felt too short as shown by the fact that many major libraries like sphinx, jupyterhub and a few others (including one I maintain) Panel did not adapt in time. In practice semver is indeed a dream most libraries can't live up to but this change in a minor version left a lot of people scrambling to catch up. I'm not in a place to ask anything of you, not knowing what the maintenance team looks like, but I would implore longer deprecation cycles for such major deprecations.

[maartenbreddels]

Would you considering adding a shim back in a patch release? I think that this will break a lot of downstream use cases.

I think with a module level getattr, this can be done cleanly.

Personally, I prefer what the Java world does. Deprecation yes, removal never, it just hurts too many people/users/developers.

[cculianu]

Personally, I prefer what the Java world does. Deprecation yes, removal never, it just hurts too many people/users/developers.

Indeed. In this case I would estimate hundreds of thousands, if not millions of dollars worth of developer time has been needlessly wasted, worldwide, due to all the breakages in public and private projects. I know our group spent about $2000 in developer time troubleshooting this and updating our docker images for CI to pin jinja2 to 3.0.0. And likely our predicament has been repeated globally thousands of times...

[davidism]

And yet, if downstream applications took on the responsibility of pinning their dependency tree, and testing with warnings as errors, they would have been able to control exactly when they received updates and been able to respond proactively to any changes. This is universal, it's not specific to Jinja. If you pin at least one dependency you have to pin the whole tree otherwise any library at any time could update while another remains pinned.

If applications weren't already doing that, me delaying a change that I advertised in the changelog and with deprecation warnings, isn't going to change that situation. It's just going to make them admit that they weren't pinning dependencies later instead of now.

I'm locking this now as I don't see this discussion continuing productively.