filters applied in wrong order

[fkv1]

According to the documentation:

To apply more than one filter, separate them by a comma:
${" some value " | h,trim}
The HTML escaping function is applied first, the “trim” function second.

But when I do:

<%
  def keep_spaces(p_s):
    return p_s.replace(' ','&nbsp;')
%>
${x |h, keep_spaces}

it transforms " ABC" into

  ABC

instead of just

  ABC

[zzzeek]

cant reproduce, please provide a script

example

from mako.template import Template


t = Template("""
<%
  def keep_spaces(p_s):
    return p_s.replace(' ','&nbsp;')
%>
${x |h, keep_spaces}

""")

print(t.render(x=' ABC'))

output:

$ python test3.py 


 ABC

[zzzeek]

oh, it's not wrong. h is applied first. take a look what you get:

from mako.template import Template


t = Template("""
<%
  def keep_spaces(p_s):
    breakpoint()
    return p_s.replace(' ','&nbsp;')
%>
${x |h, keep_spaces}

""")

print(t.render(x=' ABC'))

run it:

$ python test3.py 
> /home/classic/dev/mako/memory:0x7f6d8b5501a0(26)keep_spaces()
(Pdb) !p_s
Markup(' ABC')
(Pdb) !p_s.replace(' ',' ')
Markup(' ABC')

the docs are correct

[zzzeek]

what's causing confusion is 'h' is giving you a Markup object , not a string, so you have to work with that API

[fkv1]

Markup objects seem to be difficult to handle, and I think that this behaviour comes as a surprise to every inadept programmer. Is there any advantage in 'h' returning a markup object instead of a simple string?

I've made my code work by changing the keep_spaces() function to:

    return p_s.replace(' ','\u00a0')

(At least it works with Firefox, which treats the unicode character for non-breaking space the same way as the html entity.)

But this doesn't feel like a general solution to the problem.

[zzzeek]

Markup objects seem to be difficult to handle, and I think that this behaviour comes as a surprise to every inadept programmer. Is there any advantage in 'h' returning a markup object instead of a simple string?

Likely security issues, CVEs and such. Someone contributed it long ago. Feel free to make your own "simple" HTML filter.

But this doesn't feel like a general solution to the problem.

Mako is not really maintained very much, only recently have we gotten some PRs, and we definitely don't want to deal with HTML escaping that's fertile ground for CVE posters.

[fkv1]

What security issues? I can't think of any. I think there's a much simpler explanation: The cgi module has been deprecated in newer python versions. That's why Mako switched from cgi.escape() to markupsafe.escape(). The fact that the result is a different data type probably has been overlooked or not perceived as a concern, because it doesn't normally change the outcome – except in very special cases as in my example. There has certainly never been a regression test for such a rare case, so the issue remained unnoticed.

[zzzeek]

Any security issues. As mentioned, I'm not looking at Mako much, nobody else is, but Markupsafe, lots of people are looking at it and they have security in mind. It's lead paragraph is all about security: https://github.com/pallets/markupsafe?tab=readme-ov-file#markupsafe .

Here's the commit: a0354c3 , which mentions that it adds escaping for single quotes. I didn't think of that, but Markupsafe did. Whatever else is going on in HTML-escaping land, an area that I haven't worked professionally in since the late 1990s, is more likely to be accommodated by a well used third party library rather than Mako which I'm not really looking at on any regular basis (until this week when suddenly two people seem very interested in it).

I could just as well write my own crypto algorithm, and say, "it has no security issues!" because nobody is using it.

Markupsafe is an industry standard library used by Jinja and it is not controversial that any HTML templating library is better served by using a well-worn library like this rather than rolling their own.

[fkv1]

Thank you for your quick replies, which stand in stark contrast to your repeated remarks that Mako is no longer maintained. When the entire house is collapsing, it doesn't make sense to talk about one broken light bulb.

When I had to choose between Mako a year ago, most comparisons stated that Mako is the slightly wider and longer established of the two, so it seemed like a safe bet. Now that you indicate it is a dead horse, does it mean we all need to migrate from Mako to Jinja sooner or later?

[zzzeek]

Thank you for your quick replies, which stand in stark contrast to your repeated remarks that Mako is no longer maintained.

I'm trying to see where I wrote that, I wrote, "Mako is not really maintained very much, only recently have we gotten some PRs, and we definitely don't want to deal with HTML escaping that's fertile ground for CVE posters.", so, colloquialism here, "no longer maintained" != "not really maintained very much", we of course do releases and bug fixes and all that. I just do not have a current initiative of the pattern "what am I going to add to Mako this month, test, document, release, support, fix for years, all by myself, to make it better?" I have too much else going on to devise, test, implement, and support new features alone. I dont know what the current Python templating ecosystem looks like and if it is in fact still just Jinja2 and Mako that would seem...pretty surprising.

When the entire house is collapsing, it doesn't make sense to talk about one broken light bulb.

But after all that, this is not a broken light bulb. The Markup object is the best choice here and we certainly can't just switch it out as that would break billions of existing templates that rely on its behavior. the fact that it continues to escape new strings that are added to it is certainly good for security; users who want to render raw HTML and open themselves up for scripting attacks if they aren't careful have to explicitly opt into that, and that insulates Mako from causing security problems even more.

Mako using a widely used and trusted library for HTML escaping is still what I'd be doing even if Mako were my only project full time.

When I had to choose between Mako a year ago, most comparisons stated that Mako is the slightly wider and longer established of the two, so it seemed like a safe bet.

That's not true at all, take a look at stats:

https://www.pepy.tech/projects/jinja2 - 27,452,524 downloads in 7 days
https://www.pepy.tech/projects/mako - 6,371,501 in 7 days

Jinja2 is the template language I see in basically everything Python that I didnt write, Sphinx, Ansible, etc. Mako is my own little thing that I personally prefer but as it allows raw Python code, most people dont really want that.

Now that you indicate it is a dead horse, does it mean we all need to migrate from Mako to Jinja sooner or later?

it's certainly not going anywhere and releases will keep working, but we can't just change API behaviors that have been the same way for 15 years

[fkv1]

Well, you already changed the API behavior in 2010 breaking existing templates.
That said, I think that neither change (string -> markup object, and back) breaks many existing templates, as the |h filter is not usually combined with other filtes, or if so, it comes last.

I don't see how continuing escaping added strings would add to security. That's a not directly documented behaviour, thus unexpected to the developers, and it's never a good thing when functions behave differently than expected. Secondly, if I wanted the added string to be escaped, I'd add it before applying the |h filter. That's a no-brainer.

What makes this even worse is that there's no way to switch that escaping off, apart from converting the markup object back to a string. Converting forth and back is detrimental to performance.

Of course, every developer can write their own HTML-escaping filter, but you can't recommend that when you talk about security concerns, as every code that is written from scratch can contain mistakes, and all the supposed magic that markupsafe.escape() does would have to be re-invented by each developer individually.

I tried to overload the |h filter with a self-written function anyway (by redefining the h function), but for some reason Mako's h function was called instead of mine. I bet there is a proper way to do it and I'm just too unskilled as a python developer, but anyway it's all unnecessarily complicated. It would all be so easy if the h filter returned just a string, as (as far I see) all the other predefined filters do.

[zzzeek]

we dont have a public API for overloading the existing filters. the h filter is documented as using a markupsafe object so short of having that word be hyperlinked, everything is working as it should. we are not changing any behaviors of Mako at this time.