external filters in format strings are not called

[kstephan-wescale]

Expected Behaviour

In my muttrc I have

set sidebar_format = "nmutt_sidebar_format '%_D%?F? [%F]?' '%N' '%S'|"

which should call the nmutt_sidebar_format script with the provided arguments after expansion and give lines like:

trash            61

This works fine with versions until including neomutt-20240329.

Actual Behaviour

With neomutt-20240416 as well as neomutt-git-20240416.r33.gb01e1fd78 it yields lines like that:

nmutt_sidebar_format 'Trash' '0' '61'|

Steps to Reproduce

Use external filters in format strings as explained in 30.3. Filters.

How often does this happen?

• Always

When did it start to happen?

• When I upgraded from neomutt-20240329 to neomutt-20240416.

NeoMutt Version

neomutt -v output

NeoMutt 20240416
Copyright (C) 2015-2024 Richard Russon and friends
NeoMutt comes with ABSOLUTELY NO WARRANTY; for details type 'neomutt -vv'.
NeoMutt is free software, and you are welcome to redistribute it
under certain conditions; type 'neomutt -vv' for details.

System: Linux 6.8.6-arch1-1 (x86_64)
ncurses: ncurses 6.4.20230520 (compiled with 6.4.20230520)
libidn2: 2.3.7 (compiled with 2.3.7)
GPGME: 1.23.2
GnuTLS: 3.8.5
libnotmuch: 5.6.0
storage: kyotocabinet, gdbm, lmdb
compression: lz4, zlib, zstd

Configure options: --prefix=/usr --sysconfdir=/etc --libexecdir=/usr/lib --gpgme --sqlite --autocrypt --lua --notmuch --gss --gnutls --sasl --with-ui=ncurses --with-idn2=/usr --disable-idn --idn2 --lmdb --kyotocabinet --gdbm --ubsan --lz4 --zlib --zstd

Compilation CFLAGS: -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -fexceptions         -Wp,-D_FORTIFY_SOURCE=2 -Wformat -Werror=format-security         -fstack-clash-protection -fcf-protection -g -ffile-prefix-map=/build/neomutt/src=/usr/src/debug/neomutt -flto=auto -fno-delete-null-pointer-checks -D_ALL_SOURCE=1 -D_GNU_SOURCE=1 -D__EXTENSIONS__ -D_XOPEN_SOURCE_EXTENDED -I/usr/include/lua5.3 -I/usr/include -I/usr/include -I/usr/include -DNCURSES_WIDECHAR -I/usr/include -I/usr/include/p11-kit-1 -I/usr/include -I/usr/include -fsanitize=undefined -fno-omit-frame-pointer -fno-common -g -O0

Compile options:
  +autocrypt +fcntl -flock -fmemopen +futimens +getaddrinfo +gnutls +gpgme
  -gsasl +gss +hcache -homespool +idn +inotify -locales_hack +lua -mixmaster
  +nls +notmuch -openssl +pgp +regex +sasl +smime +sqlite +truecolor

Devel options:
  ubsan

MAILPATH="/var/mail"
PKGDATADIR="/usr/share/neomutt"
SENDMAIL="/usr/sbin/sendmail"
SYSCONFDIR="/etc"

To learn more about NeoMutt, visit: https://neomutt.org
If you find a bug in NeoMutt, please raise an issue at:
    https://github.com/neomutt/neomutt/issues
or send an email to: <neomutt-devel@neomutt.org>

Extra Info

• Operating System and its version – up-to-date Arch Linux
• Were you using multiple copies of NeoMutt at once? – no
• Were you using 'screen' or 'tmux'? – yes, tmux
• Is your email local (maildir) or remote (IMAP)? – local

[flatcap]

Ah, sorry. That got overlooked in the upgrade.

Please can you give branch [devel/expando] a try.

Thanks

[kstephan-wescale]

Thanks for getting onto it so fast!
It throws a segfault, here’s some gdb output:

@(gdb) run
Thread 1 "neomutt" received signal SIGSEGV, Segmentation fault.
                                                               check_for_pipe (root=<optimized out>) at expando/expando.c:132
132         if (last->start[i] == '\\')
@(gdb) backtrace
#0  check_for_pipe (root=<optimized out>) at expando/expando.c:132
#1  expando_render (exp=<optimized out>, rdata=0x555555709fa0 <SidebarRenderData>, data=0x7fffffffdd10, flags=<optimized out>, max_cols=19, buf=0x555555738450) at expando/expando.c:209
#2  0x0000555555671186 in make_sidebar_entry (buflen=256, shared=0x5555557b2af0, sbe=0x5555559497f0, width=19, buf=0x5555559498f0 "") at sidebar/window.c:593
#3  sb_recalc (win=<optimized out>) at sidebar/window.c:873
#4  0x00005555555f4a6d in window_recalc (win=win@entry=0x555555949aa0) at gui/mutt_window.c:579
#5  0x00005555555f4a88 in window_recalc (win=win@entry=0x5555559453f0) at gui/mutt_window.c:585
#6  0x00005555555f4a88 in window_recalc (win=win@entry=0x55555592e9c0) at gui/mutt_window.c:585
#7  0x00005555555f4a88 in window_recalc (win=win@entry=0x55555592e700) at gui/mutt_window.c:585
#8  0x00005555555fcd0b in window_redraw (win=<optimized out>) at gui/mutt_window.c:642
#9  0x000055555559d3b9 in dlg_index (dlg=<optimized out>, m_init=<optimized out>) at index/dlg_index.c:1243
#10 0x00005555555615f1 in main (argc=<optimized out>, argv=<optimized out>, envp=<optimized out>) at /usr/src/debug/neomutt-git/neomutt/main.c:1430

[flatcap]

Thanks for the backtrace; very helpful.
Turns out I'm an idiot :-)
When working backwards through a string, don't use ++

Please can you give the [devel/expando] branch another try.
It's working robustly now (and has some tests, too)

Thanks!

[kstephan-wescale]

It works like a charm. Thanks a ton for your work!

[flatcap]

Brilliant! Thanks!

[elpeh]

Hello, continuing...: 202404125 leads to calling the external filter program for (index, in this case) format string.
BUT: documentation states: "30.3 Filters [...] If the returned string ends in %, it will be passed through the formatter a second time. This allows the filter to generate a replacement format string including % expandos. [...]"
This reevaluation is not happening, generated format string value is printed in index literally (in contrast to 20240416 and earlier).

[flatcap]

Well spotted, @elpeh.
I'm working on it.

[elpeh]

Thanks. As my setup is quite old, I'll meanwhile try to replace all/most using the index-format-hook feature.