make all-pdf fails invocation of xindy

[JulienPalard]

On cpython, branch 3.7, if we run a make all-pdf in build/latex/ we're getting:

$ make all-pdf
latexmk -pdf -dvi- -ps-  'c-api.tex'
Latexmk: This is Latexmk, John Collins, 11 Nov. 2012, version: 4.35.
**** Report bugs etc to John Collins <collins at phys.psu.edu>. ****
Latexmk: applying rule 'makeindex c-api.idx'...
Rule 'makeindex c-api.idx': File changes, etc:
   Non-existent destination files:
      'c-api.ind'
------------
Run number 1 of rule 'makeindex c-api.idx'
------------
------------
Running 'internal xindy -L english -C utf8  -M sphinx.xdy -I xelatex  -o "c-api.ind" "c-api.idx"'
------------
Latexmk: calling xindy( -L english -C utf8 -M sphinx.xdy -I xelatex -o c-api.ind c-api.idx )
Unsupported input markup xelatex.

usage: xindy [-V?h] [-qv] [-d magic] [-o outfile.ind] [-t log] \
            [-L lang] [-C codepage] [-M module] [-I input] \
            [--interactive] [--mem-file xindy.mem] \
	    [idx0 idx1 ...]

GNU-STYLE LONG OPTIONS FOR SHORT OPTIONS:

 -V / --version
 -? / -h / --help
 -q / --quiet
 -v / --verbose
 -d / --debug          (multiple times)
                       (supported: script, keep_tmpfiles, markup, level=n)
 -o / --out-file
 -t / --log-file
 -L / --language
 -C / --codepage
 -M / --module         (multiple times)
 -I / --input-markup   (supported: latex, omega, xindy)

Latexmk: Errors, so I did not complete making targets

Environment info

• OS: Ubuntu 14.04.5 LTS
• Python version: 3.6.2
• Sphinx version: 1.8.1
• xindy from http://us.archive.ubuntu.com/ubuntu/ version 2.4-1.3

[JulienPalard]

With xindy 2.5.1.20160104-5 from http://ftp.fr.debian.org/debian buster/main it works.

[tk0miya]

I checked recent releases:

• xindy on bionic (18.04) ... suppoted
• xindy on xenial (16.04) ... not supported

@jfbu I think -I xelatex option for xindy is too new. So I think it should be removed. What do you think?

[jfbu]

@tk0miya non-support of -I xelatex by old xindy is annoying. I can not access a 16.04 system until 10 days from now.

To dispel potential misunderstanding this option is not in Sphinx context in any direct way related to using or not using xelatex. Basically it tells xindy to accept Unicode UTF-8 encoded characters in index files, and it was needed also with pdflatex for Sphinx, for some reason (perhaps related to support for Cyrillic). As I remember the matter we can not drop it without breaking something. At the current time I have forgotten the exact details. Will come back later.

edit later on, I figured this option is perhaps not really needed, despite what I wrote above. Initially I used texindy but it proved better to call xindy directly. The documentation is not clear at all on what -I xelatex actually does; I added it initially to make sure xindy did not use its file xindy/modules/tex/inputenc/utf8.xdy (for reasons explained at

sphinx/sphinx/texinputs/sphinx.xdy

Lines 45 to 51 in a4cce3e

	;; But, this means for pdflatex and Latin scripts that the xindy file
	;; tex/inputenc/uf8.xdy is not usable because it refers to the macro
	;; \IeC only sporadically, and as tex.xdy is not loaded, a rule such as
	;; (merge-rule "\'e" "é" :string)
	;; does not work, it must be
	;; (merge-rule "\IeC {\'e}" "é" :string)
	;; So Sphinx contributes LICRlatin2utf8.xdy to mitigate that problem.

), but this might have been at a time I was still experimenting with texindy, not xindy directly.

[jfbu]

The TeXLive distributed xindy.pl perl script supports -I xelatex since TeXLive 2014 (it has there version number v 1.18 2011/01/18) but the TeXLive 2013 version does not support it (it came with v 1.16 2010/05/10).

[jfbu]

hmm.. on some testing with options -v -d level=3 -d keep_tmpfiles to xindy, it appears that using -I xelatex or -I latex or nothing changes nothing on my test case which mixes Cyrillic and English. It might be that I thought -I xelatex was needed when I was experimenting with using texindy, not xindy directly.

By the way, the option -I omega appears to be broken on xindy side, as it tries to use tex2xindy -o but this gives rise to an error due to missing quoting

Running filter: "/Library/TeX/texbin/tex2xindy -o" <"/var/folders/np/4_9pppfs54xdv9m79m7rwx4w0000gn/T/BBvkTG5e5G" >"/var/folders/np/4_9pppfs54xdv9m79m7rwx4w0000gn/T/NREKR8e2sE"
sh: /Library/TeX/texbin/tex2xindy -o: No such file or directory

Anyway, I guess we can simply drop the option. I will do some further testing.

[jfbu]

@JulienPalard I can not build PDF docs of cpython due to this

.. index:: single: : (colon); path separator (POSIX)
   single: ; (semicolon)
.. data:: pathsep

at lines 4058-4060 in cpython/Doc/library/os.rst. The semi-colon at line 4059 is mis-interpreted it seems and this becomes

\index{: (colon)!path separator (POSIX)}\index{!(semicolon)}\ignorespaces \index{pathsep (in module os)}

in latex file library.tex and then this causes

ERROR: CHAR: index 0 should be less than the length of the string

during xindy processing. (the problematic LaTeX code is \index{!(semicolon)}, which tries to index an empty string, with a link to semicolon indexing).

[jfbu]

The same problem arises at Lines 21-25

.. index::
   single: clause
   single: suite
   single: ; (semicolon)

of Doc/reference/compound_stmts.rst.

[jfbu]

@tk0miya is it at all possible to index the ; character using index role or directive? my various attempts fail.

[jfbu]

@JulienPalard with #5577 I can successfully build (English) Python documentation for current 3.7 branch. And with #5574 it should work also with old xindy.

[JulienPalard]

Can confirm that with #5577 and #5574 it works. But I never spotted the need for #5577.

[jfbu]

@JulienPalard thanks for confirming. On my laptop, with Sphinx 1.8 dev tip, I had the #5576 problem with both reference.tex and library.tex as reported above, for the 3.7 branch.

This #5576 is a real Sphinx LaTeX writer bug, good thing that it was caught on that occasion! (makeindex has more forgiveness than xindy, the empty index entry did not bother it).

At #5583, another #5576 fix is proposed, and I posted screenshots to show how the index of library.pdf index looks like with it.

[jfbu]

Closing as the reported issue is fixed for upcoming 1.8.2 release at a695099. Thanks for report!