Feature Request: support upLaTeX for PDF builds of Japanese projects

[jfbu]

Many projects will use Unicode characters. Sphinx uses pLaTeX for Japanese documents but pLaTeX is not Unicode aware.

It appears upLaTeX adds Unicode support and can help handling problems with stray Unicode characters: in particular using \usepackage[utf8]{inputenc} and babel is possible with upLaTeX. From the little I understand it is easier with upLaTeX to have mixed languages documents, for example with parts in English and others in Japanese.

Currently one can induce Sphinx into using upLaTeX via this latexmkjarc file in the repertory of the project

$latex = 'uplatex ' . $ENV{'LATEXOPTS'} . ' %O %S';
$dvipdf = 'dvipdfmx %O -o %D %S';
$makeindex = 'rm -f %D; upmendex -f -d %B.dic -s python.ist %S || echo "upmendex exited with error code $? (ignoring)" && : >> %D';
add_cus_dep( "glo", "gls", 0, "makeglo" );
sub makeglo {
 return system( "upmendex -s gglo.ist -o '$_[0].gls' '$_[0].glo'" );
}

and an latex_additional_files = ['latexmkjarc'] line in conf.py. I am not sure the options to uplatex and upmendex above are optimal. I just took the current file which uses platex and mendex and dropped options not existing for uplatex/upmendex.

But we also need extra uplatex option in document class in case of 'manual' type (which uses jsbook.cls).

This option does not work for 'howto' documents and one must use ujreport.cls as replacement for jreport.cls. Currently this setting appears to require some monkey-patch by user of sphinx.builders.latex's default_latex_docclass. (see #4136 (comment))

Perhaps hyperref options for the encoding of PDF strings (particularly, bookmarks) is perhaps not good with upLaTeX. Currently sphinx.sty contains this logic:

% to make pdf with correct encoded bookmarks in Japanese
% this should precede the hyperref package
\ifx\kanjiskip\undefined
% for non-Japanese: make sure bookmarks are ok also with lualatex
  \PassOptionsToPackage{pdfencoding=unicode}{hyperref}
\else
  \RequirePackage{atbegshi}
  \ifx\ucs\undefined
    \ifnum 42146=\euc"A4A2
      \AtBeginShipoutFirst{\special{pdf:tounicode EUC-UCS2}}
    \else
      \AtBeginShipoutFirst{\special{pdf:tounicode 90ms-RKSJ-UCS2}}
    \fi
  \else
    \AtBeginShipoutFirst{\special{pdf:tounicode UTF8-UCS2}}
  \fi
\fi

I don't know if this is the good thing to do for upLaTeX. (I don't even know if it is so good for pLaTeX, for example I noticed there are problems with EN DASH).

Adding upLaTeX support could be good in particular for localization to Japanese of documents when localization is only partial. The CPython docs is an example.

Relates

the discussion at #4136

Environment info

• Sphinx version: 1.6.5

[tk0miya]

+1 moving to uplatex is reasonable.

• I don't know upmendex is optimal or not. mendex also supports UTF-8 with -U option.
• We need to discuss what document class is better for uplatex. In Japanese docs, we usually uses js* classes. But jsreport was appeared in past few years. so it is distributed in TeXLive since 2017. So it might be better to use jsbook (w/ uplatex option) and ujreport at present.
• We also have to change the macro to set up PDF bookmarks. The trick should be used for only platex.
• In addition, we should set up European characters handling to support Greek and Cyrillic characters.

I will ask the migration ways to Japanese TeX experts, and investigate personally.

[JulienPalard]

I don't know about uplatex and never tried it, but it looks like the Japanese translation of the Python documentation could benefit from it, as it contains some characters like kelvin sign, and ſ. See previously referenced issue.

[t-tk]

The most large difference between mendex -U and upmendex is supported scripts.
The former supports English and Japanese. The latter supports Latin (in­clud­ing non-English), Greek, Cyril­lic, Korean Han­gul and Han (Hanzi ideographs, Chinese) scripts, as well as Ja­panese Kana.
Another difference is upmendex is newer (since 2016 in TeX Live) than mendex (very long history).

[tk0miya]

Now we still support Ubuntu xenial. Therefore we also support TeXLive 2015. Using upmendex will be next major updates.

[tk0miya]

Done in #6841. I don't know it is perfect. So please file an issue if you'll get trouble.
Thanks,