LaTeX: release of LaTeX from June 2021 causes extra vertical whitespace in cells of complex tables

[jfbu]

Describe the bug
A clear and concise description of what the bug is.

With recently released LaTeX of 2021-06-01 which adds "paragraph hooks", they cause extra whitespace due to bad interaction with varwidth package.

To Reproduce
Steps to reproduce the behavior:

complex tables
==============


+--------------+----------+-----------+-----------+
| row 1, col 1 | column 2 | column 3  | column 4  |
+--------------+----------+-----------+-----------+
| row 2        | one                              |
| row 2        |                                  |
| row 2        | two                              |
| row 2        |                                  |
| row 2        | three                            |
+--------------+----------+-----------+-----------+
| row 3        |          |           |           |
+--------------+----------+-----------+-----------+


+--------------------+----------------+
|  2 rows and 2 cols | 1 row x 3 cols |
|                    +----+-----+-----+
|                    |  A |  B  |  C  |
+---+----------------+----+-----+-----+
| 1 |       2        |  3 |  4  |  5  |
+---+----------------+----+-----+-----+


+----------+--------------------------------+------------------------+----------+----------+----------+----------+----------+
| Header 1 | Long Header 2 that should wrap | Common column Header 3 | Header 4 | Header 5 | Header 6 | Header 7 | Header 8 |
|          |                                +-----------+------------+          |          |          |          |          |
|          |                                | Header 3a | Header 3b  |          |          |          |          |          |
+==========+================================+===========+============+==========+==========+==========+==========+==========+
| Value 1  | Value 2 does actually wrap     | Value 3a  | Value 3b   | Value 4  | Value 5  | Value 6  | Value 7  | Value 8  |
+----------+--------------------------------+-----------+------------+----------+----------+----------+----------+----------+

and make latexpdf. (I am using #9314 because these tables show the Sphinx 4.0 bug #9313).

With TeXLive 2021 and the latest LaTeX release we get this

With TeXLive 2019 we got this with no extra space

Environment info

• OS: [e.g. Unix/Linux/Mac/Win/other with version]
• Python version: [e.g. 3.7.1]
• Sphinx version: 4.x with Fix #9313 #9314, or prior to 4.0
• Sphinx extensions: [e.g. sphinx.ext.autodoc, recommonmark]
• Extra tools: Bug shows with LaTeX release 2021-06-01.

Additional context

This is LaTeX upstream ticket latex3/latex2e#583

But I have not checked in detail yet.

Not setting a milestone, hoping it gets fixed upstream at LaTeX.

However, as varwidth is a third-party package not maintained by LaTeX team, it is not guaranteed there will be a fix from upstream and we perhaps will have to take some measures.

Notice that the LaTeX 2021-06-01 added paragraph hooks could also badly interfere with Sphinx code-blocks but this was anticipated at #9000.

I have not yet checked for other potential breakage from upstream LaTeX changes.

[jfbu]

There has been a LaTeX upstream update but it does not solve the issue in our context: the extra whitespace appears below not above the paragraph in merged cells of complex tables:

1. The upstream latex update is via version of file latex2e-first-aid-for-external-files.ltx dated 2021/06/08. It is a file which is automatically included during the latex format generation which happens when the user updates the installation via package manager. One sees no trace of it in log file.
2. As this is potentially time consuming I have not yet checked if the above phenomenon is a bad interaction with our usage of varwidth or really is due to lacunary fix of varwidth compatibility by latex2e-first-aid-for-external-files.ltx
3. The "latex2e-first-aid" mechanism is supposed to be provisory until the third-party packages get fixed. But, after one or two years, they will stop including their fixes in the latex format itself. So whatever happens the safest is probably for us to apply a fix ourselves. Which however is complicated and annoying because it implies making it in a way compatible to whatever happens on upstream latex or package side.

[jfbu]

As of 2021/06/24 version of LaTeX internal file latex2e-first-aid-for-external-files this problem has been fixed upstream.

I am thus closing this as it was a transient upstream problem.