Space+Underline is not rendered properly #4901

ceccopierangiolieugenio · 2023-12-06T15:10:58Z

Details

Browser and browser version:
- Firefox: 120.0
- Vivaldi: 6.4.3160.47 (Stable channel) stable (64-bit)
- Edge: 119.0.2151.97 (Official build) (64-bit)
OS version:
- Linux: Pop!_OS 22.04 LTS
xterm.js version:
- "xterm": "^5.3.0"

Steps to reproduce

follow the installation steps on the homepage: https://xtermjs.org

npm install xterm

Use this html testing file:

<!doctype html>
  <html>
    <head>
      <link rel="stylesheet" href="node_modules/xterm/css/xterm.css" />
      <script src="node_modules/xterm/lib/xterm.js"></script>
    </head>
    <body>
      <div id="terminal"></div>
      <script>
        var term = new Terminal();
        term.open(document.getElementById('terminal'));
        term.write('                      # CSI Ps m  Character Attributes (SGR).\n\r')
        term.write('Hello from \x1B[1;0;31mxterm.js   test\x1B[0m Ps=0 -> Normal (default), VT100.\n\r')
        term.write('Hello from \x1B[1;1;31mxterm.js   test\x1B[0m Ps=1 -> Bold, VT100.\n\r')
        term.write('Hello from \x1B[1;2;31mxterm.js   test\x1B[0m Ps=2 -> Faint, decreased intensity, ECMA-48 2nd.\n\r')
        term.write('Hello from \x1B[1;3;31mxterm.js   test\x1B[0m Ps=3 -> Italicized, ECMA-48 2nd.\n\r')
        term.write('Hello from \x1B[1;4;31mxterm.js   test\x1B[0m Ps=4 -> Underlined, VT100.\n\r')
        term.write('Hello from \x1B[1;5;31mxterm.js   test\x1B[0m Ps=5 -> Blink, VT100. - Bold in X11R6 xterm.\n\r')
        term.write('Hello from \x1B[1;7;31mxterm.js   test\x1B[0m Ps=7 -> Inverse, VT100.\n\r')
        term.write('Hello from \x1B[1;8;31mxterm.js   test\x1B[0m Ps=8 -> Invisible, i.e., hidden, ECMA-48 2nd, VT300.\n\r')
        term.write('Hello from \x1B[1;9;31mxterm.js   test\x1B[0m Ps=9 -> Crossed-out characters, ECMA-48 3rd.\n\r')
      </script>
    </body>
  </html>

Issue:

as shown here:

The fifth line should render three underlined spaces but those are not vivible

This issue is noticeable also if I use pyTermTk, where only the different chars of the buffer are sent to the terminal but the entire line is affected:

Regression

xterm.js - 5.0.0
No issues:
xterm.js - 5.1.0, 5.2.0
Some artifacts are displayed:
xterm.js - 5.3.0
Underlined Spaces are not rendered:

jerch · 2023-12-07T12:44:44Z

@ceccopierangiolieugenio Whats the exact byte content sent for that underline line?

ceccopierangiolieugenio · 2023-12-07T13:16:04Z

@jerch
I managed to reproduce the issue just pushing this string to the terminal as reported in the description:
'\x1B[1;4;31mxterm.js test\x1B[0m'
Is it there a way to enable some debug info to check which exact bytes are sent/received?

jerch · 2023-12-07T13:32:28Z

You can get byte content on console.log, if you enable the debug log in the terminal. I am asking for the byte content, because your screenshot for xterm.js - 5.1.0, 5.2.0 indicates some issues with encoding (looks like mixing latin-1 with utf-8). Once we have ruled that out, we can check if the renderers do crazy things.

ceccopierangiolieugenio · 2023-12-07T14:57:57Z

New test where I used for testing:

      <div id="terminal"></div>
      <script>
        var term = new Terminal({
            logLevel: 'debug',
            allowProposedApi: true});
        term.open(document.getElementById('terminal'));
        term.write('                      # CSI Ps m  Character Attributes (SGR).\n\r')
        term.write('Hello from \x1B[1;0;31mxterm.js   test\x1B[0m Ps=0 -> Normal (default), VT100.\n\r')
        term.write('Hello from \x1B[1;1;31mxterm.js   test\x1B[0m Ps=1 -> Bold, VT100.\n\r')
        term.write('Hello from \x1B[1;3;31mxterm.js   test\x1B[0m Ps=3 -> Italicized, ECMA-48 2nd.\n\r')
        term.write('Hello from \x1B[1;4;31mxterm.js   test\x1B[0m Ps=4 -> Underlined, VT100.\n\r')
        term.write('Hello from \x1B[4;2mxterm.js   test\x1B[0m Ps=4 -> Underlined, VT100.\n\r')
        term.write('Hello from \x1B[4;5mxterm.js   test\x1B[0m Ps=4 -> Underlined, VT100.\n\r')
        term.write('Hello from \x1B[1;9;31mxterm.js   test\x1B[0m Ps=9 -> Crossed-out characters, ECMA-48 3rd.\n\r')
      </script>

Here is a comparison between the different versions, the bytes sent are identical;

xterm.js 5.0.0

Sent: "1 �[1;0;31mA B  C�[0mD"
[ 49, 32, 27, 91, 49, 59, 48, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "2 �[1;1;31mA B  C�[0mD"
[ 50, 32, 27, 91, 49, 59, 49, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "3 �[1;3;31mA B  C�[0mD" 
[ 51, 32, 27, 91, 49, 59, 51, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "4 �[1;4;31mA B  C�[0mD" 
[ 52, 32, 27, 91, 49, 59, 52, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "5 �[4;2mA B  C�[0mD" 
[ 53, 32, 27, 91, 52, 59, 50, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "6 �[4;5mA B  C�[0mD" 
[ 54, 32, 27, 91, 52, 59, 53, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "7 �[1;9;31mA B  C�[0mD" 
[ 55, 32, 27, 91, 49, 59, 57, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]

xterm.js 5.1.0

| | | |

Sent: "1 �[1;0;31mA B  C�[0mD" 
[ 49, 32, 27, 91, 49, 59, 48, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "2 �[1;1;31mA B  C�[0mD" 
[ 50, 32, 27, 91, 49, 59, 49, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "3 �[1;3;31mA B  C�[0mD" 
[ 51, 32, 27, 91, 49, 59, 51, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "4 �[1;4;31mA B  C�[0mD" 
[ 52, 32, 27, 91, 49, 59, 52, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "5 �[4;2mA B  C�[0mD" 
[ 53, 32, 27, 91, 52, 59, 50, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "6 �[4;5mA B  C�[0mD" 
[ 54, 32, 27, 91, 52, 59, 53, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "7 �[1;9;31mA B  C�[0mD" 
[ 55, 32, 27, 91, 49, 59, 57, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]

xterm.js 5.3.0

| | |

Sent: "1 �[1;0;31mA B  C�[0mD" 
[ 49, 32, 27, 91, 49, 59, 48, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "2 �[1;1;31mA B  C�[0mD" 
[ 50, 32, 27, 91, 49, 59, 49, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "3 �[1;3;31mA B  C�[0mD" 
[ 51, 32, 27, 91, 49, 59, 51, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "4 �[1;4;31mA B  C�[0mD" 
[ 52, 32, 27, 91, 49, 59, 52, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "5 �[4;2mA B  C�[0mD" 
[ 53, 32, 27, 91, 52, 59, 50, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "6 �[4;5mA B  C�[0mD" 
[ 54, 32, 27, 91, 52, 59, 53, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]
Sent: "7 �[1;9;31mA B  C�[0mD" 
[ 55, 32, 27, 91, 49, 59, 57, 59, 51, 49, 109, 65, 32, 66, 32, 32, 67, 27, 91, 48, 109, 68, 10, 13 ]

jerch · 2023-12-07T18:30:56Z

Hmm where does that dotted uppercase A come from (not even sure which unicode thingy that is)? How do you send the data to the terminal? If those numbers in the array are indeed the sent bytes to the terminal, then this weird char should not have occurred at all.
Can you expose the term variable on top level and try to post here the bufferline content? You can grab it like that from the console:

<youTermVar>._core.buffer.lines.get(<lineNumber>).translateToString()

(Note that I cannot repro any of this, so there is mostly something off on your end and how you provide the data to the terminal.)

jerch · 2023-12-07T18:58:48Z

Plz also see https://stackoverflow.com/a/1462039/12548337, might explain that Å char. Maybe you forgot to declare the html document as utf-8?

tisilent · 2023-12-08T02:02:06Z

@jerch I have verified. It's encoding issue, but why does the xterm demo work properly. 😟

tisilent · 2023-12-08T02:39:20Z

😟

ceccopierangiolieugenio · 2023-12-08T09:33:05Z

Thanks @jerch it is exactly the issue reported in:
https://stackoverflow.com/a/1462039/12548337
I fixed it adding in the header:

   <head>
      <meta charset="utf-8">
   <head>

The issue was caused by the encoding of the '\xa0' in:

xterm.js/src/browser/renderer/dom/DomRendererRowFactory.ts

Lines 272 to 274 in 957e3e0

    
           if (text === ' ') { 
        
             text = '\xa0'; // = &nbsp; 
        
           }

Unfortunately I never managed to run the demo, I amnot a js/ts expert and I don'know how to build and start it.

jerch · 2023-12-08T10:12:05Z

@jerch I have verified. It's encoding issue, but why does the xterm demo work properly. 😟

The demo uses utf-8 encoding on its document. For the DOM renderer this is indeed crucial, as we forward any unicode chars directly to the DOM. With non unicode-capable encoding the DOM then would make something else out of those chars.
No clue if this has an impact on the other renderers and whether the document encoding also changes text drawing onto canvas.

jerch · 2023-12-08T10:36:34Z

@Tyriar The need for an utf-8 encoded document for the DOM renderer feels like a surprising limitation (in fact it isnt when thinking about it for a few seconds, also it is recommended as default with html5). Should we mention that in some sort of a document setup guide?

Tyriar added the needs more info label Dec 7, 2023

ceccopierangiolieugenio closed this as completed Dec 8, 2023

ceccopierangiolieugenio reopened this Dec 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Space+Underline is not rendered properly #4901

Space+Underline is not rendered properly #4901

ceccopierangiolieugenio commented Dec 6, 2023 •

edited

jerch commented Dec 7, 2023 •

edited

ceccopierangiolieugenio commented Dec 7, 2023 •

edited

jerch commented Dec 7, 2023

ceccopierangiolieugenio commented Dec 7, 2023

jerch commented Dec 7, 2023 •

edited

jerch commented Dec 7, 2023 •

edited

tisilent commented Dec 8, 2023

tisilent commented Dec 8, 2023

ceccopierangiolieugenio commented Dec 8, 2023 •

edited

jerch commented Dec 8, 2023 •

edited

jerch commented Dec 8, 2023 •

edited

Space+Underline is not rendered properly #4901

Space+Underline is not rendered properly #4901

Comments

ceccopierangiolieugenio commented Dec 6, 2023 • edited

Details

Steps to reproduce

Issue:

Regression

jerch commented Dec 7, 2023 • edited

ceccopierangiolieugenio commented Dec 7, 2023 • edited

jerch commented Dec 7, 2023

ceccopierangiolieugenio commented Dec 7, 2023

xterm.js 5.0.0

xterm.js 5.1.0

xterm.js 5.3.0

jerch commented Dec 7, 2023 • edited

jerch commented Dec 7, 2023 • edited

tisilent commented Dec 8, 2023

tisilent commented Dec 8, 2023

ceccopierangiolieugenio commented Dec 8, 2023 • edited

jerch commented Dec 8, 2023 • edited

jerch commented Dec 8, 2023 • edited

ceccopierangiolieugenio commented Dec 6, 2023 •

edited

jerch commented Dec 7, 2023 •

edited

ceccopierangiolieugenio commented Dec 7, 2023 •

edited

jerch commented Dec 7, 2023 •

edited

jerch commented Dec 7, 2023 •

edited

ceccopierangiolieugenio commented Dec 8, 2023 •

edited

jerch commented Dec 8, 2023 •

edited

jerch commented Dec 8, 2023 •

edited