Shifting characters

[rivo]

It looks like some Unicode characters will cause the output to shift. Below is an example program that writes three characters in the first three positions (0, 1, and 2) but the last character (a normal dollar sign) is actually output in position 3 instead of 2.

I don't know if this is an actual error on tcell's end because when I output the same string to the console, the first character will actually occupy two cells instead of one, so I'm actually forcing tcell to do something that's probably not right. Still, I would expect a character to be put in a specific position to be output at that position. Maybe you can comment.

package main

import "github.com/gdamore/tcell/v2"

func main() {
	// Setup.
	screen, err := tcell.NewScreen()
	if err != nil {
		panic(err)
	}
	if err = screen.Init(); err != nil {
		panic(err)
	}
	width, _ := screen.Size()

	// Draw ruler.
	for x := 0; x < width; x++ {
		screen.SetContent(x, 0, '0'+rune(x%10), nil, tcell.StyleDefault)
	}

	// Draw three characters.
	str := []rune("खास$")
	screen.SetContent(0, 1, str[0], str[1:2], tcell.StyleDefault) // First character.
	screen.SetContent(1, 1, str[2], nil, tcell.StyleDefault)      // Second character.
	screen.SetContent(2, 1, str[3], nil, tcell.StyleDefault)      // Third character (dollar sign).

	// Sync.
	screen.Sync()

	// Finish upon key press.
	for {
		_, ok := screen.PollEvent().(*tcell.EventKey)
		if ok {
			break
		}
	}
	screen.Fini()
}

(This is macOS with iTerm2.)

[darkhz]

@rivo here you've said:

last character (a normal dollar sign) is actually output in position 3 instead of 2.

Shouldn't the last character be at position 4?

[rivo]

This line in the code above puts the dollar sign in position 2:

screen.SetContent(2, 1, str[3], nil, tcell.StyleDefault)      // Third character (dollar sign).

But as you can see in the screenshot, it appears in position 3. (It's a bit difficult to see because it overlaps with the previous character.)

[gdamore]

Terminals can do unwanted things when you try to overlap wide and narrow characters. It's poor form really to even attempt this.

Tcell issues the request, but because of the way we draw, we might wind up thinking that the characters are adjacent or not depending on the flow. And what happens when you issue those characters in a single stream (without a position reset first) is likely what you see.

One thing we can explore in tcell is to detect this case and reset the position in the case of an attempt to overlap. This probably has a better (but not guaranteed) chance of working reasonably (overlapping characters).

But really, its kind of an error on the part of the caller to expect anything reasonable when using overlapping characters like this. Unicode (and emoticons especially) make this a lot messier because there isn't yet universal agreement about how many cells such characters should occupy, and it can often depend on the specific font being used for display.

[gdamore]

The other thing is that whatever we do we are going to be very dependent on the width of the glyph on screen matching what we think (per the runewidth library) the width should be. Discrepancies are not entirely uncommon here, but there isn't really anything I can do about it.

[rivo]

Yeah, I guessed so. The problem is, currently, the go-runewidth package can't reliably determine the width of these characters (see mattn/go-runewidth#59). I mean, it was already quite difficult to find a solution for Unicode characters which consist of multiple runes. And it seems now that what we came up with is not 100% bulletproof.

I guess you can close this issue then. I'll follow up on the go-runewidth side.