Zero width chars not showing in diff

[aryzing]

🐛 Bug Report

Differences in strings containing zero width chars are not highlighted, making them incredibly hard to spot and debug. Eg, one of the following strings has as its first character (character at position 0) a byte order mark character as its first character, and the other has the letter "T" as its first character.

Additional useful info

• UTF-16 spec on Byte Order Mark character (scroll to end of page). Its hex code is FEFF, or 65279 in decimal.
• Related issues: expect: Highlight substring differences when matcher fails, part 1 #8448, Better reporting for string differences #6881

To Reproduce

Just create two strings that look identical, one of which contains a character such as the BOM character.

Expected behavior

A clear indication of the difference.

envinfo

  System:
    OS: Linux 5.4 Ubuntu 18.04.5 LTS (Bionic Beaver)
    CPU: (8) x64 Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz
  Binaries:
    Node: 14.4.0 - ~/.nvm/versions/node/v14.4.0/bin/node
    Yarn: 1.21.1 - /usr/bin/yarn
    npm: 6.14.5 - ~/.nvm/versions/node/v14.4.0/bin/npm
  npmPackages:
    jest: ^26.0.1 => 26.0.1 

[devil-cyber]

hey can I work on this issue

[SimenB]

@devil-cyber sure, thanks!

[devil-cyber]

Can you please explain a bit how to get satrted

…

On Tue, Oct 6, 2020, 4:27 PM Simen Bekkhus ***@***.***> wrote: @devil-cyber <https://github.com/devil-cyber> sure, thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10584 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMPIV6SRF6YBAJ7PXUA6NDLSJLZZ7ANCNFSM4SD3APZA> .

[SimenB]

This is really @pedrottimark's domain, but it should be somewhere in https://github.com/facebook/jest/tree/5da90b58eb6673e6f406f9ebc369b1f0fb747384/packages/jest-diff. Maybe @jeysal could provide some more details?

[devil-cyber]

Ok

…

On Tue, Oct 6, 2020, 5:00 PM Simen Bekkhus ***@***.***> wrote: This is really @pedrottimark <https://github.com/pedrottimark>'s domain, but it should be somewhere in https://github.com/facebook/jest/tree/5da90b58eb6673e6f406f9ebc369b1f0fb747384/packages/jest-diff. Maybe @jeysal <https://github.com/jeysal> could provide some more details? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10584 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMPIV6RNGEGSFLYBHMN4X6LSJL5T5ANCNFSM4SD3APZA> .

[jeysal]

I would expect most of the changes to be in pretty-format more likely than jest-diff.
We want to serialize a string starting with certain characters in a different way.
There should probably be a brief discussion on how exactly this should be visualized, ideally with input from @aryzing as well.
Maybe "\uFEFFbla", but then how do we distinguish it from a string actually containing those characters (afaik backslashes are not an escape character in formatted strings)? Maybe move it before the "?

[devil-cyber]

Ok

…

On Tue, Oct 6, 2020, 5:06 PM Tim Seckinger ***@***.***> wrote: I would expect most of the changes to be in pretty-format more likely than jest-diff. We want to serialize a string starting with certain characters in a different way. There should probably be a brief discussion on how exactly this should be visualized, ideally with input from @aryzing <https://github.com/aryzing> as well. Maybe "\uFEFFbla", but then how do we distinguish it from a string actually containing those characters (afaik backslashes are not an escape character in formatted strings)? Maybe move it before the "? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10584 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMPIV6URBXMS6UGEMQXIOIDSJL6MXANCNFSM4SD3APZA> .

[aryzing]

@jeysal on my side no strong preferences on display, just thought I'd bring this up as it might trip up others in the future. Perhaps there's appetite for visually separating parts of the string corresponding to invisible characters or whitespace characters other than the typical whitespace you get with the spacebar.

Maybe (fyi, 00A0 is no-break space)

Expected: "Rest of string"
Received: \uFEFF + "Rest of" + \u00A0 + "string"

Like I said, no strong feelings either way, I'm sure that any reasonable indication of the presence of such characters will be fine.

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[aryzing]

Believe this isn't resolved yet, is it?

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 30 days.

[ibuibu]

I was just thinking about a solution for the issue.

Sorry I couldn't declare it before working on it.

I will try to create a PR as well.

I thought #13997 was a very good approach.

I thought the issue was that simply replacing a zero-width space with a no-break space makes it difficult to recognize that it is a zero-width space.
Also, there are multiple zero-width spaces other than FEFF, so I think this needs to be detected as well.

I took the approach of using Note to tell that a zero-width space exists. The disadvantages of this approach are that it is difficult to visually see where the zero-width space exists and that it is impossible to detect when there are multiple zero-width spaces.

I hope my PR will be of help and reference for a good solution.