Handling invalid unicode escapes

[GULPF]

Background: json5/json5-spec#12

JSON5.parse uses the ES6 behavior and accepts invalid Unicode sequences like "\uDEAD". However, this is not the case for the CLI. The CLI instead outputs U+FFFD (the replacement character). This probably happens because of Node trying to ensure that the output is valid UTF-8. I think it's best if these codepoints are escaped in the output, so that "\uDEAD" generates the escaped output "\uDEAD".

[jordanbtucker]

I wasn't able to reproduce this issue. Can you please provide some code that demonstrates this?

[GULPF]

input.json5 contains "\uDEAD". After running json5 input.json5 -o output.json, output.json contains the following bytes (in hex): 22 EF BF BD 22, which is a string only containing the replacement character.

[jordanbtucker]

Thanks for the example. That clarified the issue for me.

Your assumption about why this happens is correct. Since both JSON and JSON5 allow for invalid code unit sequences, the CLI should preserve them, however this would be a breaking change. The best approach may be to add a CLI flag in v2.2 that escapes invalid code point sequences. In v3.0 we can make that behavior the default and replace the CLI flag with one that replaces invalid code unit sequences with \uFFFD.