Does simdjson parse escaped invalid UTF-8 bytes?

[DUOLabs333]

I'm currently using Boost.JSON when sending JSON messages. I am thinking about using this library to read JSON, while using Boost.JSON to write it. However, as part of my program, Boost.JSON writes out arbitrary bytes as escaped character sequences. I know that simdjson doesn't accept invalid UTF-8, but will the library correctly parse escaped bytes into native bytes?

[lemire]

The simdjson supports the full JSON RFC, including escaped characters inside strings.

[DUOLabs333]

I know that Boost.JSON escapes all invalid UTF-8 as I don't see any validation errors when I read the JSON back in. I may test if base64 fixes my issue before switching to simdjson.

[jkeiser]

If it's escaped it might work; but since escape codes represent Unicode codepoints, I don't think json escape sequences can represent any possible binary input. Literally I don't think it's possible without just slamming raw binary invalid utf-8 into the string.

[DUOLabs333]

Update: As it turns out, Simdjson errors out on at least one escaped byte (which byte(s), I don't know, as I'm sending ~100000000 bytes with each parse). Using Turbo-Base64 fixes my problem.

[lemire]

@jkeiserWe validate Unicode characters during decoding because we guarantee an UTF-8 output.

@DUOLabs333 You can put base64 data in a string, it is a safe solution. It will probably result in a smaller payload and faster decoding.

[jkeiser]

@lemire yep. And even if we didn't validate Unicode escape sequences to ensure they were in the proper range, there are binary sequences that just can't be represented by either valid utf-8 or Unicode escape sequences translated to utf-8. e.g. 1100000 1100000. Without a base 64 or other encoding, arbitrary binary data just can't be put into JSON.

[jkeiser]

@DUOLabs333 I just realized, you can disable utf-8 validation with a #define, and that might let you do binary data in strings without losing any other functionality. I think we are 8-bit clean without utf-8 validation with the exception of \ and ", which unavoidably must be escaped. I'm not sure your format; while it isn't JSON, I imagine it's something like "JSON but strings can contain arbitrary binary data".

[jkeiser]

But beware; if you write JSON like this, your pseudo-JSON is likely to be rejected by most JSON parsers (and is guaranteed to be rejected by standards compliant JSON parsers).

[DUOLabs333]

Which define?

[jkeiser]

#define SIMDJSON_UTF8VALIDATION 0 before you include any simdjson files.